[experiments] Daily Experiment Report — 2026-10-10 #67383

🧪 Daily Experiment Report — 2026-10-10

52 experiments analysed across 49 workflows. 26 are 🟢 READY on sample size, but none reached a PROMOTE or REJECT decision: 38 are EXTEND and 14 are INCONCLUSIVE. Run-level outcome charts were skipped; the core decision is the canonical result.

⚡ Quick Stats

🔎 Key findings

• insufficient_observations (24 experiments) blocks every READY experiment with enough samples, for example testqualitysentinel/model_size (174 vs 176) and smokecopilot (183 vs 183). No outcome observations feed the decision layer yet.
• guardrail_unsupported (3 experiments): cicoach/prompt_style, dailysecurityredteam/reasoning_depth and dailyfact/reasoning_depth are READY, but run_success_rate and empty_output_rate are unsupported native metrics.
• unsupported_multi_variant (14 experiments): any experiment with 3 or more variants is INCONCLUSIVE. This includes awfailureinvestigator/tone_variant (221/204/198), deepreport/output_format (4 variants) and dailycompilerquality/output_format. The three model_size experiments also record stray agent and small-agent variants (dailydocupdater, dailycavemanoptimizer, dailydochealer).
• insufficient_samples (9 experiments) are still collecting, for example dailyarchitecturediagram (12/8) and dependabotcampaign (6/6). copilotprnlpanalysis has 0 usable samples.

🔁 Self-Tuning Continuation Plan

Top experiment actions

1. Add per-run outcome observations (run success, output length) to the state so the 24 insufficient_observations experiments can resolve. Start with the READY two-variant ones: testqualitysentinel, issuearborist, dailynews and the smokecopilot* family.
2. Support native run_success_rate and empty_output_rate guardrails to unblock cicoach, dailysecurityredteam and dailyfact.
3. Reduce awfailureinvestigator, deepreport and dailycompilerquality to two variants, or add multi-variant support to core analysis. Clean up the stray agent and small-agent variant names in the model_size experiments.

Top eval actions

1. No eval scores were available in this run. Define graders for output-format experiments (ste, prose, structured) that emit verbosity and readability observations.
2. Add a grader for empty or failed outputs, mapped to the empty_output_rate guardrail.
3. Add accuracy-oriented evals for the reasoning_depth and model_size experiments, since they have no quality metric.

Decision-pipeline gaps for next PR

• Native metrics lacking observations: run_success_rate, empty_output_rate, plus primary metrics in the 24 affected experiments.
• Make factorial interaction diagnostics (for example smokecopilot subagent_model × caveman) a normalized core signal.
• Support K≥3 variants in decision logic (Bonferroni-corrected pairwise comparisons against control).

Descriptive window: last 30 runs · No tracking issues were configured, so no issue comments or labels were posted.
Run: 38035975287

Generated by 🧪 daily-experiment-report · copilot · auto · 21.7 AIC · ⌖ 0.93 AIC · ⊞ 16.3K · ◷

• expires on Oct 13, 2026, 12:01 AM UTC-08:00