[agent-job-health] Agent Job Health Monitor daily report #67304

Summary

• Window evaluated: 2026-10-09T15:31:57Z – 2026-10-09T23:21:02Z UTC (~7.8h). The local pre-downloaded log cache did not cover a full 24h window this run — see note below.
• total_runs: 263 · distinct_workflows: 49 · agent_failures (confirmed): 8
• Run-weighted fleet failure rate: 3.04% (8/263, confirmed basis). Upper bound if every indeterminate run were an agent failure: 18.25% (48/263).
• Workflow-weighted: median 0.0%, mean 13.62% (over the 41 workflows with full job-level data).
• Tracked vs. novel: 0 tracked / 8 confirmed-novel agent-job failures (issue search for known signatures returned no accessible matches — see caveat below). Plus 40 runs in an indeterminate cluster (see Novel Failure Clusters).
• Trend: Confirmed-basis rate (3.0%) is lower than 2026-10-07 (12.77%), but this is not a like-for-like comparison — today's local sample is much smaller and narrower than prior runs (7.8h vs ~24h), and 40 of today's 61 overall-failed runs could not be attributed to a specific job. Not treating this as a real improvement.

Failure Rate by Step (confirmed agent-job failures only, n=8)

Tracked Failures

None. Issue search (search_issues) returned zero accessible results for every query tried against today's failure signatures and the known chronic-offender names (PR Sous Chef, Issue Monster, Contribution Check, Copilot CLI segfault) — all of which appear in today's sample with a 0% failure rate, consistent with them being already-mitigated/tracked. Every other search had all candidate results removed by the integrity policy (resource trust level below "approved"), so absence of a hit here should not be read as confirmation that no tracking issue exists — only that this run could not verify one.

Novel Failure Clusters

1. Indeterminate PR-reviewer-fleet failures — 40 runs, 8 workflows (AI Moderator, Visual Regression Checker, Matt Pocock Skills Reviewer, Test Quality Sentinel, Impeccable Skills Reviewer, Ponytail Reviewer, Design Decision Gate 🏗️, PR Code Quality Reviewer). Engine: mixed (copilot/codex). Representative run: §37956584779 (AI Moderator, pull_request, conclusion failure). Root cause unknown: both list_workflow_jobs and get_job_logs return total_count: 0 for every one of these runs, and the local log cache never downloaded run_summary.json/aw_info.json for them either — all triggered by pull_request events on Copilot-authored branches (actor Copilot/copilot-swe-agent). Suspected cause: restricted job-log visibility for PR runs opened by the Copilot coding agent, not necessarily the agent job itself failing (could be detection, which is out of scope). Recommended action: investigate why Actions job-level data is unavailable for these runs (token scope? retention? a different workflow structure for Copilot-branch PRs?) before concluding this is an agent-job regression.
2. Build gh-aw Docker image step failure — 2 runs, 2 workflows (Daily Safe Output Tool Optimizer §37990260860, Agentic Workflow Audit Agent §37992875335), engine: claude. The step failed in ~11s (not a real compile failure — looks like a docker buildx setup/cache error); available log excerpt only shows post-job teardown, not the actual error. Below the 10%/2-workflow issue threshold on its own but worth a follow-up log pull.
3. Execute GitHub Copilot CLI step failure — 2 runs, 2 workflows (Matt Pocock Skills Reviewer §37997730167, Daily Reliability Review §38001741108). Low volume this window.
4. Execute Codex CLI / Install Codex CLI step failures — 3 runs total across Daily Go Test Parallelizer (x2) and Daily Cache Strategy Analyzer (x1), single-workflow-dominant, below threshold.

Schedule Heartbeat

No blind spots detected in the checked sample. Because the local log cache covered only ~7.8h instead of 24h, 202 of 241 schedule-triggered workflows had no run in the local sample — this is expected given the narrow window, not evidence of a blind spot by itself. A stratified sample of 32 of those 202 (daily/sub-daily cadence, excluding smoke-* fixtures and weekly+ cadences) was checked directly against the GitHub Actions API (list_workflow_runs, most recent run of any kind): 32/32 had a run within their expected cadence (all within the last ~20h), several with conclusion: failure (already covered under indeterminate/novel clusters above where applicable) but none silent beyond 2× cadence + 1 day. Full exhaustive verification of all 241 scheduled workflows was not completed this run (budget); the remaining ~170 (mostly daily/every 2 days smoke fixtures) were not individually checked.

Recommendations

1. Investigate the Actions job-visibility gap for pull_request-triggered runs on Copilot-authored branches (40 runs this window alone) — this blocks accurate agent-job attribution for a meaningful share of the PR-reviewer fleet and should be fixed before the next report, not worked around again.
2. Pull full logs for the "Build gh-aw Docker image" failures (Daily Safe Output Tool Optimizer, Agentic Workflow Audit Agent) — the ~11s failure duration suggests a buildx/cache setup issue, not a real compile break; worth a quick fix before it spreads to more workflows.
3. No action needed on the schedule front — sampled heartbeat check found 32/32 healthy; recommend a future run budget more log-download capacity so the local cache actually spans 24h, which would make both the fleet rate and the heartbeat check load-bearing without manual API backfill.
4. Re-run issue-tracking lookups with elevated integrity scope if available — this run's search_issues calls were filtered for nearly every query, so the tracked/novel split above should be treated as a lower bound on "tracked."

Generated by 🩺 Agent Job Health Monitor · claude · agent · 824.6 AIC · ⊞ 6.7K · ◷

• expires on Oct 12, 2026, 3:48 PM UTC-08:00