Add model and engine misconfiguration diagnostics to agentic-workflows by Copilot · Pull Request #67488 · github/gh-aw

Copilot · 2026-10-10T18:24:54Z

Model/endpoint failures led the skill to recommend older models before diagnosing outdated gh-aw compilers and wire-API mismatches. Misplaced CLI and AWF version pins also lacked actionable guidance.

Routing and checklists: Route symptoms directly to aligned guidance in the full debug guide and root debug.md.
Diagnosis and remedies: Explain prerelease installation, alias resolution, wire-API precedence, session-wide endpoint constraints, and relevant artifacts. Prioritize upgrading and recompiling, then compatible model selection; reserve downgrades for last resort.
Version fields: Map Copilot CLI install failures to engine.version and AWF failures to sandbox.agent.version; explicitly reject engine.copilot.version.
Regression coverage: Add redacted customer-case fixtures and contract tests guarding remedy order, version-field mapping, artifact guidance, and routing.

Fixes agentic-workflows skill: add a checklist for model and engine misconfiguration #67486

Co-authored-by: SivaKesava1 <11771739+SivaKesava1@users.noreply.github.com>

github-actions · 2026-10-10T18:38:17Z

🧠 Matt Pocock Skills Reviewer is reviewing this pull request using Matt Pocock's engineering skills...

github-actions · 2026-10-10T18:38:22Z

🔬 Test Quality Sentinel is analyzing test quality on this pull request...

github-actions · 2026-10-10T18:38:22Z

✂️ Ponytail Reviewer has started processing this pull request

github-actions · 2026-10-10T18:38:25Z

🔎 PR Code Quality Reviewer is reviewing code quality for this pull request...

Copilot

🟡 Changes recommended

The new regression suite is not registered with any required Make or CI test target.

1 open finding

Contract tests are not wired into required CI execution · New

What changed in this PR

Adds targeted diagnostics for model, endpoint, and version-pin misconfiguration in agentic workflows.

Changes:

Routes relevant failures to a new debugging checklist.
Documents compiler, wire-API, version-field, and artifact diagnosis.
Adds documentation contract fixtures and tests.

File	Description
`debug.md`	Adds condensed public troubleshooting guidance.
`.github/aw/debug-agentic-workflow.md`	Adds the comprehensive diagnostic checklist.
`.github/skills/agentic-workflows/SKILL.md`	Routes matching symptoms to the checklist.
`.github/skills/agentic-workflows/tests/test_guidance.py`	Adds documentation contract tests.
`.github/skills/agentic-workflows/tests/fixtures.json`	Adds synthetic regression scenarios.

🧠 Review effort: Balanced

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

SivaKesava1 · 2026-10-10T20:22:45Z

@copilot Please merge main. The red agentic workflow checks on this PR came from a safe outputs regression on main (the safe outputs MCP server could not load memory_schema_contract.cjs), which #67506 fixed. Please keep this PR to the #67486 scope and don't fix unrelated failures here. We'll review the checklist content once the checks are green.

…-for-misconfiguration Co-authored-by: SivaKesava1 <11771739+SivaKesava1@users.noreply.github.com>

Copilot · 2026-10-10T20:29:58Z

@copilot Please merge main. The red agentic workflow checks on this PR came from a safe outputs regression on main (the safe outputs MCP server could not load memory_schema_contract.cjs), which ...

Merged main (including #67506) into this PR in merge commit 53e9b864a8, keeping the changes scoped to #67486.

gh-aw-bot · 2026-10-10T20:49:00Z

@copilot address the following outstanding work in one pass:

Update this branch with the latest main using make merge-main, resolving any conflicts and preserving the intended changes.
Review (.github/skills/agentic-workflows/tests/test_guidance.py:92): These contract tests are only executable manually: no Make target or CI workflow discovers this directory, and test-all covers Go, JavaScript, WASM, and shell tests only. As a result, guidance can regress without this suite failing, so the promised regression guard is ineffective until this command is wired into a required test target or workflow. - Add model and engine misconfiguration diagnostics to agentic-workflows #67488 (comment)

Push the necessary fixes, reply to each listed review thread and resolve it when addressed. Ignore feedback already answered or resolved. Use the pr-finisher skill and stop when only human review or CI remains; do not trigger CI.

Sous-chef head: 53e9b86
Sous-chef work: 260832280d089284589673418b5b6ec56f78f5eaa47828823e7adffd46fa4fb4
Sous-chef state: 14b8584b9c5fdffacbace89f6295c20c6b8c8a4d3c5dcaff6bfce01ff66da8c1

Generated by 👨‍🍳 PR Sous Chef · pi · haiku45 · 3.99 AIC · ⌖ 7.65 AIC · ⊞ 1K · ◷
_{Comment /souschef to run again}

Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>

Copilot · 2026-10-10T21:04:29Z

@copilot address the following outstanding work in one pass:

Update this branch with the latest main using make merge-main, resolving any conflicts and preserving the intended changes....

make merge-main completed; the branch already contained the latest main. Added make test-guidance and wired it into test-all; validation passed and the change is in 58752b2. I replied to the requested review thread, but its resolution could not be performed: the available GitHub tools do not expose thread resolution, and gh has no GH_TOKEN in this environment. CI was not triggered.

SivaKesava1 · 2026-10-10T21:08:19Z

@copilot Content review of the checklist (in addition to gh-aw-bot's point about running the guidance tests in CI):

The wire API order in step 2 describes main before Select Copilot wire APIs from AWF metadata and surface model mismatches #67487. Select Copilot wire APIs from AWF metadata and surface model mismatches #67487 (open) makes AWF's per-model endpoint metadata (/reflect supported_endpoints) take precedence and fail fast on a mismatch at startup (Model endpoint mismatch: … Pin a compatible model, remove the COPILOT_PROVIDER_WIRE_API override, or upgrade gh-aw.). Either describe the order in a way that stays true after that lands, or add the AWF metadata step and the startup mismatch message.
"One wire API for the whole session, including sub-agents" holds for the default CLI mode only. With Fix Copilot SDK model routing and cross-family sub-agents #67509 (open), Copilot SDK mode (engine.copilot-sdk: true) runs each model through a provider for its own wire API, so a Claude sub-agent under a GPT main works there. Please mention SDK mode as the alternative once that lands, and keep the CLI-mode advice.
Add the silent fallback symptom. On AWF v0.28.50 a cross-family sub-agent fails with AWF's Routing model "<model>" to /chat/completions is incompatible … 400, and the main agent then redoes the work on its own model, so the run can succeed with no error in the final output. The checklist should say to look for subagent.failed events in usage/aw_session.jsonl and the audit's "Sub-agent Failed" finding and deviated requests. Please add AWF's Routing model … is incompatible message to the signature list.

Please keep this PR to the #67486 scope.

Co-authored-by: SivaKesava1 <11771739+SivaKesava1@users.noreply.github.com>

Copilot · 2026-10-10T21:17:35Z

@copilot Content review of the checklist (in addition to gh-aw-bot's point about running the guidance tests in CI):

The wire API order in step 2 describes main before Select Copilot wire APIs from AWF metadata and surface model mismatches #67487. Select Copilot wire APIs from AWF metadata and surface model mismatches #67487 (open) m...

Updated both checklists for AWF /reflect endpoint metadata and startup mismatch failures, documented the CLI-wide constraint and SDK-mode alternative, and added silent v0.28.50 sub-agent failure signatures/artifacts with contract coverage. Committed as a47b968; make test-guidance and final validation pass.

SivaKesava1 · 2026-10-10T21:30:13Z

@copilot /pr-finisher

Checked a47b968: the checklist now covers AWF /reflect precedence and the startup mismatch, SDK mode as the alternative, and the silent sub-agent fallback. make test-guidance passes (4 tests), and the diff stays within the #67486 scope. Thanks.

One fix before finishing, for the open review thread on test_guidance.py: test-guidance is now part of make test-all, but no workflow in .github/workflows/ runs test-all or test-guidance, so the guard still doesn't run in CI. Please add a step that runs make test-guidance to an existing Ubuntu job in .github/workflows/ci.yml (python3 is preinstalled on GitHub-hosted Ubuntu runners). Run it on changes to .github/skills/agentic-workflows/**, .github/aw/debug-agentic-workflow.md and debug.md if the changes filter supports that, otherwise always. Then reply on that thread. Keep everything else unchanged and within #67486's scope.

Co-authored-by: SivaKesava1 <11771739+SivaKesava1@users.noreply.github.com>

SivaKesava1 · 2026-10-10T22:09:19Z

Checked 14e8202. The guidance tests now run in CI (make test-guidance in ci.yml, filtered on .github/skills/agentic-workflows/**, .github/aw/debug-agentic-workflow.md and debug.md), so we resolved the review thread.

Blocking: the routing line in SKILL.md was removed. This commit deleted the entry that a47b968 added under "follow it directly":

AWF model/endpoint 400s, silent cross-family sub-agent failures, model: auto failures, install-step 404s after a version pin, or questions about version fields: go directly to Model and engine misconfiguration before suggesting model changes.

SKILL.md is now identical to main, so the skill no longer sends these cases to the new checklist. That routing is the core of #67486: the BASF case went wrong because the agent suggested switching to an older model instead of checking the gh-aw version and wire API first. Please restore the line, or explain in a reply why it had to go (for example a size or lint limit) and where these cases are now routed. Please also add a guidance test that asserts SKILL.md routes these signatures to the misconfiguration section, so this can't silently regress.

… in guidance tests Co-authored-by: SivaKesava1 <11771739+SivaKesava1@users.noreply.github.com>

Copilot · 2026-10-10T22:18:07Z

Checked 14e8202. The guidance tests now run in CI (make test-guidance in ci.yml, filtered on .github/skills/agentic-workflows/**, .github/aw/debug-agentic-workflow.md and debug.md), so we ...

The routing line is restored in 0e2132a.

Why it went missing: 14e8202 didn't remove it on purpose. make recompile runs gh aw init, which rebuilds SKILL.md from the embedded template pkg/cli/data/agentic_workflows_skill.md. a47b968 had only edited SKILL.md, so the rebuild dropped the line.

What changed:

I added the line to the template and regenerated SKILL.md, so it now stays after init or make recompile.
test_routing_and_evidence_links now checks both SKILL.md and the template. Each must have a line linking to ../../aw/debug-agentic-workflow.md#model-and-engine-misconfiguration that lists AWF model/endpoint 400s, silent cross-family sub-agent failures, `model: auto` failures, install-step 404s after a version pin, and version fields.

make test-guidance passes with the fix and fails without it. No other files changed.

memory.md had grown to 407 lines, over the 400-line instruction-file cap, after recent repo-memory JSON-schema-validation changes (#67439). Extract the self-contained "standalone ledger (experimental)" section into a new ledger.md, replacing it with a short pointer. Update the cross-reference in memory-stateful-patterns.md to link directly to ledger.md instead of an anchor inside memory.md. debug-agentic-workflow.md is also over the cap (432 lines) after recent model/engine-misconfiguration diagnostics (#67488), but its checklist content is pinned inline by a regression test (.github/skills/agentic-workflows/tests/test_guidance.py) that also requires it mirrored in root debug.md, so it is left as a documented exception rather than split. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Initial plan

c8577ed

Copilot AI assigned Copilot and SivaKesava1 Oct 10, 2026

Copilot AI linked an issue Oct 10, 2026 that may be closed by this pull request

agentic-workflows skill: add a checklist for model and engine misconfiguration #67486

Closed

Copilot started work on behalf of SivaKesava1 October 10, 2026 18:25 View session

Copilot AI and others added 2 commits October 10, 2026 18:33

Add model and engine misconfiguration checklist and regression contracts

8b843b2

Co-authored-by: SivaKesava1 <11771739+SivaKesava1@users.noreply.github.com>

Clarify model policy rejection and compatible sub-agent remedies

3e35095

Co-authored-by: SivaKesava1 <11771739+SivaKesava1@users.noreply.github.com>

Copilot AI changed the title ~~[WIP] Add checklist for diagnosing model and engine misconfiguration~~ Add model and engine misconfiguration diagnostics to agentic-workflows Oct 10, 2026

Copilot finished work on behalf of SivaKesava1 October 10, 2026 18:36

Copilot AI requested a review from SivaKesava1 October 10, 2026 18:36

SivaKesava1 marked this pull request as ready for review October 10, 2026 18:36

Copilot AI balanced review requested due to automatic review settings October 10, 2026 18:36

Copilot started reviewing on behalf of SivaKesava1 October 10, 2026 18:37 View session

Copilot AI reviewed Oct 10, 2026

View reviewed changes

Comment thread .github/skills/agentic-workflows/tests/test_guidance.py

Copilot started work on behalf of SivaKesava1 October 10, 2026 20:23 View session

Merge remote-tracking branch 'origin/main' into copilot/add-checklist…

53e9b86

…-for-misconfiguration Co-authored-by: SivaKesava1 <11771739+SivaKesava1@users.noreply.github.com>

Copilot finished work on behalf of SivaKesava1 October 10, 2026 20:30

github-actions Bot mentioned this pull request Oct 10, 2026

[pr-sous-chef] PR Sous Chef Status Report - 2026-10-10 10:25 #67397

Closed

Copilot started work on behalf of gh-aw-bot October 10, 2026 20:49 View session

github-actions Bot mentioned this pull request Oct 10, 2026

[Contribution Check Report] Contribution Check — 2026-10-10 #67317

Closed

Run guidance contracts in test-all

58752b2

Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>

Copilot finished work on behalf of gh-aw-bot October 10, 2026 21:05

Copilot AI requested a review from gh-aw-bot October 10, 2026 21:05

Copilot started work on behalf of SivaKesava1 October 10, 2026 21:08 View session

Copilot AI and others added 2 commits October 10, 2026 21:14

Plan model diagnostics follow-up

9a07654

Co-authored-by: SivaKesava1 <11771739+SivaKesava1@users.noreply.github.com>

Clarify model routing and silent sub-agent failures

a47b968

Co-authored-by: SivaKesava1 <11771739+SivaKesava1@users.noreply.github.com>

Copilot finished work on behalf of SivaKesava1 October 10, 2026 21:18

Copilot started work on behalf of SivaKesava1 October 10, 2026 21:30 View session

Copilot AI and others added 2 commits October 10, 2026 21:38

Run guidance contract tests in CI

180cc88

Co-authored-by: SivaKesava1 <11771739+SivaKesava1@users.noreply.github.com>

Apply remaining changes

14e8202

Co-authored-by: SivaKesava1 <11771739+SivaKesava1@users.noreply.github.com>

Copilot finished work on behalf of SivaKesava1 October 10, 2026 21:42

Copilot started work on behalf of SivaKesava1 October 10, 2026 22:10 View session

Restore model misconfiguration routing in skill template and guard it…

0e2132a

… in guidance tests Co-authored-by: SivaKesava1 <11771739+SivaKesava1@users.noreply.github.com>

Copilot finished work on behalf of SivaKesava1 October 10, 2026 22:18

SivaKesava1 requested a review from pelikhan October 10, 2026 22:46

SivaKesava1 and others added 3 commits October 10, 2026 15:46

Merge branch 'main' into copilot/add-checklist-for-misconfiguration

d192f51

Merge branch 'main' into copilot/add-checklist-for-misconfiguration

e6a3730

Merge branch 'main' into copilot/add-checklist-for-misconfiguration

e2f4716

pelikhan merged commit 9ded002 into main Oct 10, 2026
33 of 34 checks passed

pelikhan deleted the copilot/add-checklist-for-misconfiguration branch October 10, 2026 23:48

github-actions Bot mentioned this pull request Oct 11, 2026

[instructions] Split experimental ledger docs out of memory.md #67685

Open

Conversation

Copilot AI commented Oct 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 10, 2026

Uh oh!

github-actions Bot commented Oct 10, 2026

Uh oh!

github-actions Bot commented Oct 10, 2026

Uh oh!

github-actions Bot commented Oct 10, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

SivaKesava1 commented Oct 10, 2026

Uh oh!

Copilot AI commented Oct 10, 2026

Uh oh!

gh-aw-bot commented Oct 10, 2026

Uh oh!

Copilot AI commented Oct 10, 2026

Uh oh!

SivaKesava1 commented Oct 10, 2026

Uh oh!

Copilot AI commented Oct 10, 2026

Uh oh!

SivaKesava1 commented Oct 10, 2026

Uh oh!

SivaKesava1 commented Oct 10, 2026

Uh oh!

Copilot AI commented Oct 10, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Copilot AI commented Oct 10, 2026 •

edited

Loading