Add model and engine misconfiguration checklist and regression contracts

Co-authored-by: SivaKesava1 <11771739+SivaKesava1@users.noreply.github.com>
pelikhan · Oct 10, 2026 · Oct 10, 2026 · Oct 10, 2026 · Oct 10, 2026 · Oct 10, 2026
commit 8b843b2f6543cb22b78bc0d193c632ae267db1bf
diff --git a/.github/aw/debug-agentic-workflow.md b/.github/aw/debug-agentic-workflow.md
@@ -207,6 +207,9 @@ telemetry-free emitted lock; no workflow was executed."
 
 ## Collect Existing Evidence
 
+For model/endpoint errors, `model: auto` failures, or install failures after version
+pins, collect the artifacts in [Model and engine misconfiguration](#model-and-engine-misconfiguration).
+
 Use cached evidence or the workflow name/run URL already supplied. Without
 accessible existing evidence, start from source and fixtures, not a new run.
 For a run URL, extract its ID and begin with a compact audit:
@@ -263,6 +266,10 @@ does not identify a run uniquely. An absent/ambiguous match remains unknown.
 
 ## Identify the First Failing Boundary
 
+AWF model/endpoint 400s and version-pinned install-step 404s belong to
+[Model and engine misconfiguration](#model-and-engine-misconfiguration), not
+transient retries or prompt tuning.
+
 Separate the primary failure from cascades. Classify each failed message by
 tool/action, HTTP status, job, and time: a workflows-permission 403 and a
 `Bad credentials` 401 need different fixes.
@@ -285,6 +292,114 @@ Load syntax, safe-output, engine-runtime, campaign, or experiment references
 only for the mechanism under investigation. If a workflow itself collects logs,
 it needs `actions: read` and CLI installation before invoking `gh aw`.
 
+## Model and engine misconfiguration
+
+Use this checklist for AWF 400s mentioning models or endpoints, `model: auto`
+failures, install-step 404s after a version pin, or questions about version fields.
+
+### 1. Check the gh-aw compiler version first
+
+Read `compiler_version` in the lock file's `gh-aw-metadata` header and `cli_version`
+in `aw_info.json`. These identify the compiler used for the run; the locally
+installed `gh aw version` alone does not establish that the lock was recompiled.
+
+Compare with the newest published gh-aw release, **including prereleases**:
+
+```bash
+gh release list --repo github/gh-aw --limit 20
+gh extension install github/gh-aw --force --pin v0.91.7
+gh aw version
+gh aw compile WORKFLOW
+```
+
+The tag above is an example from the October 2026 customer case, not a permanent
+latest version. Select the newest applicable tag from the release list; increase
+the limit if needed. `gh extension install` and `gh extension upgrade` select the
+latest non-prerelease by default, so a successful upgrade can still leave gh-aw
+several releases behind. `--force --pin TAG` installs a specific prerelease even
+when the extension is already installed; later upgrades retain that pin.
+
+If the compiler predates the relevant fix, **upgrade gh-aw and recompile before
+trying other fixes**. In the customer case, v0.89.21 predated the wire-API
+inference fix in [github/gh-aw#64177](https://github.com/github/gh-aw/pull/64177);
+v0.91.7 was the newest prerelease reported then. Do not change only a runtime pin
+or retry the old lock and expect the compiler fix to apply.
+
+### 2. Match the resolved model to the wire API
+
+The Copilot harness resolves aliases such as `auto` through the AWF alias map and
+model catalog to a concrete model ID. An alias is not a fixed model: inspect the
+run's resolution rather than assuming what `auto` selected.
+
+For the normal Copilot harness path, `COPILOT_PROVIDER_WIRE_API` is chosen in this
+order:
+
+1. An explicit `engine.env.COPILOT_PROVIDER_WIRE_API` override.
+2. The resolved model's catalog `wire_api`.
+3. For a `-utility` model absent from the catalog, the base model's catalog entry.
+4. The `gpt-5+` name rule, which selects `responses`.
+5. The Copilot CLI default, `/chat/completions`, if nothing selected a wire API.
+
+`responses` selects `/responses`; `completions` selects `/chat/completions`.
+The CLI uses one wire API for the whole session, including sub-agents. A main
+model and sub-agent model requiring different endpoints cannot be repaired by
+changing only the sub-agent's model name. When `engine.model-routing` is enabled,
+also inspect AWF's selected model and endpoint rather than assuming this normal
+alias/inference path applies.
+
+Treat these signatures as model/endpoint incompatibility evidence, not a
+transient failure or a prompt problem:
+
+- `Cannot translate Copilot request feature`
+- `Unsupported Responses custom tool`
+- `model_policy_violation`
+- `not accessible via the … endpoint`
+
+Confirm the actual model and request path before diagnosing: the error may come
+from a sub-agent rather than the main model.
+
+### 3. Apply fixes in this order
+
+1. **Upgrade gh-aw and recompile.** Use a release containing the relevant fix.
+2. **Pin a model that supports the required endpoint.** Check the current catalog
+   and observed request path; do not assume all aliases or GPT models are interchangeable.
+3. **Remove a conflicting `COPILOT_PROVIDER_WIRE_API` override.** Let the updated
+   harness infer the endpoint unless an override is demonstrably required.
+4. **Use sub-agent models from the main model's family.** Verify endpoint
+   compatibility too; see [github/gh-aw#67460](https://github.com/github/gh-aw/issues/67460)
+   and [github/copilot-cli#5103](https://github.com/github/copilot-cli/issues/5103).
+
+**Switching to an older model is a last resort**, only if these fixes do not work.
+Leading with `model: gpt-4.1` trades capability for a workaround and leaves the
+underlying misconfiguration in place. Report any remaining endpoint constraint.
+
+### 4. Identify the version field from the failing install step
+
+| Setting | Meaning | Failing step and remedy |
+| --- | --- | --- |
+| `engine.version` | Agent CLI version; for Copilot, the Copilot CLI (1.0.x in the customer case), not gh-aw or AWF. | **Install GitHub Copilot CLI** 404s: remove this pin to use the compiled default, or verify the exact Copilot CLI release exists. |
+| `sandbox.agent.version` | AWF release version in `vX.Y.Z` form; must match a [github/gh-aw-firewall release](https://github.com/github/gh-aw-firewall/releases). | **Install AWF binary** failures: remove this pin to use the compiled default, or verify the AWF release and asset exist. |
+
+There is no `engine.copilot.version` field. Do not move a gh-aw version or an AWF
+version into `engine.version`. In most cases, remove the misplaced pin and
+recompile to use the compiler's compatible defaults. Removing a runtime pin does
+not upgrade gh-aw itself.
+
+### 5. Read the run artifacts
+
+| Artifact | Evidence |
+| --- | --- |
+| `agent-stdio.log` | `[copilot-harness]` lines for model alias resolution and the chosen `COPILOT_PROVIDER_WIRE_API`, including explicit overrides. |
+| `sandbox/firewall/logs/api-proxy-logs/token-usage.jsonl` | Model and path for each request: which endpoint each call actually used. |
+| `aw_info.json` | `model`, `requested_model`, `cli_version` (gh-aw compiler), `version` (agent CLI), and `awf_version` (AWF). |
+| `gh aw audit RUN_ID` | Combined view of run metadata, errors, and downloaded evidence. |
+
+Older versions may omit fields or harness diagnostics; absence is not proof of
+correct routing. Keep evidence redacted and distinguish observed facts from
+inference. Regression fixtures and contract tests for the customer case live in
+`.github/skills/agentic-workflows/tests/`; run them with
+`python3 -m unittest discover -s .github/skills/agentic-workflows/tests -v`.
+
 ## Fix and Report
 
 Apply the strategy and development compilation above for a regression-backed patch.

diff --git a/.github/skills/agentic-workflows/SKILL.md b/.github/skills/agentic-workflows/SKILL.md
@@ -99,6 +99,7 @@ After loading the matching workflow prompt or skill, follow it directly:
 - Configure or add declarative engines: `.github/aw/configure-agentic-engine.md`
 - Update existing workflows: `.github/aw/update-agentic-workflow.md`
 - Diagnose, patch, audit, or actively debug workflows: `.github/aw/debug-agentic-workflow.md` (local-first strategy, evidence triage and live gates)
+- AWF 400s mentioning models or endpoints, `model: auto` failures, install-step 404s after a version pin, or questions about version fields: go directly to [Model and engine misconfiguration](../../aw/debug-agentic-workflow.md#model-and-engine-misconfiguration) before suggesting model changes.
 - Upgrade workflows and fix deprecations: `.github/aw/upgrade-agentic-workflows.md`
 - Create shared components or MCP wrappers: `.github/aw/create-shared-agentic-workflow.md`
 - Create report-generating workflows: `.github/aw/report.md`

diff --git a/.github/skills/agentic-workflows/tests/fixtures.json b/.github/skills/agentic-workflows/tests/fixtures.json
@@ -0,0 +1,31 @@
+{
+  "model_mismatch": {
+    "evidence_kind": "redacted synthetic reconstruction, not live run evidence",
+    "lock_header": "# gh-aw-metadata: {\"compiler_version\":\"v0.89.21\"}",
+    "workflow": "engine:\n  id: copilot\n  model: auto\n",
+    "agent_stdio": "[copilot-harness] copilot model alias resolution: auto -> gpt-5.6-luna-utility\n[copilot-harness] COPILOT_PROVIDER_WIRE_API=completions (reconstructed old-run endpoint diagnostic)\n400 Cannot translate Copilot request feature 'tools[custom]' for gpt-5.6-luna-utility: /responses is incompatible",
+    "token_usage": {
+      "model": "gpt-5.6-luna-utility",
+      "path": "/chat/completions"
+    },
+    "expected_diagnosis": ["outdated gh-aw compiler", "resolved model and wire-API mismatch"],
+    "expected_fixes": ["Upgrade gh-aw and recompile.", "Pin a model that supports the required endpoint.", "Remove a conflicting `COPILOT_PROVIDER_WIRE_API` override.", "Use sub-agent models from the main model's family."],
+    "last_resort": "Switching to an older model is a last resort"
+  },
+  "misplaced_versions": {
+    "workflow": "engine:\n  id: copilot\n  version: 1.94.0\nsandbox:\n  agent:\n    version: 1.94.0\n",
+    "failures": [
+      {
+        "step": "Install GitHub Copilot CLI",
+        "error": "curl: (22) The requested URL returned error: 404",
+        "field": "engine.version"
+      },
+      {
+        "step": "Install AWF binary",
+        "error": "release asset not found",
+        "field": "sandbox.agent.version"
+      }
+    ],
+    "invalid_field": "engine.copilot.version"
+  }
+}
diff --git a/.github/skills/agentic-workflows/tests/test_guidance.py b/.github/skills/agentic-workflows/tests/test_guidance.py
@@ -0,0 +1,92 @@
+"""Documentation contract regressions, not an LLM or live workflow evaluation."""
+
+import json
+from pathlib import Path
+import unittest
+
+
+SKILL = Path(__file__).resolve().parents[1]
+ROOT = SKILL.parents[2]
+GUIDES = (ROOT / ".github/aw/debug-agentic-workflow.md", ROOT / "debug.md")
+FIXTURES = json.loads((SKILL / "tests/fixtures.json").read_text())
+ANCHOR = "#model-and-engine-misconfiguration"
+
+
+def checklist(text):
+    return text.split("## Model and engine misconfiguration\n", 1)[1].split("\n## ", 1)[0]
+
+
+class GuidanceTests(unittest.TestCase):
+    def test_customer_case_remedy_order(self):
+        case = FIXTURES["model_mismatch"]
+        self.assertIn("v0.89.21", case["lock_header"])
+        self.assertIn("model: auto", case["workflow"])
+        self.assertIn("gpt-5.6-luna-utility", case["agent_stdio"])
+        self.assertIn("Cannot translate Copilot request feature", case["agent_stdio"])
+        for path in GUIDES:
+            with self.subTest(path=path):
+                text = checklist(path.read_text())
+                positions = [text.index(fix) for fix in case["expected_fixes"]]
+                self.assertEqual(positions, sorted(positions))
+                self.assertLess(positions[-1], text.index(case["last_resort"]))
+                self.assertLess(text.index(case["last_resort"]), text.index("model: gpt-4.1"))
+                for required in (
+                    "v0.89.21", "64177", "compiler_version", "cli_version",
+                    "including prereleases", "latest non-prerelease",
+                    "gh extension install github/gh-aw --force --pin",
+                    "underlying misconfiguration in place",
+                ):
+                    self.assertIn(required, " ".join(text.split()))
+
+    def test_wire_api_and_artifact_contract(self):
+        for path in GUIDES:
+            with self.subTest(path=path):
+                text = checklist(path.read_text())
+                precedence = text[text.index("precedence"):] if "precedence" in text else text[text.index("order:"):]
+                positions = [precedence.index(term) for term in (
+                    "override", "wire_api", "-utility", "gpt-5+", "CLI default",
+                )]
+                self.assertEqual(positions, sorted(positions))
+                for required in (
+                    "/responses", "/chat/completions", "whole session",
+                    "sub-agents", "67460", "5103",
+                    "Cannot translate Copilot request feature",
+                    "Unsupported Responses custom tool", "model_policy_violation",
+                    "not accessible via the … endpoint",
+                    "agent-stdio.log", "[copilot-harness]",
+                    "sandbox/firewall/logs/api-proxy-logs/token-usage.jsonl",
+                    "requested_model", "awf_version", "aw_info.json", "gh aw audit RUN_ID",
+                ):
+                    self.assertIn(required, text)
+
+    def test_misplaced_version_fields(self):
+        case = FIXTURES["misplaced_versions"]
+        for path in GUIDES:
+            with self.subTest(path=path):
+                text = checklist(path.read_text())
+                for failure in case["failures"]:
+                    field_start = text.index(f"`{failure['field']}`")
+                    boundary = text.find("\n", field_start) if path == GUIDES[0] else text.index("point", field_start)
+                    self.assertIn(failure["step"], text[field_start:boundary])
+                self.assertIn("1.0.x", text)
+                self.assertIn("vX.Y.Z", text)
+                self.assertIn("gh-aw-firewall", text)
+                self.assertIn(f"There is no `{case['invalid_field']}` field", text)
+                self.assertIn("remove", text)
+                self.assertIn("compiled default", text)
+
+    def test_routing_and_evidence_links(self):
+        routing = SKILL.joinpath("SKILL.md").read_text()
+        line = next(line for line in routing.splitlines() if ANCHOR in line)
+        for symptom in ("AWF 400s", "model: auto", "install-step 404s", "version fields"):
+            self.assertIn(symptom, line)
+        self.assertTrue((SKILL / "../../aw/debug-agentic-workflow.md").resolve().is_file())
+        full = GUIDES[0].read_text()
+        for section in ("Collect Existing Evidence", "Identify the First Failing Boundary"):
+            content = full.split(f"## {section}\n", 1)[1].split("\n## ", 1)[0]
+            self.assertIn(ANCHOR, content)
+        self.assertIn(f".github/aw/debug-agentic-workflow.md{ANCHOR}", GUIDES[1].read_text())
+
+
+if __name__ == "__main__":
+    unittest.main()
diff --git a/debug.md b/debug.md
@@ -30,7 +30,9 @@ If it is installed, run:
 gh extension upgrade aw
 ```
 
-to upgrade to latest. If it is not installed, run the installation script from the main branch of the gh-aw repository:
+to upgrade to the latest non-prerelease. This can lag behind prereleases; see
+[Model and engine misconfiguration](#model-and-engine-misconfiguration).
+If it is not installed, run the installation script from the main branch of the gh-aw repository:
 
 ```bash
 curl -sL https://raw.githubusercontent.com/github/gh-aw/main/install-gh-aw.sh | bash
@@ -83,6 +85,54 @@ gh aw compile <workflow-name>
 
 3. Check for syntax errors or validation warnings.
 
+## Model and engine misconfiguration
+
+For AWF model/endpoint 400s, `model: auto` failures, install-step 404s after a
+version pin, or questions about version fields, use the
+[full checklist](.github/aw/debug-agentic-workflow.md#model-and-engine-misconfiguration):
+
+1. **Check the compiler first.** Read `compiler_version` from the lock file's
+   `gh-aw-metadata` header and `cli_version` from `aw_info.json`. Compare with
+   `gh release list --repo github/gh-aw --limit 20`, including prereleases.
+   `gh extension install` and `gh extension upgrade` default to the latest
+   non-prerelease. To install a specific prerelease, use
+   `gh extension install github/gh-aw --force --pin TAG`, then verify and recompile.
+   In the October 2026 case, v0.89.21 predated wire-API inference
+   ([#64177](https://github.com/github/gh-aw/pull/64177)); v0.91.7 was the reported
+   newest prerelease, not a permanent latest tag.
+2. **Check model/endpoint compatibility.** The harness resolves `auto` to a
+   concrete model. Wire-API precedence is explicit `engine.env` override →
+   catalog `wire_api` → `-utility` base-model catalog fallback → `gpt-5+` name rule
+   → CLI default `/chat/completions`. `COPILOT_PROVIDER_WIRE_API=responses` uses
+   `/responses`; `completions` uses `/chat/completions`. One wire API applies to the
+   whole session, including sub-agents. For `engine.model-routing`, inspect AWF's
+   selected endpoint too. `Cannot translate Copilot request feature`,
+   `Unsupported Responses custom tool`, `model_policy_violation`, and
+   `not accessible via the … endpoint` indicate incompatibility, not a transient
+   retry or prompt problem.
+3. **Apply fixes in this order:**
+   1. **Upgrade gh-aw and recompile.**
+   2. **Pin a model that supports the required endpoint.**
+   3. **Remove a conflicting `COPILOT_PROVIDER_WIRE_API` override.**
+   4. **Use sub-agent models from the main model's family.** Verify endpoint
+      compatibility; see [github/gh-aw#67460](https://github.com/github/gh-aw/issues/67460)
+      and [github/copilot-cli#5103](https://github.com/github/copilot-cli/issues/5103).
+   **Switching to an older model is a last resort** if these fail: leading with
+   `model: gpt-4.1` trades capability for a workaround and leaves the
+   underlying misconfiguration in place.
+4. **Check the version field.** `engine.version` is the agent CLI version
+   (Copilot CLI, 1.0.x in the customer case): **Install GitHub Copilot CLI** 404s
+   point here. `sandbox.agent.version` is the AWF release in `vX.Y.Z` form and
+   must match a GitHub release of `github/gh-aw-firewall`: **Install AWF binary**
+   failures point here. There is no `engine.copilot.version` field. Usually remove
+   the misplaced pin and recompile to use the compiled default.
+5. **Read the evidence.** Inspect `[copilot-harness]` alias and
+   `COPILOT_PROVIDER_WIRE_API` lines in `agent-stdio.log`; model and path per request
+   in `sandbox/firewall/logs/api-proxy-logs/token-usage.jsonl`; and `model`,
+   `requested_model`, `cli_version` (gh-aw), `version` (agent CLI), and `awf_version`
+   in `aw_info.json`. Use `gh aw audit RUN_ID` for the combined view. Older runs may
+   omit diagnostics; do not assume routing was correct.
+
 ## Step 4: Commit and Push Changes
 
 Commit the changes, e.g.