Skip to content

Add model-routing smoke workflows with deterministic runner/proxy evidence checks - #67523

Merged
pelikhan merged 4 commits into
mainfrom
copilot/add-model-routing-smoke-workflows
Oct 10, 2026
Merged

pelikhan merged 4 commits into
mainfrom
copilot/add-model-routing-smoke-workflows

Conversation

Copilot AI commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

None of the 57 existing smoke workflows uses engine.model-routing, so routing bugs (#67487, #67509, sub-agent name attribution) only showed up in real runs. This PR adds three routed smokes. Each passes or fails on runner- and proxy-written evidence, not on the agent's reply.

Workflows (shared label_command: smoke-routing, plus schedule: every 2 days and workflow_dispatch)

Workflow Engine allowed-models Sub-agents Extra checks
smoke-copilot-routed Copilot CLI gpt-5.4-mini, gpt-5.6-luna, claude-haiku-4.5 none (CLI uses one wire API per session, gh-aw-firewall#9509) —
smoke-pi-routed pi same claude-haiku-4.5 → /v1/messages, gpt-5.4-mini → /responses S1–S3
smoke-copilot-sdk-routed Copilot SDK gpt-5.6-luna claude-haiku-4.5 → /chat/completions S1–S4, M1 (main on /responses)

Shared helper: actions/setup/js/smoke_model_routing_assertions.cjs

Each workflow calls the helper from an if: always() post-step. Post-steps run before the usage-artifact step, so the helper builds the unified session itself. It then reads aw_session.jsonl, the api-proxy token-usage.jsonl (same candidate paths as the #67509 post-step) and /reflect routing metadata.

ID Check
E1 / E2 Agent execution succeeded; evidence files exist and parse
R1 Runner-written workflow.info routing status is selected, and a firewall-provenance firewall.model_routing selection names a model in allowed-models. An agent-written model_routing.outcome never counts.
R2 Exactly one routing_classification request
R3 A status 200 non-classifier request on the selected model, on an endpoint /reflect lists for that model (falls back to the selection's endpoint)
R4 No non-classifier request on a model outside allowed-models plus the declared sub-agent models
S1–S3 Each declared sub-agent: exactly one subagent.started; completed and not failed; a status 200 request on its declared model and endpoint, with the models its events record matching the declared model
S4 Sub-agent events carry the declared agent name, not a per-call display name
M1 Main-agent requests use the expected endpoint

Model IDs and endpoints are normalized before comparison. Real proxy logs contain provider-qualified and dated models (copilot-completions/…, claude-haiku-4-5-20251001) and query-suffixed paths (/v1/messages?beta=true).

Every failure prints its check ID and the observed values:

FAIL S3 sub-agent haiku-whoami: no 200 request for claude-haiku-4.5 on /chat/completions; observed: /responses 400
FAIL R1 routing not selected: runner-written routing status is missing, expected selected; no firewall model_routing selection record with a selected model (observed 0 selection record(s)); ignored agent-written model_routing.outcome (status=selected)

Post-step usage:

const { main } = require(path.join(process.env.RUNNER_TEMP, "gh-aw/actions/smoke_model_routing_assertions.cjs"));
await main({
  core, engine: "copilot", executionOutcome: process.env.SMOKE_EXECUTION,
  allowedModels: ["gpt-5.6-luna"], mainEndpoint: "/responses",
  subAgents: [{ name: "haiku-whoami", model: "claude-haiku-4.5", endpoint: "/chat/completions" }],
  requireDeclaredAgentNames: true,
});

Tests

smoke_model_routing_assertions.test.cjs uses synthetic aw_session.jsonl and token-usage.jsonl fixtures. It covers:

  • a passing case for each variant;
  • wrong model: selected model outside allowed-models, and a sub-agent running on another model;
  • wrong endpoint, including the SDK sub-agent whose only request is /responses with status 400;
  • non-200 status;
  • missing selection event;
  • harness-only routing outcome;
  • classifier count of 0 or 2;
  • missing or broken evidence files;
  • an end-to-end case that builds the session from raw runner and proxy files.

Docs and generated files

  • DEVGUIDE.md lists the three workflows and the smoke-routing label, and explains how to run them on a PR: add the label and approve the runs. To run them again, re-add the label, because label_command removes it.
  • Lock files were generated by make recompile. agentic_commands.yml changed only to list the new label command.

Notes for reviewers

  • The pi event names come from a real smoke-pi-sub-agents run (37735099783): subagent.started, subagent.configured and subagent.request, the same names the Copilot SDK uses. That run predates gh_aw_subagent_result, so it has no completion events. The current pi extension writes gh_aw_subagent_result, which maps to subagent.completed or subagent.failed.
  • The pi GPT sub-agent's endpoint (/responses) is inferred from how pi handled gpt-5-mini in that run. If pi picks a different endpoint, S3 will fail and name the observed one.
  • These workflows have not run in CI yet, so no checks have been compared against main. Following the scope rule, this PR will not fix failures that also occur on main.
  • Routing, harness and AWF behavior are unchanged.

Co-authored-by: SivaKesava1 <11771739+SivaKesava1@users.noreply.github.com>
@SivaKesava1
SivaKesava1 marked this pull request as ready for review October 10, 2026 23:00
Copilot AI balanced review requested due to automatic review settings October 10, 2026 23:00
Copilot AI changed the title [WIP] Add model-routing smoke workflows with deterministic checks Add model-routing smoke workflows with deterministic runner/proxy evidence checks Oct 10, 2026
Copilot AI requested a review from SivaKesava1 October 10, 2026 23:07

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The assertion helper conflates main-agent and sub-agent traffic, allowing required checks to false-pass.

1 open finding
What changed in this PR

Adds deterministic model-routing smoke coverage for Copilot CLI, Pi, and Copilot SDK, with the SDK workflow depending on open PR #67509.

Changes:

  • Adds three scheduled and label-triggered routed smoke workflows.
  • Adds a shared evidence assertion helper with synthetic tests.
  • Documents the smoke-routing workflow group.

Validation: Static security review found pinned actions and scoped permissions; compilation/scanners were not run. A focused check reproduced a false-positive assertion.

File Description
DEVGUIDE.md Documents routed smoke usage.
actions/​setup/​js/​smoke_model_routing_assertions.cjs Implements evidence checks.
actions/​setup/​js/​smoke_model_routing_assertions.test.cjs Tests assertion scenarios.
.github/​workflows/​smoke-copilot-routed.md Defines Copilot CLI smoke.
.github/​workflows/​smoke-copilot-routed.lock.yml Compiled Copilot CLI workflow.
.github/​workflows/​smoke-pi-routed.md Defines Pi cross-family smoke.
.github/​workflows/​smoke-pi-routed.lock.yml Compiled Pi workflow.
.github/​workflows/​smoke-copilot-sdk-routed.md Defines Copilot SDK smoke.
.github/​workflows/​smoke-copilot-sdk-routed.lock.yml Compiled SDK workflow.
.github/​workflows/​agentic_commands.yml Registers generated command help metadata.

🧠 Review effort: Balanced


💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

else fail("R2", `expected exactly 1 routing_classification request, observed ${classifierRequests.length}: ${describeRequests(classifierRequests)}`);

// R3: agent traffic on the selected model and a supported endpoint.
const agentRequests = requests.filter(request => !isClassifierRequest(request));

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in f652d66: R3/R4/M1 now use only purpose: agent requests, while S3 uses only purpose: subagent requests and correlates shared request IDs when available. Added regressions for purpose confusion, missing purpose, sub-agent-only model use, and request-ID mismatches.

@gh-aw-bot

Copy link
Copy Markdown
Collaborator

@copilot address the following outstanding work in one pass:

  1. Update this branch with the latest main using make merge-main, resolving any conflicts and preserving the intended changes.
  2. Review (actions/setup/js/smoke_model_routing_assertions.cjs:180): This treats every non-classifier record as interchangeable traffic, even though token-usage distinguishes purpose: agent from purpose: subagent (and missing purpose is unknown). As a result, sub-agent-only traffic can satisfy R3, a main-agent call on a declared sub-agent model can evade R4, and a main-agent call can later satisfy S3; a focused check confirms R3 currently passes with no purpose: agent request. Partition requests by purpose, use only main-agent traffic for R3/M1/R4, and only sub-agent traffic (preferably request-ID-correlated) for S3, with regression fixtures for these false-pass cases. - Add model-routing smoke workflows with deterministic runner/proxy evidence checks #67523 (comment)

Push the necessary fixes, reply to each listed review thread and resolve it when addressed. Ignore feedback already answered or resolved. Use the pr-finisher skill and stop when only human review or CI remains; do not trigger CI.

Sous-chef head: bb1124c
Sous-chef work: 7357eea04c7797e82b1be6bb85a956c64a76c899ebf211b193b7e6d4ce6359ed
Sous-chef state: fd3850f61ee8f3a8cf8bb91c10c48589c56b246737d6739e1ddee44cc1ed0b60

Generated by 👨‍🍳 PR Sous Chef · pi · haiku45 · 5.27 AIC · ⌖ 8.47 AIC · ⊞ 1K · ◷
Comment /souschef to run again

Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
@pelikhan
pelikhan merged commit feb7220 into main Oct 10, 2026
2 of 3 checks passed
@pelikhan
pelikhan deleted the copilot/add-model-routing-smoke-workflows branch October 10, 2026 23:48
Copilot AI restored the copilot/add-model-routing-smoke-workflows branch October 10, 2026 23:49
Copilot AI requested a review from gh-aw-bot October 10, 2026 23:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add model-routing smoke workflows with deterministic checks (routed Copilot CLI, routed pi, routed Copilot SDK)

5 participants