Repository navigation
Add model-routing smoke workflows with deterministic runner/proxy evidence checks - #67523
Conversation
Co-authored-by: SivaKesava1 <11771739+SivaKesava1@users.noreply.github.com>
There was a problem hiding this comment.
🟡 Changes recommended
The assertion helper conflates main-agent and sub-agent traffic, allowing required checks to false-pass.
1 open finding
What changed in this PR
Adds deterministic model-routing smoke coverage for Copilot CLI, Pi, and Copilot SDK, with the SDK workflow depending on open PR #67509.
Changes:
- Adds three scheduled and label-triggered routed smoke workflows.
- Adds a shared evidence assertion helper with synthetic tests.
- Documents the
smoke-routingworkflow group.
Validation: Static security review found pinned actions and scoped permissions; compilation/scanners were not run. A focused check reproduced a false-positive assertion.
| File | Description |
|---|---|
DEVGUIDE.md |
Documents routed smoke usage. |
actions/setup/js/smoke_model_routing_assertions.cjs |
Implements evidence checks. |
actions/setup/js/smoke_model_routing_assertions.test.cjs |
Tests assertion scenarios. |
.github/workflows/smoke-copilot-routed.md |
Defines Copilot CLI smoke. |
.github/workflows/smoke-copilot-routed.lock.yml |
Compiled Copilot CLI workflow. |
.github/workflows/smoke-pi-routed.md |
Defines Pi cross-family smoke. |
.github/workflows/smoke-pi-routed.lock.yml |
Compiled Pi workflow. |
.github/workflows/smoke-copilot-sdk-routed.md |
Defines Copilot SDK smoke. |
.github/workflows/smoke-copilot-sdk-routed.lock.yml |
Compiled SDK workflow. |
.github/workflows/agentic_commands.yml |
Registers generated command help metadata. |
🧠 Review effort: Balanced
💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.
| else fail("R2", `expected exactly 1 routing_classification request, observed ${classifierRequests.length}: ${describeRequests(classifierRequests)}`); | ||
|
|
||
| // R3: agent traffic on the selected model and a supported endpoint. | ||
| const agentRequests = requests.filter(request => !isClassifierRequest(request)); |
There was a problem hiding this comment.
Fixed in f652d66: R3/R4/M1 now use only purpose: agent requests, while S3 uses only purpose: subagent requests and correlates shared request IDs when available. Added regressions for purpose confusion, missing purpose, sub-agent-only model use, and request-ID mismatches.
|
@copilot address the following outstanding work in one pass:
Push the necessary fixes, reply to each listed review thread and resolve it when addressed. Ignore feedback already answered or resolved. Use the pr-finisher skill and stop when only human review or CI remains; do not trigger CI. Sous-chef head: bb1124c
|
Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>

None of the 57 existing smoke workflows uses
engine.model-routing, so routing bugs (#67487, #67509, sub-agent name attribution) only showed up in real runs. This PR adds three routed smokes. Each passes or fails on runner- and proxy-written evidence, not on the agent's reply.Workflows (shared
label_command: smoke-routing, plusschedule: every 2 daysandworkflow_dispatch)allowed-modelssmoke-copilot-routedgpt-5.4-mini,gpt-5.6-luna,claude-haiku-4.5smoke-pi-routedclaude-haiku-4.5→/v1/messages,gpt-5.4-mini→/responsessmoke-copilot-sdk-routedgpt-5.6-lunaclaude-haiku-4.5→/chat/completions/responses)job-discriminator: ${{ github.run_id }}.create-issueis configured like the other smokes, with a separateclose-older-keyper workflow.smoke-copilot-sdk-routeddepends on Fix Copilot SDK model routing and cross-family sub-agents #67509 (Copilot SDK providers per wire API). It is expected to fail until Fix Copilot SDK model routing and cross-family sub-agents #67509 merges.Shared helper:
actions/setup/js/smoke_model_routing_assertions.cjsEach workflow calls the helper from an
if: always()post-step. Post-steps run before the usage-artifact step, so the helper builds the unified session itself. It then readsaw_session.jsonl, the api-proxytoken-usage.jsonl(same candidate paths as the #67509 post-step) and/reflectrouting metadata.workflow.inforouting status isselected, and a firewall-provenancefirewall.model_routingselection names a model inallowed-models. An agent-writtenmodel_routing.outcomenever counts.routing_classificationrequest/reflectlists for that model (falls back to the selection's endpoint)allowed-modelsplus the declared sub-agent modelssubagent.started; completed and not failed; a status 200 request on its declared model and endpoint, with the models its events record matching the declared modelModel IDs and endpoints are normalized before comparison. Real proxy logs contain provider-qualified and dated models (
copilot-completions/…,claude-haiku-4-5-20251001) and query-suffixed paths (/v1/messages?beta=true).Every failure prints its check ID and the observed values:
Post-step usage:
Tests
smoke_model_routing_assertions.test.cjsuses syntheticaw_session.jsonlandtoken-usage.jsonlfixtures. It covers:allowed-models, and a sub-agent running on another model;/responseswith status 400;Docs and generated files
DEVGUIDE.mdlists the three workflows and thesmoke-routinglabel, and explains how to run them on a PR: add the label and approve the runs. To run them again, re-add the label, becauselabel_commandremoves it.make recompile.agentic_commands.ymlchanged only to list the new label command.Notes for reviewers
smoke-pi-sub-agentsrun (37735099783):subagent.started,subagent.configuredandsubagent.request, the same names the Copilot SDK uses. That run predatesgh_aw_subagent_result, so it has no completion events. The current pi extension writesgh_aw_subagent_result, which maps tosubagent.completedorsubagent.failed./responses) is inferred from how pi handledgpt-5-miniin that run. If pi picks a different endpoint, S3 will fail and name the observed one.main. Following the scope rule, this PR will not fix failures that also occur onmain.