Skip to content

Correct model-routing attribution for endpoint overrides - #67013

Merged
pelikhan merged 15 commits into
mainfrom
copilot/model-routing-fix-rejected-error
Oct 9, 2026
Merged

pelikhan merged 15 commits into
mainfrom
copilot/model-routing-fix-rejected-error

Conversation

Copilot AI commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Claude runs could use AWF’s selected model successfully through Claude’s compatible endpoint, yet attribution rejected the route because it checked only AWF’s original endpoint. That hid the routed model and effort from workflow outputs, metadata, logs, and footers.

  • Shared endpoint policy: Centralize per-engine endpoints and override rules; keep Copilot strict.
  • Corroborated attribution: Accept an override only when model metadata and proxy request records confirm the effective endpoint. Older harnesses can use /reflect with the same proxy corroboration.
  • Endpoint metadata: Persist both the effective endpoint and AWF’s selected endpoint in routing outcomes and aw_info.json; expose both in Go log metadata.

For example, a Claude route selected on /chat/completions can be attributed through /v1/messages when the proxy confirms the endpoint-only deviation:

{
  "status": "selected",
  "endpoint": "/v1/messages",
  "selected_endpoint": "/chat/completions"
}

@SivaKesava1
SivaKesava1 marked this pull request as ready for review October 8, 2026 22:47
Copilot AI balanced review requested due to automatic review settings October 8, 2026 22:47

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot wasn't able to review any files in this pull request. Check if the Files changed in this pull request are included in default exclusions.

SivaKesava1 and others added 2 commits October 8, 2026 15:47
Co-authored-by: SivaKesava1 <11771739+SivaKesava1@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix routing rejection for Claude-engine runs with AWF Correct model-routing attribution for endpoint overrides Oct 8, 2026
Copilot AI requested a review from SivaKesava1 October 8, 2026 22:57
@gh-aw-bot

Copy link
Copy Markdown
Collaborator

@copilot address the following outstanding work in one pass:

  1. Update this branch with the latest main using make merge-main, resolving any conflicts and preserving the intended changes.
  2. Fix failing check impacted-go-tests (FAILURE): https://github.com/github/gh-aw/actions/runs/37857149403/job/113584089881.
  3. Fix failing check JS Tests (shard 2/4) (FAILURE): https://github.com/github/gh-aw/actions/runs/37857149588/job/113584092524.

Push the necessary fixes, reply to each listed review thread and resolve it when addressed. Ignore feedback already answered or resolved. Use the pr-finisher skill and stop when only human review or CI remains; do not trigger CI.

Sous-chef head: f613a2c
Sous-chef work: 043185828d65a534dd8b2b6b4d00e0d47bf21a2940d3f277c0ed15da7f668648 42e5d572d9d75b7168fe7d0aeda0717cc07e65ed240f6d299e6fec661214f1ac
Sous-chef state: 1ca5d1466df369e383f034cb2545a8a9a09f351d35af326c2116d8f49abd397f

Generated by 👨‍🍳 PR Sous Chef · pi · haiku45 · 3.41 AIC · ⌖ 8.26 AIC · ⊞ 1K · ◷
Comment /souschef to run again

Copilot AI and others added 2 commits October 8, 2026 23:33
Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
@SivaKesava1

Copy link
Copy Markdown
Collaborator

@copilot Tested f613a2c1 in our sandbox: routed workflows compiled at this head with --gh-aw-ref, default AWF. All four runs succeeded:

Run Engine, task aw_info.json model_routing Proxy request records Footer
37859425051 Claude t01 selected, claude-sonnet-5 medium, endpoint: /v1/messages, selected_endpoint: /chat/completions 6/6 as_selected claude · routed: sonnet50 medium ✅
37859433804 Claude t10 selected, claude-opus-5 max, /v1/messages / /chat/completions 26/26 as_selected on /v1/messages claude · routed: opus50 max ✅
37859443458 pi t10 selected, claude-opus-5 max, but endpoint: /chat/completions ❌ 22/22 as_selected on /v1/messages pi · routed: opus50 max ✅
37859452520 Copilot t01 selected, gpt-5.6-luna high, /responses as before copilot · routed: gpt56 high ✅

The core fix works: before this PR, both Claude runs were recorded as rejected / unsupported_endpoint with footer claude · routing rejected.

Two remaining gaps:

  1. pi records the wrong effective endpoint. The pi harness logs inference routing: mode=awf-routed model=claude-opus-5 effort=max endpoint=/v1/messages selected_endpoint=/chat/completions, and the proxy served all 22 requests on /v1/messages. But agent/awf-routing-outcome.json has "endpoint":"/chat/completions","selected_endpoint":"/chat/completions", which aw_info.json then copies. pi_models_json.cjs should pass the endpoint it actually configured (the one it logs) as endpoint. Attribution still says selected only because /chat/completions is in pi's endpoint list, so the corroboration step never compares it with the proxy's upstream_endpoint.
  2. The effective endpoint isn't visible in audit. AwInfoModelRouting gained SelectedEndpoint, but gh aw audit --json for the Claude t01 run still shows only model_routing.endpoint: /chat/completions (from the proxy selection), and never /v1/messages. The PR description says Go log metadata exposes both endpoints. Please surface the effective endpoint from aw_info.json model_routing next to AWF's in the audit and logs model_routing output.

Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>

Copilot AI commented Oct 8, 2026

Copy link
Copy Markdown
Contributor Author

@copilot address the following outstanding work in one pass:

  1. Update this branch with the latest main using make merge-main, resolving any conflicts and preserving the intended changes....

Merged the latest main with make merge-main. Updated the generated logs-jsonl schema and corrected the frontmatter test for the current schema rendering; impacted Go and JS tests, lint, formatting, and the final progress gate passed. Fixes are pushed through 2fd45b7. No review threads were returned to resolve; CI is stale on this head and needs a maintainer re-trigger.

Copilot AI requested a review from gh-aw-bot October 8, 2026 23:58
Copilot AI and others added 2 commits October 9, 2026 00:06
Co-authored-by: SivaKesava1 <11771739+SivaKesava1@users.noreply.github.com>
…ed-error' into copilot/model-routing-fix-rejected-error

# Conflicts:
#	actions/setup/js/work_queue_documentation.test.cjs

Co-authored-by: SivaKesava1 <11771739+SivaKesava1@users.noreply.github.com>

Copilot AI commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor Author

@copilot address the following outstanding work in one pass:

  1. Update this branch with the latest main using make merge-main, resolving any conflicts and preserving the intended changes....

The branch already contained the latest main (e6b87a4) as an ancestor. I regenerated the output schemas to fix the Go test failure and corrected the work-queue assertion for the generated frontmatter; the routing fixes are pushed in afbcd53 (integrated at HEAD 1bad85d). The final local impacted-test gate passes. I did not trigger CI, and there were no review threads available to resolve.

@copilot Tested f613a2c1 in our sandbox: routed workflows compiled at this head with --gh-aw-ref, default AWF. All four runs succeeded:

| Run | Engine, task | aw_info.json model_routing | Pro...

Pi now records the endpoint it actually configures. Audit and structured logs preserve AWF’s endpoint as endpoint and expose the effective and selected endpoints as effective_endpoint and selected_endpoint; the audit text output includes them as well. Regression coverage and generated schemas are updated. Pushed in afbcd53 (HEAD 1bad85d); focused tests and the final local gate pass. CI was not triggered.

@SivaKesava1

Copy link
Copy Markdown
Collaborator

@copilot resolve merge conflicts

@SivaKesava1

Copy link
Copy Markdown
Collaborator

@copilot Re-tested 1bad85da with fresh sandbox runs: Claude t01 37868180194 and pi t10 37868171098.

  • Claude t01 is still correct: aw_info.json has selected, claude-sonnet-5 medium, endpoint: /v1/messages, selected_endpoint: /chat/completions.
  • pi: agent/awf-routing-outcome.json is now correct ("endpoint":"/v1/messages","selected_endpoint":"/chat/completions").

Two gaps remain:

  1. pi's aw_info.json still records the wrong endpoint. model_routing in agent/aw_info.json still has "endpoint":"/chat/completions", although the outcome file and all proxy request records (upstream_endpoint: /v1/messages) say /v1/messages. resolveModelRoutingOutcome apparently keeps AWF's endpoint whenever it's in the engine's list, which is always the case for pi. When the harness outcome reports a different effective endpoint that the proxy's request records corroborate, aw_info.json should record that effective endpoint (and AWF's in selected_endpoint), whether or not AWF's endpoint is also in the list.

  2. Audit's new effective_endpoint never appears for real runs. applyAwInfoModelRouting reads findAwInfoPath(runDir), but in a downloaded run that file comes from the activation/info artifacts: a root aw_info.json with model: agent and no model_routing. The copy the agent job updates exists only in the agent artifact, as agent/aw_info.json. With this build, gh aw audit 37859425051 --json (fresh download folder) still shows only model_routing.endpoint: /chat/completions, and no effective_endpoint/selected_endpoint. Please read the agent's aw_info.json when it exists, and add a test where a run folder contains both copies.

…-fix-rejected-error

# Conflicts:
#	pkg/cli/model_routing.go
#	pkg/cli/model_routing_test.go

Co-authored-by: SivaKesava1 <11771739+SivaKesava1@users.noreply.github.com>

Copilot AI commented Oct 9, 2026

Copy link
Copy Markdown
Contributor Author

@copilot resolve merge conflicts

Resolved the conflicts in model-routing analysis and tests while preserving both the endpoint metadata and main’s per-agent attribution. Merged latest main in commit 563b1fb, now pushed.

Copilot AI and others added 2 commits October 9, 2026 01:34
Co-authored-by: SivaKesava1 <11771739+SivaKesava1@users.noreply.github.com>
Co-authored-by: SivaKesava1 <11771739+SivaKesava1@users.noreply.github.com>

Copilot AI commented Oct 9, 2026

Copy link
Copy Markdown
Contributor Author

@copilot Re-tested 1bad85da with fresh sandbox runs: Claude t01 37868180194 and pi t10 [37868171098](https://github.co...

Pi now adopts the harness-reported endpoint only after model metadata and proxy requests corroborate it, while Copilot remains strict. Added a test covering both root and agent/aw_info.json copies; the agent artifact’s routing metadata is selected. Focused tests and the final impacted validation gate pass. Pushed as 721899ba.

@gh-aw-bot

Copy link
Copy Markdown
Collaborator

@copilot address the following outstanding work in one pass:

  1. Update this branch with the latest main using make merge-main, resolving any conflicts and preserving the intended changes.

Push the necessary fixes, reply to each listed review thread and resolve it when addressed. Ignore feedback already answered or resolved. Use the pr-finisher skill and stop when only human review or CI remains; do not trigger CI.

Sous-chef head: 721899b
Sous-chef work:
Sous-chef state: 44d5df982c77b9b5a9923503efb85682957b240bf3caee5cd830bc18efbca0d6

Generated by 👨‍🍳 PR Sous Chef · pi · haiku45 · 5.01 AIC · ⌖ 7.7 AIC · ⊞ 1K · ◷
Comment /souschef to run again

…-fix-rejected-error

# Conflicts:
#	actions/setup/js/work_queue_documentation.test.cjs

Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>

Copilot AI commented Oct 9, 2026

Copy link
Copy Markdown
Contributor Author

@copilot address the following outstanding work in one pass:

  1. Update this branch with the latest main using make merge-main, resolving any conflicts and preserving the intended changes....

Merged latest main and resolved the conflict in commit 2729cc8. The generated schemas are refreshed and focused plus impacted Go/JS tests pass. make agent-report-progress is blocked only by custom Go lint findings reproduced on main; no CI was triggered.

@SivaKesava1

Copy link
Copy Markdown
Collaborator

Re-tested 2729cc8f:

  • pi is fixed. Run 37878880558 (pi t10, claude-opus-5 max): agent/aw_info.json now has "endpoint":"/v1/messages","selected_endpoint":"/chat/completions", which matches the outcome file and all 28 proxy request records (as_selected, /v1/messages, 200).
  • Claude runs are unchanged and correct (verified on 1bad85da).

The audit display of effective_endpoint still doesn't work on real runs, but the cause is outside this PR. When gh aw audit downloads a run, it flattens the agent artifact into the run folder's root, so agent/aw_info.json becomes <run>/aw_info.json, and the activation/info artifacts' early root aw_info.json (model: agent, no model_routing) overwrites it. The downloaded folder has no agent/ subfolder, so the new findAwInfoPath candidate never matches. With this build, gh aw audit 37868180194 --json still shows only model_routing.endpoint: /chat/completions. We'll file that as a separate issue, so please leave it out of this PR.

@copilot /pr-finisher

Copilot AI commented Oct 9, 2026

Copy link
Copy Markdown
Contributor Author

@copilot address the following outstanding work in one pass:

  1. Update this branch with the latest main using make merge-main, resolving any conflicts and preserving the intended changes....

make merge-main completed successfully. The branch already contains the latest main at c268866; no merge conflicts or additional changes were needed.

@SivaKesava1

Copy link
Copy Markdown
Collaborator

Ready for review. Sandbox-verified: Claude t01/t10 and pi t10 are now attributed selected with the routed model and effort; aw_info.json records the effective endpoint (/v1/messages) and AWF's selected_endpoint (/chat/completions), matching the proxy's request records; footers read routed: sonnet50 medium / routed: opus50 max. Copilot t01 is unchanged. No review threads. Audit's display of the effective endpoint depends on an artifact-download issue we're filing separately.

@SivaKesava1
SivaKesava1 requested a review from pelikhan October 9, 2026 04:58
@pelikhan
pelikhan merged commit 322fcd0 into main Oct 9, 2026
46 checks passed
@pelikhan
pelikhan deleted the copilot/model-routing-fix-rejected-error branch October 9, 2026 05:46
@github-actions

github-actions Bot commented Oct 9, 2026

Copy link
Copy Markdown
Contributor

🎉 This pull request is included in a new release.

Release: v0.91.7

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Model routing: Claude-engine runs recorded as 'routing rejected (unsupported_endpoint)' although AWF routing succeeded

5 participants