Model routing: show AWF routing decisions in gh aw logs and gh aw audit #66638

Summary

Workflows that use engine.model-routing now write AWF's routing decisions to sandbox/firewall/logs/api-proxy-logs/model-routing.jsonl, next to token-usage.jsonl. AWF added this file in gh-aw-firewall#9451 (first released in v0.28.35), and gh-aw's default AWF version is v0.28.37, so every routed workflow produces it. The agent artifact already uploads the whole firewall logs folder, but gh aw logs and gh aw audit never read the file. On main (679cb02), nothing under pkg/cli refers to model-routing.jsonl, model_routing, or routing_classification.

So for a routed run, audit shows tokens and credits, but it doesn't say:

• what the router classified the task as;
• which model and effort it chose, or which router version made the choice;
• whether the agent and its sub-agents actually used that model;
• how much of the cost went to the routing classifier itself.

The model routing guide (reference/model-routing.md, added in #66338) says that gh aw logs and gh aw audit "do not yet provide a dedicated routing summary", tells authors to read the JSONL files themselves, and promises CLI support later. This issue proposes that support.

Evidence: run 37548242023

I audited https://github.com/githubnext/gh-aw-routing-sandbox/actions/runs/37548242023 with gh aw audit 37548242023. It's a private test repository, with workflow Routing test: t01-explain-trivial, AWF v0.28.37, and router gh-aw-router 0.1.3. Then I compared the report with the run's model-routing.jsonl.

What audit reports:

• The engine is engine=copilot/auto/v1.0.90.
• Usage is tokens: in=95.5k out=941 cache_read=70.0k reqs=5 and aic=0.76.
• The behavior fingerprint is adaptive/moderate/selective_write/moderate/standalone.
• The comparison line is comparison: stable vs baseline 37546193453 | No action needed; this run matches the selected successful baseline closely.
• The text report has no routing section. In audit --json, model-routing.jsonl appears only in the list of downloaded files, and the step name "Prepare model-routing conversation" appears only in the job list. Neither audit.json nor run_summary.json has any routing field.

What model-routing.jsonl records for the same run (6 records):

• 1 classification record: the classifier was github-copilot/gpt-5.6-luna at effort high, attempt 1.
• 1 selection record:
  • The objective was cost/auto and the provider was copilot.
  • The labels were task_type: explain, scope: local, and task_complexity: trivial, with mode economy.
  • The router selected github-copilot/gpt-5.6-luna at effort high (selected_id: choice-0017), with endpoint /responses.
  • There were 34 ranked choices, 34 eligible choices, and a catalogue overlap of 34. Classification wasn't degraded, and selection took 4.9 s.
  • The record also has conversation_sha256, interaction_id: 37548242023-1, github_repository, and github_workflow_ref.
• 4 request records: all had routed: as_selected and outcome: completed with status 200. All 4 request_ids match rows in token-usage.jsonl.

What audit gets wrong or leaves out:

1. Classifier cost is counted in the agent's totals. token-usage.jsonl has 5 rows, while model-routing.jsonl has only 4 requests. The extra row has purpose: "routing_classification" and cost 0.059 AIC. audit counts it in reqs=5 and aic=0.76, so about 8% of this run's cost was routing overhead, and the report doesn't show it.

2. The "stable" baseline used a different route. The baseline 37546193453 had the same labels (explain/local/trivial, economy), but it ran on AWF v0.28.35 with router 0.1.2 and selected gpt-5.6-luna at medium. This run used AWF v0.28.37 and router 0.1.3 and selected gpt-5.6-luna at high. Credits were about the same (0.76), so "stable" is fair for cost, but the report doesn't mention that the router version and the selected effort changed. Baseline selection in pkg/cli/audit_comparison.go scores runs by behavior fingerprint only. It knows nothing about the route.

3. The route mix across runs is lost. audit downloaded nine baselines of the same routing-test family. Their routing logs show that the selected model drives the cost:

   Labels (type / complexity)	Mode	Selected	AIC
   explain / trivial	economy	gpt-5.6-luna:medium	0.72 to 0.76
   chore / easy	economy	gpt-5.6-luna:high	1.87
   fix / hard	robust	gpt-5.6-sol:xhigh	41.8
   plan / hard	balanced	gpt-5.6-sol:high	48.7
   implement / medium	robust	gpt-5.6-sol:max	89.5
   implement / medium	robust	claude-opus-5:high	462
   plan / hard	robust	claude-opus-5:max	618
   plan / hard	balanced	claude-opus-5:xhigh	933

   None of this appears in audit or logs. You can only get it by reading the JSONL files by hand.

4. Deviations are hidden, and old records report them incorrectly. In the three Claude baselines (AWF v0.28.35), every request was logged as routed: deviated.

   • Most of these deviations are only ["endpoint"], because AWF before v0.28.39 counted the endpoint difference as a deviation (gh-aw-firewall#9505, fixed by gh-aw-firewall#9507).
   • Some are real effort deviations: ["effort","endpoint"], where claude-opus-5 ran at low instead of the selected max or xhigh. Examples are run 37402330643 (62 of 95 requests) and run 37505153056 (15 of 101).
   • audit doesn't show either kind of deviation.

Record format

AWF's docs/api-proxy-sidecar.md (section "Model routing audit log") is the source of truth, and gh-aw should read the file loosely rather than copy the schema. The fields observed in these runs are listed below. Some names differ from the earlier planning notes.

• Every record: _schema (for example model-routing/v0.28.37), timestamp, event: "model_routing", and stage.
• classification: purpose, attempt, classifier_model, and classifier_effort.
• selection:
  • the configuration: objective (goal, mode) and provider;
  • the classification: labels (task_type, scope, task_complexity), mode, classifier_model, classifier_effort, classifier_attempts, degraded_classification, and degraded_reason;
  • the choice: selected_id, selected_provider, selected_model, selected_effort, wire_model, and endpoint;
  • the candidates: ranked_choices, eligible_choices, and catalogue_overlap;
  • the router: router (name, version) and latency_ms;
  • the context: conversation_sha256, interaction_id, github_repository, and github_workflow_ref.
• failure: a code and detail.
• request:
  • routed (as_selected, deviated, or unobserved), deviations, and unavailable;
  • method, pathname, and provider;
  • the requested_* and selected_* model, effort, provider, and endpoint;
  • request_id, outcome (completed, rejected, failed, or aborted), and the HTTP status.

Since AWF v0.28.43 (gh-aw-firewall#9523), request records also have requested_endpoint and upstream_endpoint, which differ when the proxy translated the request between Responses and Chat Completions. The file contains no prompt or conversation text.

Proposed plan

1. Find and parse the routing log.

   • Find model-routing.jsonl next to token-usage.jsonl, reusing the search in pkg/cli/token_usage_find.go, which already covers the sandbox/firewall/logs and sandbox/firewall/audit layouts.
   • Parse it leniently: ignore unknown fields and stages, and accept older schema versions.
   • If the file is missing, report "not routed" (or "routing log not available" when the AWF config enabled routing). Don't treat it as an error.

2. Add a routing section to the run summary and audit report.

   • Add it to RunSummary in pkg/cli/logs_models.go and to the report in pkg/cli/audit_report.go, in both text and JSON output.
   • Show the status (selected, failed, or not routed) and the objective.
   • Show the labels and mode, the classifier model, effort, and attempts, and whether classification was degraded and why.
   • Show the selected model, effort, and endpoint, the top three ranked choices, the router name and version, and the selection latency.
   • Show request counts by routed and outcome, and list the models and efforts that deviated.
   • On failure, show the failure code and detail, so an exit code 78 can be explained from the report.

3. Split the cost. Join request records to token-usage.jsonl by request_id and split credits and tokens into three parts:

   • the classifier, from rows with purpose: "routing_classification";
   • traffic that used the selected model;
   • traffic that deviated from it, such as sub-agents or overrides.

   This shows how much of a run's cost was the classifier, the routed model, and everything else. When reading records from AWF before v0.28.39, count a request whose only deviation is endpoint as using the selected model, and say so in the output.

4. Use the route in comparisons.

   • Add the selected model, effort, mode, and router version to the run's comparison data.
   • When the current run and the baseline selected a different model, effort, or router version, say so in the comparison line, even if the cost is about the same. Don't call two runs "stable" without noting that they ran on different routes.
   • Optionally, prefer baselines with the same labels.

5. Show the route mix across runs in gh aw logs. Aggregate the routes across the downloaded runs and show which labels led to which selection, with run counts and total and average credits for each selection. Also show the total classifier cost and the share of traffic that deviated.

6. Keep the routing log in the fallback artifact.

   • Add model-routing.jsonl to the small agent-output-fallback artifact. Today that artifact carries only the three token-usage.jsonl path variants (pkg/workflow/compiler_yaml_artifacts.go), using the same ARC/DinD path rewrite. Then routing data is still available when the agent artifact is missing or too large.
   • aw-prompts/user.txt is already in the agent artifact (Include split prompts in the unified agent session #66011). The routing conversation is that text sent as a single user message, so audit can recompute the SHA-256 and check it against the selection's conversation_sha256. This confirms the router saw the expected prompt without AWF storing any prompt text.

7. Add tests.

   • Parser tests with fixture files for these cases:
     • a selection;
     • a degraded classification;
     • a failure record;
     • no file;
     • an older AWF v0.28.35 Claude selection where only the endpoint deviated;
     • a real effort deviation;
     • a log with unknown fields.
   • Golden output tests for logs and audit in text and JSON, for both routed and non-routed runs.
   • A comparison test where the route differs but the cost doesn't.
   • Compiler snapshot tests for the fallback artifact path list.

8. Update the docs.

   • reference/model-routing.md:
     • Replace the Observability paragraph that says gh aw logs and gh aw audit "do not yet provide a dedicated routing summary" and tells authors to read the JSONL files themselves. Describe instead the routing section in audit, the cost split (classifier, selected model, deviated), the route-aware comparison, and the route mix in logs, with a short example of the report.
     • Keep the link to AWF's schema documentation, for authors who need the full field list.
     • Update the troubleshooting rows (exit code 78 / failure, degraded_reason, routed: "deviated") to say where audit now shows each one.
     • Explain how audit treats endpoint-only deviations in records from AWF before v0.28.39.
   • reference/audit.md:
     • Add the routing section to the list of report sections for gh aw audit <run-id-or-url>.
     • Add the new routing fields to "JSON output schemas".
     • Explain that the comparison now notes route changes.
     • Under "gh aw logs --format <fmt>", describe the route mix in the logs output.
   • reference/cost-management.md:
     • In "Monitoring Costs with gh aw logs", explain how to read credits per selection and the split between the classifier, the selected model, and deviated traffic for routed runs.
     • Add a link to the model routing page from "AI Credits with Dynamic Model Selectors".
   • reference/artifacts.md:
     • List model-routing.jsonl next to token-usage.jsonl in the API proxy log directory listing, and add its schema to the JSON Schemas table, linking to AWF's docs if AWF doesn't publish a schema file.
     • List it in the agent-output-fallback contents once step 6 lands.
     • Fix the existing claim that token-usage.jsonl is only in a separate firewall-audit-logs artifact. This run uploaded only activation, agent, agent-output-fallback, usage, safe-outputs-items, and info. Both JSONL files came from the agent artifact at sandbox/firewall/logs/api-proxy-logs/.

Out of scope

• Changing AWF's record format or the router's selection behavior.
• Adding routing to the per-run $GITHUB_STEP_SUMMARY token table. That can be a follow-up issue.

Related

• Fixed issues: Model routing: the routing-conversation step reads a prompt path that doesn't exist in the agent job #66221/Fix model routing to read the agent-job task prompt #66290 (routing reads user.txt), Model routing: routed workflows run stale AWF and router images regardless of the AWF version #66226/Route model-routing workflows to version-matched, pinned images #66291 (current images and the AWF v0.28.37 default), gh-aw-firewall#9505/[aw] agentic workflows out of sync #9507 (endpoint-only deviations), and Docs: add a model routing guide for engine.model-routing #66337/Docs: add experimental Copilot model-routing guide #66338 (the routing guide that defers CLI support to this issue).
• gh-aw-firewall#9447/Purge orphaned .campaign.g.md files and their lock files with --purge flag #9451 added the model-routing.jsonl log in AWF.