Repository navigation
Add ordered engine model fallback using native AWF recovery - #66593
Conversation
Integrate gh-aw-firewall v0.28.44 fallbackModels for same-provider recovery, retain shared harness recovery for cross-provider chains, provision all provider credentials, and record actual-model attribution. Remove the implicit Codex default model. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Remove shared harness model switching, retry adapters, runtime selectors, and their tests. Require native AWF fallback support and reject cross-provider chains until gh-aw-firewall supports them. Preserve native usage attribution and existing single-model harness retries. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Provider validation, normalized fallback handling, Gemini routing, schema documentation, and release classification contain unresolved issues.
Review effort: Balanced
Findings: 1
Open (1)
What changed in this PR
Adds AWF-native ordered model fallback to the workflow compiler, including validation, provider setup, attribution, and documentation.
Changes:
- Compiles concrete same-provider fallback chains into
apiProxy.fallbackModels. - Records the model actually serving fallback requests across usage metadata and telemetry.
- Upgrades AWF to v0.28.44, regenerates 324 lock files, and removes the implicit Codex model default.
| File | Description |
|---|---|
.changeset/ordered-model-fallback.md |
Documents the release and migration. |
.github/aw/actions-lock.json |
Updates pinned action metadata. |
.github/workflows/*.lock.yml (324 files) |
Regenerates workflows with AWF v0.28.44. |
actions/setup/js/generate_footer.cjs |
Attributes footers to fallback models. |
actions/setup/js/handle_noop_message.cjs |
Uses fallback attribution in no-op messages. |
actions/setup/js/messages_footer.cjs |
Updates footer model selection. |
actions/setup/js/model_fallback.cjs |
Implements fallback evidence handling. |
actions/setup/js/model_fallback_usage.test.cjs |
Tests fallback usage attribution. |
actions/setup/js/parse_token_usage.cjs |
Persists actual-model usage metadata. |
actions/setup/js/parse_token_usage.test.cjs |
Tests token-usage integration. |
actions/setup/js/send_otlp_span.cjs |
Adds fallback model telemetry. |
docs/src/content/docs/reference/engines.md |
Documents ordered fallback semantics. |
pkg/actionpins/data/action_pins.json |
Updates AWF action pins. |
pkg/constants/engine_constants.go |
Removes the implicit Codex default. |
pkg/constants/version_constants.go |
Updates AWF versions and compatibility gates. |
pkg/parser/schemas/main_workflow_schema.json |
Adds the fallback frontmatter schema. |
pkg/workflow/awf_config.go |
Models AWF fallback configuration. |
pkg/workflow/awf_config_build.go |
Emits native fallback chains. |
pkg/workflow/awf_env.go |
Updates fallback credential handling. |
pkg/workflow/claude_engine.go |
Integrates provider-aware fallback setup. |
pkg/workflow/codex_engine.go |
Integrates fallback and default changes. |
pkg/workflow/compiler_activation_steps.go |
Adds provider validation steps. |
pkg/workflow/compiler_orchestrator_engine.go |
Adjusts early validation. |
pkg/workflow/compiler_orchestrator_workflow.go |
Validates merged fallback configuration. |
pkg/workflow/copilot_engine.go |
Supports fallback provider credentials. |
pkg/workflow/data/action_pins.json |
Updates compiler action pins. |
pkg/workflow/engine.go |
Parses fallback-models. |
pkg/workflow/engine_fallback_awf.go |
Normalizes and validates AWF fallback chains. |
pkg/workflow/engine_fallback_awf_test.go |
Tests AWF configuration and gates. |
pkg/workflow/engine_fallback_models.go |
Implements provider and pricing resolution. |
pkg/workflow/engine_fallback_models_test.go |
Tests extraction, credentials, and validation. |
pkg/workflow/engine_helpers.go |
Applies fallback provider environment values. |
pkg/workflow/llm_provider.go |
Extends provider inference and credentials. |
pkg/workflow/model_routing.go |
Generalizes model-pattern expansion. |
pkg/workflow/schemas/awf-config.schema.json |
Adds apiProxy.fallbackModels. |
pkg/workflow/strict_mode_env_validation.go |
Validates fallback credential overrides. |
pkg/workflow/testdata/TestWasmGolden_AllEngines/codex.golden |
Updates Codex default output. |
pkg/workflow/threat_detection_steps.go |
Isolates detection attribution. |
pkg/workflow/wasm_golden_test.go |
Handles the empty Codex default. |
pkg/workflow/workflow_builder.go |
Resolves fallback model pricing. |
specs/awf-config-sources-spec.md |
Records the new AWF configuration source. |
|
@copilot address the following outstanding work in one pass:
Push the necessary fixes, reply to each listed review thread and resolve it when addressed. Ignore feedback already answered or resolved. Use the pr-finisher skill and stop when only human review or CI remains; do not trigger CI. Sous-chef head: 8b2e3e3
|
…-fallback # Conflicts: # .github/workflows/blog-auditor.lock.yml Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
@copilot resolve the merge conflicts in this pull request |
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Deploy fallback model helpers with safe outputs and load the telemetry fallback helper before runtime filesystem mocks are installed. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Match the constants regression test to the intentional v0.28.44 firewall upgrade. Keep the exact version assertion. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The branch already included latest
Latest |
Reject unsafe runtime model identifiers with one shared allowlist at telemetry, metadata and footer boundaries. Cover injection payloads, latest-valid selection and normal provider-qualified model names. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
@copilot resolve the merge conflicts in this pull request |
|
@copilot address the following outstanding work in one pass:
Push the necessary fixes, reply to each listed review thread and resolve it when addressed. Ignore feedback already answered or resolved. Use the pr-finisher skill and stop when only human review or CI remains; do not trigger CI. Sous-chef head: 48cbce3
|
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…imes Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Merged the latest |
|
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
/matt |
|
/review |
|
✅ PR Code Quality Reviewer completed the code quality review.
|
|
🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅
|
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
Skills-Based Review 🧠
Reviewed the non-generated diff (4 files: actions-lock.json, agentic_commands.yml, smoke-service-ports.md, smoke-work-queue.md) using /grill-with-docs and /tdd lenses. No blocking issues found in this scope.
📋 Analysis
Scope note
The pre-fetched diff (3000-line cap) only surfaces the lock-file digest bump and two smoke-test workflow additions demonstrating engine.fallback-models; the bulk of the 386-file changeset (324 regenerated .lock.yml files, the Go/JS implementation, and docs/src/content/docs/reference/engines.md) is excluded as generated or truncated. I did not review those files per the filtering rule.
/grill-with-docs
- The new
fallback-modelsconcept is documented indocs/src/content/docs/reference/engines.md(confirmed present, not part of this diff) with clear semantics on which failure classes trigger fallback and the same-provider-only constraint — good alignment between code and docs. smoke-service-ports.mdandsmoke-work-queue.mddemonstrate the feature end-to-end (schemaengine.id+engine.fallback-models+ top-levelmodel), consistent with the documented frontmatter shape.
/tdd
- The two new smoke workflows act as living examples/integration smoke tests for the fallback feature rather than unit tests, which is appropriate given this repo's pattern of
.github/workflows/smoke-*.mdfiles; focused Go/JS tests are reported in the PR description (engine_fallback_models_test.go, etc.) but aren't in the filtered diff to verify directly.
Previously flagged security items
- The GHAS findings on
generate_footer.cjs/model_fallback.cjs/messages_footer.cjs(unescaped model-name interpolation from agent-influenced telemetry) are already marked fixed in commit60b33f192cper existing review threads — no new occurrence in this diff.
Unrelated-looking change
agentic_commands.ymldrops the/ruflo -> ruflo-backed-taskroute. This looks like routing-table drift from workflow regeneration rather than an intentional part of this PR's fallback-models feature; worth a one-line confirmation in the PR description if intentional, otherwise no action needed since it's lock-file-adjacent generated content.
No actionable inline comments were raised — the reviewable surface is small, mechanical, and consistent with documented behavior.
🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · copilot · sonnet50 · 49.5 AIC · ⌖ 14.7 AIC · ⊞ 10.1K
Comment /matt to run again
Comment MemoryPeek at saved memory (pr-code-quality-reviewer)Note This comment is managed by comment memory.Expand the saved memory block to view or edit the persistent context for this thread.
|
There was a problem hiding this comment.
Requesting changes
This PR still has two merge blockers: the new fallback-provider parser does not treat codex/... as the OpenAI backend, so some invalid cross-provider fallback chains can compile and then fail later in AWF; and the branch also removes the existing /ruflo workflow/route even though the change set is about ordered model fallback, which is an unrelated user-facing regression.
Blocking themes
- Fallback validation gap:
engine.fallback-modelsneeds the same provider normalization rules as the rest of the model-routing stack, includingcodex -> openai. - Accidental command removal: dropping
/ruflochanges repository behavior outside this feature and should either be reverted here or handled as an explicit deprecation/removal change.
🔎 Code quality review by PR Code Quality Reviewer · copilot · gpt54 · 128.9 AIC · ⌖ 5.64 AIC · ⊞ 19.7K
Comment /review to run again
| if strings.EqualFold(prefix, "gemini") || strings.EqualFold(prefix, "google") { | ||
| return "gemini" | ||
| } | ||
| if provider, known := llmProviderAliases[strings.ToLower(prefix)]; known { |
There was a problem hiding this comment.
This provider parser never recognizes the codex/ prefix, so engine.fallback-models: [codex/...] is treated as if it were still on the primary provider instead of OpenAI.
💡 Why this blocks the change
engine.model and the universal-LLM paths already accept codex/... as an OpenAI backend, but fallbackModelProvider() only knows about openai, copilot, anthropic, and gemini/google. That means a workflow like engine.id: copilot with fallback-models: [codex/gpt-5] can slip past the new same-provider validation even though it is cross-provider, and nativeAWFFallbackModels() can then emit the literal codex/gpt-5 string into AWF instead of rejecting it up front.
Please normalize codex to the OpenAI provider in the fallback parser and add a regression test that proves codex/... fallbacks are rejected as cross-provider on non-OpenAI engines.
| @@ -1,4 +1,4 @@ | |||
| # gh-aw-commands: {"payload_version":"v1","schema_version":"v1","compiler_version":"dev","commands":["*","ace","approach-validator","archie","cloclo","craft","dependabot-burner","grumpy","matt","mergefest","nit","plan","poem-bot","ponytail","review","ruflo","scout","security-review","smoke-agent-all-merged","smoke-agent-all-none","smoke-agent-public-approved","smoke-agent-public-none","smoke-agent-scoped-approved","smoke-aider","smoke-call-workflow","smoke-checkout-pr-dispatch","smoke-claude","smoke-claude-on-copilot","smoke-codex","smoke-copilot","smoke-copilot-aoai-apikey","smoke-copilot-aoai-entra","smoke-copilot-arm","smoke-copilot-mai","smoke-copilot-sdk","smoke-copilot-small","smoke-create-cross-repo-pr","smoke-crush","smoke-cursor","smoke-deepseek-harness","smoke-drive","smoke-gemini","smoke-github-claude","smoke-goose","smoke-kiro","smoke-multi-pr","smoke-opencode","smoke-otel-backends","smoke-pi","smoke-project","smoke-pydantic","smoke-service-ports","smoke-temporary-id","smoke-test-tools","smoke-update-cross-repo-pr","souschef","squad-plan","summarize","tidy","unbloat","windows"],"workflows":["ace-editor","approach-validator","archie","cloclo","craft","dependabot-burner","design-decision-gate","grumpy-reviewer","mattpocock-skills-reviewer","mergefest","necromancer","pdf-summary","plan","poem-bot","ponytail-reviewer","pr-code-quality-reviewer","pr-nitpick-reviewer","pr-sous-chef","ruflo-backed-task","scout","security-review","skillet","smoke-agent-all-merged","smoke-agent-all-none","smoke-agent-public-approved","smoke-agent-public-none","smoke-agent-scoped-approved","smoke-aider","smoke-call-workflow","smoke-checkout-pr-dispatch","smoke-claude","smoke-claude-on-copilot","smoke-codex","smoke-copilot","smoke-copilot-aoai-apikey","smoke-copilot-aoai-entra","smoke-copilot-arm","smoke-copilot-mai","smoke-copilot-sdk","smoke-copilot-small","smoke-create-cross-repo-pr","smoke-crush","smoke-cursor","smoke-deepseek-harness","smoke-drive","smoke-gemini","smoke-github-claude","smoke-goose","smoke-kiro","smoke-multi-pr","smoke-opencode","smoke-otel-backends","smoke-pi","smoke-project","smoke-pydantic","smoke-service-ports","smoke-temporary-id","smoke-test-tools","smoke-update-cross-repo-pr","squad-plan","test-quality-sentinel","tidy","unbloat-docs","windows"]} | |||
| # gh-aw-commands: {"payload_version":"v1","schema_version":"v1","compiler_version":"dev","commands":["*","ace","approach-validator","archie","cloclo","craft","dependabot-burner","grumpy","matt","mergefest","nit","plan","poem-bot","ponytail","review","scout","security-review","smoke-agent-all-merged","smoke-agent-all-none","smoke-agent-public-approved","smoke-agent-public-none","smoke-agent-scoped-approved","smoke-aider","smoke-call-workflow","smoke-checkout-pr-dispatch","smoke-claude","smoke-claude-on-copilot","smoke-codex","smoke-copilot","smoke-copilot-aoai-apikey","smoke-copilot-aoai-entra","smoke-copilot-arm","smoke-copilot-mai","smoke-copilot-sdk","smoke-copilot-small","smoke-create-cross-repo-pr","smoke-crush","smoke-cursor","smoke-deepseek-harness","smoke-drive","smoke-gemini","smoke-github-claude","smoke-goose","smoke-kiro","smoke-multi-pr","smoke-opencode","smoke-otel-backends","smoke-pi","smoke-project","smoke-pydantic","smoke-service-ports","smoke-temporary-id","smoke-test-tools","smoke-update-cross-repo-pr","souschef","squad-plan","summarize","tidy","unbloat","windows"],"workflows":["ace-editor","approach-validator","archie","cloclo","craft","dependabot-burner","design-decision-gate","grumpy-reviewer","mattpocock-skills-reviewer","mergefest","necromancer","pdf-summary","plan","poem-bot","ponytail-reviewer","pr-code-quality-reviewer","pr-nitpick-reviewer","pr-sous-chef","scout","security-review","skillet","smoke-agent-all-merged","smoke-agent-all-none","smoke-agent-public-approved","smoke-agent-public-none","smoke-agent-scoped-approved","smoke-aider","smoke-call-workflow","smoke-checkout-pr-dispatch","smoke-claude","smoke-claude-on-copilot","smoke-codex","smoke-copilot","smoke-copilot-aoai-apikey","smoke-copilot-aoai-entra","smoke-copilot-arm","smoke-copilot-mai","smoke-copilot-sdk","smoke-copilot-small","smoke-create-cross-repo-pr","smoke-crush","smoke-cursor","smoke-deepseek-harness","smoke-drive","smoke-gemini","smoke-github-claude","smoke-goose","smoke-kiro","smoke-multi-pr","smoke-opencode","smoke-otel-backends","smoke-pi","smoke-project","smoke-pydantic","smoke-service-ports","smoke-temporary-id","smoke-test-tools","smoke-update-cross-repo-pr","squad-plan","test-quality-sentinel","tidy","unbloat-docs","windows"]} | |||
There was a problem hiding this comment.
This PR also deletes the existing /ruflo command and removes its workflow, which is an unrelated behavior regression rather than fallout from ordered model fallback.
💡 Why this blocks the change
On main, /ruflo is a routed issue-comment entry point. This patch drops it from the generated command registry and removes ruflo-backed-task.md/shared/mcp/ruflo.md, so existing automation or docs that rely on /ruflo will just stop working after merge. Nothing in the PR title, description, or changeset advertises a Ruflo retirement/migration, so this reads like accidental drift from rebasing/regeneration rather than an intentional breaking change.
If Ruflo is actually being retired, split that into its own PR with migration notes and release documentation. Otherwise rebase/recompile without deleting the route.
|
🎉 This pull request is included in a new release. Release: |
Why
A model-specific outage currently causes an agent job to fail even when another model on the same endpoint is available. Add ordered alternatives without restarting the agent, replaying its task, or duplicating workflow setup.
Fixes: #64528
Approach
engine.fallback-modelsand compile concrete same-provider chains to AWF'sapiProxy.fallbackModels. AWF owns failure classification, request recovery, model policy, and budget enforcement; gh-aw does not implement harness-level model switching.Constraints and migration
Fallback requires AWF v0.28.31 or newer and a compatible API-proxy image. Unsupported versions, disabled sandboxes, fallback aliases, and cross-provider chains fail compilation instead of silently selecting a harness recovery path. Cross-provider request routing and protocol translation are tracked in github/gh-aw-firewall#9548.
engine.harness.max-retriesremains an ordinary engine execution retry setting; it does not configure AWF's per-request fallback policy. Existing single-model harness behavior is unchanged.Also addresses #63067 by removing gh-aw's implicit Codex
gpt-5.4default. Configureengine.model, a phase-specificGH_AW_MODEL_*_CODEXvariable, orGH_AW_DEFAULT_MODEL_CODEXwhen a fixed model is required. Documentation, a changeset, and the affected golden fixture are updated.Validation
Passed:
make build,make fmt, and the AWFmake build/make recompile/make recompileintegration sequence.make check-workflow-drift: all 324 workflow lock files match their sources.The full publication gate is not green: custom Go linters report existing findings in touched files, and repository-wide TypeScript checking reports an unrelated
create_project.cjs:253type error. Dependency restoration used available@types/node@26.6.3because the manifest's26.6.4was unavailable; no dependency manifests were changed. This PR is intentionally a draft.