Skip to content

Support cross-provider chains in apiProxy.fallbackModels #9548

Closed

Description

@pelikhan

Problem

apiProxy.fallbackModels currently retries requests on the same provider/listener. Provider-qualified entries do not switch the upstream provider or translate its API protocol. A chain such as copilot/grok-4.7 → openai/gpt-5.4 → anthropic/claude-sonnet-4.6 therefore cannot recover entirely inside AWF.

The gh-aw implementation of ordered model fallback (github/gh-aw#64528) uses AWF-native request recovery for concrete same-provider chains, but must restart the agent through its shared harness for cross-provider chains. That risks replaying partial work and loses the main benefit of request-level fallback: preserving the live agent session.

Requested behavior

Allow apiProxy.fallbackModels to contain an ordered, provider-qualified chain spanning configured upstream providers. For example:

{
  "apiProxy": {
    "fallbackModels": [
      "openai/gpt-5.4",
      "anthropic/claude-sonnet-4.6"
    ]
  }
}

When the original request uses GitHub/Copilot inference, AWF should retry eligible failures against those providers in order, using each provider's configured endpoint and credentials. Translate request and response formats where protocols differ, including streaming responses, tool calls, and tool results, while preserving the agent-facing protocol and live session.

Acceptance criteria

  • Switch providers as well as models, without restarting the agent or replaying its task.
  • Validate that every referenced provider has the necessary endpoint, credentials, and protocol support; report actionable configuration errors rather than silently ignoring entries.
  • Preserve configured ordering and prevent automatic model substitution from inserting unconfigured alternatives.
  • Apply model policy, retirement checks, inference budgets, and other proxy guardrails to every candidate, without bypassing a rejection by switching providers.
  • Retain existing failure eligibility: model-specific unavailability, upstream 5xx, and eligible connection errors/timeouts may advance the chain; authentication failures, HTTP 429/quota, generic validation errors, and policy or budget failures must not.
  • Retry only when safe to do so; do not replay an already partially delivered response or duplicate tool execution.
  • Record requested and actual serving provider/model, chain attempt, and originating failure in token-usage and fallback evidence so attribution and cost accounting remain accurate.
  • Preserve existing same-provider and single-model behavior.
  • Add coverage for GitHub/Copilot, OpenAI, and Anthropic chains, protocol/streaming translation, exhausted chains, missing credentials, and guardrail vetoes.

Related work

This request extends the native ordered fallback available in AWF v0.28.44; it is specifically about cross-provider routing and protocol translation, not just stripping provider prefixes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions