Skip to content

AI Moderator agent job fails on every run: pi + copilot/auto gets 400 model_not_supported, misreported as generic infrastructure_error #63113

Closed

Description

@benissimo

Summary

Since ai-moderator.md moved to engine: pi / model: copilot/auto (144a253, #59695, 2026-09-09), AI Moderator's agent job has not succeeded in the run history I could see. The Pi provider sends model: "auto" to the api-proxy (POST http://api-proxy:10002/chat/completions), which rejects it:

provider_error provider=aw-gateway model=auto api=openai-completions ... error="400: {"message":"The requested model is not supported.","code":"model_not_supported","param":"model","type":"invalid_request_error"}"

The agent then emits report_incomplete / infrastructure_error ("All 1 Pi provider requests failed before safe outputs were emitted").

Numbers (AI Moderator, last 1,000 runs, 2026-09-15 → 2026-09-24)

  • 260 failure, 107 success, 374 skipped, 259 action_required.
  • 0 of the 107 success runs executed the agent job (all skipped at activation), so every run that reached the agent failed.
  • model_not_supported found in the failed-job log for 5 of 7 sampled failures (including the newest and oldest in the window); the other 2 had no retrievable failed-step log.
  • Same symptom in other engine: pi + model: copilot/auto workflows, e.g. duplicate-code-detector (17 of its last 20 runs failed; sampled failure 35924730146 shows model_not_supported).

Example run: https://github.com/github/gh-aw/actions/runs/35917931650

Why it isn't getting fixed: detection gap

The conclusion job's classifier logs Model not supported error: false for these runs even though the agent log contains model_not_supported. So they're filed as generic "[aw] AI Moderator reported incomplete result" issues (#62169, #62364, #62543, #62698, #62912, now #63068). Those accumulate bot-only comments, expire, and close as NOT_PLANNED, and no human sees the actual cause.

Prior art

#60867 found the same model_not_supported spike on Issue Monster (Pi resolving to aw-gateway/auto, 60 consecutive failures). It recommended an engine-level fix (fail fast / never send bare auto to the gateway) to be filed against the Pi model-resolution path. I couldn't find that follow-up issue.

Possibly separate

#63068's last agent output also shows Extension error (.../pi_steering_extension.cjs): Cannot read properties of undefined (reading 'steer').

Ask

  1. Pin AI Moderator (and the other pi + copilot/auto workflows) to a concrete model, or make the gateway accept/resolve auto for the Pi provider.
  2. Have the conclusion classifier recognize model_not_supported in Pi provider errors, so these runs are reported as model errors rather than generic incompletes.

Activity

  1. locked and limited conversation to collaborators on Sep 24, 2026
  2. unlocked this conversation on Sep 24, 2026
  3. lpcox commented on Sep 29, 2026

    @lpcox
    Collaborator

    🔗 AWF tracking issue: github/gh-aw-firewall#9181

    Generated by Firewall Issue Dispatcher · copilot · auto · 23.2 AIC · ⊞ 9.1K · ◷

  4. pelikhan commented on Oct 1, 2026

    @pelikhan
    Collaborator

    @copilot ensure that we have a pi + auto and pi + copilot/auto representatives int he agentic workflows. pi should default to auto mostly.

  5. pelikhan commented on Oct 7, 2026

    @pelikhan
    Collaborator

    Status (2026-10-07; Maintainer direction): The issue reports model_not_supported failures for Pi with copilot/auto, and the October 1 maintainer comment requests representative workflows for both pi + auto and pi + copilot/auto. The next step is to add those representatives and check model resolution and error classification against the reported failures.

  6. added
    bugSomething isn't working
    cluster:model-compatibilityModel selection, alias resolution, provider inventory, and runtime compatibility
    on Oct 14, 2026
  7. pelikhan commented on Oct 16, 2026

    @pelikhan
    Collaborator

    Current-state refresher (2026-10-16): Apparently resolved for the original Pi/auto regression; current Pi health has separate failures.

    Merged #66567 accepts the Copilot auto sentinel (implementation) and adds both representative paths: bare auto and copilot/auto. Upstream #9195 fixed forwarding bare auto; AI Moderator now pins a concrete model. Pi parser regressions cover provider model_not_supported errors.

    There is actual post-fix evidence: bare-auto run 38099456639 completed agent and safe_outputs successfully. Newer failures should not be conflated with the original issue: 38660741718 failed starting MCP Gateway on an undefined environment variable; 38901157923 reached successful agent/detection jobs but concluded with missing_terminal_safe_output, not the reported 400.

    This combines static inspection with existing run/job/log reads; no new run was dispatched. A fresh same-cause reproduction would justify retaining the original regression rather than tracking these distinct failures.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

awfbugSomething isn't workingcluster:model-compatibilityModel selection, alias resolution, provider inventory, and runtime compatibilitycommunity

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions