Skip to content

AI Moderator agent job fails on every run: pi + copilot/auto gets 400 model_not_supported, misreported as generic infrastructure_error #63113

Description

@benissimo

Summary

Since ai-moderator.md moved to engine: pi / model: copilot/auto (144a253, #59695, 2026-09-09), AI Moderator's agent job has not succeeded in the run history I could see. The Pi provider sends model: "auto" to the api-proxy (POST http://api-proxy:10002/chat/completions), which rejects it:

provider_error provider=aw-gateway model=auto api=openai-completions ... error="400: {"message":"The requested model is not supported.","code":"model_not_supported","param":"model","type":"invalid_request_error"}"

The agent then emits report_incomplete / infrastructure_error ("All 1 Pi provider requests failed before safe outputs were emitted").

Numbers (AI Moderator, last 1,000 runs, 2026-09-15 → 2026-09-24)

  • 260 failure, 107 success, 374 skipped, 259 action_required.
  • 0 of the 107 success runs executed the agent job (all skipped at activation), so every run that reached the agent failed.
  • model_not_supported found in the failed-job log for 5 of 7 sampled failures (including the newest and oldest in the window); the other 2 had no retrievable failed-step log.
  • Same symptom in other engine: pi + model: copilot/auto workflows, e.g. duplicate-code-detector (17 of its last 20 runs failed; sampled failure 35924730146 shows model_not_supported).

Example run: https://github.com/github/gh-aw/actions/runs/35917931650

Why it isn't getting fixed: detection gap

The conclusion job's classifier logs Model not supported error: false for these runs even though the agent log contains model_not_supported. So they're filed as generic "[aw] AI Moderator reported incomplete result" issues (#62169, #62364, #62543, #62698, #62912, now #63068). Those accumulate bot-only comments, expire, and close as NOT_PLANNED, and no human sees the actual cause.

Prior art

#60867 found the same model_not_supported spike on Issue Monster (Pi resolving to aw-gateway/auto, 60 consecutive failures). It recommended an engine-level fix (fail fast / never send bare auto to the gateway) to be filed against the Pi model-resolution path. I couldn't find that follow-up issue.

Possibly separate

#63068's last agent output also shows Extension error (.../pi_steering_extension.cjs): Cannot read properties of undefined (reading 'steer').

Ask

  1. Pin AI Moderator (and the other pi + copilot/auto workflows) to a concrete model, or make the gateway accept/resolve auto for the Pi provider.
  2. Have the conclusion classifier recognize model_not_supported in Pi provider errors, so these runs are reported as model errors rather than generic incompletes.

Activity

  1. locked and limited conversation to collaborators on Sep 24, 2026
  2. unlocked this conversation on Sep 24, 2026
  3. lpcox commented on Sep 29, 2026

    @lpcox
    Collaborator

    🔗 AWF tracking issue: github/gh-aw-firewall#9181

    Generated by Firewall Issue Dispatcher · copilot · auto · 23.2 AIC · ⊞ 9.1K · ◷

  4. pelikhan commented on Oct 1, 2026

    @pelikhan
    Collaborator

    @copilot ensure that we have a pi + auto and pi + copilot/auto representatives int he agentic workflows. pi should default to auto mostly.

  5. pelikhan commented on Oct 7, 2026

    @pelikhan
    Collaborator

    Status (2026-10-07; Maintainer direction): The issue reports model_not_supported failures for Pi with copilot/auto, and the October 1 maintainer comment requests representative workflows for both pi + auto and pi + copilot/auto. The next step is to add those representatives and check model resolution and error classification against the reported failures.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions