Prevent Codex exec tool rejection with Copilot inference by Copilot · Pull Request #65654 · github/gh-aw

Copilot · 2026-10-04T20:29:52Z

Codex runs using the Copilot-compatible endpoint fail when Codex submits its unsupported exec custom tool. Configure Codex to omit that tool for GitHub inference while preserving OpenAI behavior.

Provider-specific config: Generated Copilot config sets features.shell_tool = false; overrides that re-enable it are rejected.
Example generated config:
```
[features]
shell_tool = false
```

Fixes [AW Top 10] 01 Fix Codex failures from unsupported exec custom tool #65483

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

github-actions · 2026-10-04T20:48:39Z

🔍 Design Decision Gate 🏗️ is checking for design decision records on this pull request...

github-actions · 2026-10-04T20:48:45Z

✅ PR Code Quality Reviewer completed the code quality review.

🔎 Code quality review by PR Code Quality Reviewer

github-actions · 2026-10-04T20:48:45Z

🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer

Copilot

Copilot review overview

🟡 Changes recommended

CLI arguments can bypass the new restriction, and the unrelated schema change breaks supported queue expressions.

Review effort: Balanced
Findings: 1 · 1

Open (2)

Restore expression support for concurrency.queue schema · New
Prevent engine.args from re-enabling the unsupported exec tool · New

What changed in this PR

Configures Codex to disable its unsupported exec tool when using GitHub inference while preserving OpenAI behavior.

Changes:

Adds provider-specific shell_tool configuration and override validation.
Adds regression tests for GitHub and OpenAI providers.
Narrows concurrency.queue schema validation.

File	Description
`pkg/workflow/codex_config.go`	Adds GitHub-specific Codex configuration.
`pkg/workflow/codex_config_test.go`	Tests generated provider configurations.
`pkg/workflow/schemas/github-workflow.json`	Removes expression support from `queue`.

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

+          "type": "string",
+          "enum": ["single", "max"],


+	if githubProvider {
+		if features, ok := config.Overrides["features"].(map[string]any); ok {
+			if shellTool, exists := features["shell_tool"]; exists && shellTool != false {
+				return errors.New("engine.config: features.shell_tool is disabled for GitHub inference because the Copilot compatibility adapter does not support the Codex exec custom tool")


github-actions · 2026-10-04T20:48:57Z

✅ Test Quality Sentinel completed test quality analysis.

Test Quality Sentinel skipped because pre-fetch PR data was unavailable: unable to fetch test file diff

🧪 Test quality analysis by Test Quality Sentinel

github-actions · 2026-10-04T20:49:09Z

✅ Ponytail Reviewer completed successfully!

Lean already. Ship.

Generated by Ponytail Reviewer for #65654

github-actions

Skills-Based Review 🧠

Applied /diagnosing-bugs — requesting changes. The core Codex/Copilot fix (gating features.shell_tool by provider) is well targeted and tested, but two issues need fixing before merge.

📋 Key Themes & Highlights

Key Themes

Incomplete bypass closure: validateCodexManagedConfig only inspects TOML engine.config overrides, but engine.args (-c features.shell_tool=true) are appended last in buildCodexCommand and win over the generated flag, so the guard can be silently bypassed.
Unrelated schema regression: the concurrency.queue schema change drops expressionSyntax support, which breaks the existing TestConcurrencyQueueSchemaValidation expression case (verified locally — test fails). This looks like an accidental bundling, unrelated to the Codex/Copilot fix described in the PR body.

Positive Highlights

✅ Clean, minimal provider-gated config generation (ResolveLLMProvider(workflowData) == LLMProviderGitHub).
✅ Good regression tests added for both the default-disabled and rejected-override cases (TestCodexGitHubInferenceDisablesUnsupportedExecTool, TestCodexGitHubInferenceRejectsExecToolOverride).
✅ Clear, actionable error message pointing at the root cause (Copilot adapter limitation).

Both existing reviewer comments on this PR (TOML-only validation gap, and the concurrency schema regression) are confirmed valid — see inline comments.

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · copilot · sonnet50 · 32.1 AIC · ⌖ 14.6 AIC · ⊞ 10K
_{Comment /matt to run again}

github-actions · 2026-10-04T20:53:56Z

-            }
-          ],
+          "type": "string",
+          "enum": ["single", "max"],


[/diagnosing-bugs] Confirmed by running go test ./pkg/workflow/ -run TestConcurrencyQueueSchemaValidation: this change makes the existing test fail, because queue with a GitHub Actions expression (e.g. ${{ github.event_name == 'pull_request' && 'single' || 'max' }}) is a documented, tested case that the compiler generates (see concurrency_command_test.go's mixed-PR scenario). Dropping the expressionSyntax branch regresses that feature, not just a theoretical risk.

💡 Test failure output

--- FAIL: TestConcurrencyQueueSchemaValidation (0.01s) Error: GitHub Actions schema validation failed ... '/concurrency/queue': value must be one of 'single', 'max'

Restore the oneOf with expressionSyntax for queue, or update/remove TestConcurrencyQueueSchemaValidation's expression case if dropping expression support for queue is intentional — but as-is this is an unrelated regression bundled into a Codex-provider fix PR.

@copilot please address this.

github-actions · 2026-10-04T20:53:56Z

+	if githubProvider {
+		if features, ok := config.Overrides["features"].(map[string]any); ok {
+			if shellTool, exists := features["shell_tool"]; exists && shellTool != false {
+				return errors.New("engine.config: features.shell_tool is disabled for GitHub inference because the Copilot compatibility adapter does not support the Codex exec custom tool")


[/diagnosing-bugs] Root cause only partially addressed: validateCodexManagedConfig only blocks engine.config TOML overrides. But buildCodexCommand appends workflowData.EngineConfig.Args (customArgsParam) as the last segment of the generated codex exec ... command, after shellToolParam ( -c features.shell_tool=false). Since later -c flags win in Codex CLI arg parsing, a workflow with engine: {id: codex, args: ["-c", "features.shell_tool=true"]} silently re-enables the exec tool for GitHub inference, bypassing the new guard entirely.

💡 Suggested fix

Add a check (alongside/near validateCodexManagedConfig, or in buildCodexCommand) that scans workflowData.EngineConfig.Args for a features.shell_tool override when ResolveLLMProvider(workflowData) == LLMProviderGitHub, and rejects it the same way the TOML override is rejected. Add a regression test mirroring TestCodexGitHubInferenceRejectsExecToolOverride but using Args instead of Config.

@copilot please address this.

github-actions · 2026-10-04T20:54:05Z

Comment Memory

Peek at saved memory (pr-code-quality-reviewer)

reviewed_at: 2026-10-04T20:50:15Z
review_event: REQUEST_CHANGES
top_themes:
  - concurrency.queue schema regression rejects expressions
  - codex shell_tool guard bypass via engine.args
files_reviewed:
  - pkg/workflow/codex_config.go
  - pkg/workflow/codex_config_test.go
  - pkg/workflow/schemas/github-workflow.json
comment_count: 0

Note

This comment is managed by comment memory.

Expand the saved memory block to view or edit the persistent context for this thread.
Edit only the text inside the backtick fences; workflow metadata and the footer are regenerated automatically.

Learn more about comment memory

🔎 Code quality review by PR Code Quality Reviewer · copilot · gpt54 · 48.5 AIC · ⌖ 6.99 AIC · ⊞ 19.4K · ◷
_{Comment /review to run again}

github-actions

Request changes

This still has two merge-blocking regressions: the schema change breaks previously valid concurrency.queue expressions, and the new Codex guard is bypassable because engine.args are appended after the generated shell_tool=false settings.

🔎 Code quality review by PR Code Quality Reviewer · copilot · gpt54 · 48.5 AIC · ⌖ 6.99 AIC · ⊞ 19.4K
_{Comment /review to run again}

github-actions

Impeccable-mode review (bug_fix → `harden`, `audit`)

Verified the two existing review findings by reproducing them locally:

Schema regression confirmed (pkg/workflow/schemas/github-workflow.json:46): removing the expressionSyntax branch from concurrency.queue breaks the pre-existing TestConcurrencyQueueSchemaValidation test:
```
- at '/concurrency': 'oneOf' failed, none matched
  - at '/concurrency/queue': value must be one of 'single', 'max'
```
This change is unrelated to the Codex exec-tool fix and silently drops support for queue: ${{ ... }} expressions. It should be reverted (restore the oneOf with expressionSyntax) or split into its own PR.
shell_tool guard bypass confirmed (pkg/workflow/codex_config.go:176): validateCodexManagedConfig only inspects config.Overrides["features"] (TOML engine.config). buildCodexCommand (pkg/workflow/codex_engine.go:345-349) appends workflowData.EngineConfig.Args after the generated -c features.shell_tool=false flag with no validation, so engine: { id: codex, args: ["-c", "features.shell_tool=true"] } re-enables the unsupported exec tool for Copilot inference, defeating the whole safeguard this PR adds.

Both are genuine blocking issues (one is a regression test failure, the other defeats the PR's stated purpose for GitHub inference). Requesting changes.

🧵 Reviewed using Impeccable skills by Impeccable Skills Reviewer · copilot · sonnet50 · 59.8 AIC · ⌖ 13 AIC · ⊞ 8.1K

github-actions · 2026-10-05T23:07:49Z

🎉 This pull request is included in a new release.

Release: v0.91.0

Initial plan

10b887d

Copilot AI assigned Copilot and pelikhan Oct 4, 2026

Copilot started work on behalf of pelikhan October 4, 2026 20:30 View session

Copilot AI linked an issue Oct 4, 2026 that may be closed by this pull request

[AW Top 10] 01 Fix Codex failures from unsupported exec custom tool #65483

Closed

2 tasks

Copilot AI and others added 2 commits October 4, 2026 20:39

Disable unsupported Codex exec for Copilot

040d22a

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Apply remaining changes

a94dbe9

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI changed the title ~~[WIP] Fix Codex failures from unsupported exec custom tool~~ Prevent Codex exec tool rejection with Copilot inference Oct 4, 2026

Copilot finished work on behalf of pelikhan October 4, 2026 20:41

Copilot AI requested a review from pelikhan October 4, 2026 20:41

pelikhan marked this pull request as ready for review October 4, 2026 20:43

Copilot AI balanced review requested due to automatic review settings October 4, 2026 20:43

Copilot started reviewing on behalf of pelikhan October 4, 2026 20:43 View session

pelikhan merged commit df4e7a4 into main Oct 4, 2026
39 of 41 checks passed

pelikhan deleted the copilot/aw-top-10-fix-codex-failures branch October 4, 2026 20:48

Copilot AI reviewed Oct 4, 2026

View reviewed changes

github-actions Bot requested changes Oct 4, 2026

View reviewed changes

github-actions Bot mentioned this pull request Oct 5, 2026

Suggested test: codex/copilot-inference-shell-tool-disabled githubnext/gh-aw-test#20154

Open

pelikhan mentioned this pull request Oct 10, 2026

Fix prompt-exhaustion misclassification and restore Avenger's bash tool access #67411

Merged

Conversation

Copilot AI commented Oct 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Oct 4, 2026

Uh oh!

github-actions Bot commented Oct 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

github-actions Bot commented Oct 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Skills-Based Review 🧠

Key Themes

Positive Highlights

Uh oh!

github-actions Bot Oct 4, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Oct 4, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Oct 4, 2026

Comment Memory

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Request changes

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Impeccable-mode review (bug_fix → harden, audit)

Uh oh!

github-actions Bot commented Oct 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Copilot AI commented Oct 4, 2026 •

edited

Loading

github-actions Bot commented Oct 4, 2026 •

edited

Loading

github-actions Bot commented Oct 4, 2026 •

edited

Loading

github-actions Bot commented Oct 4, 2026 •

edited

Loading

github-actions Bot commented Oct 4, 2026 •

edited

Loading

Impeccable-mode review (bug_fix → `harden`, `audit`)