Align nine workflow tool allowlists and warn on denied prompt tools by Copilot · Pull Request #66554 · github/gh-aw

Copilot · 2026-10-07T12:13:00Z

Nine workflows required tools their effective configurations denied, blocking analysis. This change aligns those prompts and permissions and adds compiler warnings for explicit tool requirements that cannot be satisfied.

Workflow access
- Add missing shell commands and correct Claude prefix rules.
- Align prompts with native MCP versus CLI transport.
- Give GitHub-backed Codex fixed-path MCP readers instead of unsupported shell access.
- Stop Outcome Collector before reading potentially absent evaluations when no outcomes exist.
Compiler diagnostics
- Check explicit shell, native-read, and MCP instructions against merged permissions, engine capabilities, and transport settings.
- Include imported prompts; deduplicate advisory warnings without expanding permissions.
- Leave ambiguous prose and complex shell inference unchecked.
Coverage and documentation
- Add regressions for all nine workflows and compiler permission checks.
- Document warning behavior and regenerate workflow locks.

For example, this prompt now warns because jq is denied:

tools:
  bash: [echo]
---
Run `jq --version`.

Fixes [AW Top 10] 07 Fix tool allowlists that deny required tools #66491

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot

Copilot review overview

🟡 Changes recommended

Tool detection gaps and contradictory CLI/MCP instructions can still permit missed warnings and denied calls.

Review effort: Balanced
Findings: 4

Open (4)

Safeoutputs transport guidance is contradictory · New
GitHub transport guidance is inconsistent · New
Parser stops after the first MCP invocation · New
Validate availability of each safeoutputs tool · New

What changed in this PR

Aligns nine workflow tool permissions with prompt requirements and adds compile-time warnings for unavailable tools.

Changes:

Adds prompt/tool validation with import and transport awareness.
Corrects workflow allowlists, MCP/CLI usage, and Codex file access.
Adds tests, documentation, and regenerated locks.

File	Description
`pkg/workflow/reported_tool_permissions_test.go`	Tests permissions for nine workflows.
`pkg/workflow/prompt_tool_validation.go`	Implements prompt tool diagnostics.
`pkg/workflow/prompt_tool_validation_test.go`	Covers parsing and availability checks.
`pkg/workflow/compiler.go`	Invokes the new validator.
`docs/src/content/docs/reference/compilation-process.md`	Documents advisory warnings.
`.github/workflows/smoke-workflow-call.md`	Corrects Claude command prefixes.
`.github/workflows/smoke-workflow-call.lock.yml`	Regenerates the workflow.
`.github/workflows/smoke-copilot-aoai-apikey.md`	Clarifies CLI-based smoke tests.
`.github/workflows/smoke-copilot-aoai-apikey.lock.yml`	Regenerates the workflow.
`.github/workflows/sighthound-security-scan.md`	Adds fixed-path MCP readers.
`.github/workflows/sighthound-security-scan.lock.yml`	Regenerates the workflow.
`.github/workflows/outcome-collector.md`	Adds MCP readers and zero-outcome handling.
`.github/workflows/outcome-collector.lock.yml`	Regenerates the workflow.
`.github/workflows/daily-schema-audit-cursor.md`	Allows required shell commands.
`.github/workflows/daily-schema-audit-cursor.lock.yml`	Regenerates the workflow.
`.github/workflows/daily-regression-audit-kiro.md`	Expands shell access and guidance.
`.github/workflows/daily-regression-audit-kiro.lock.yml`	Regenerates the workflow.
`.github/workflows/daily-graft-intelligence.md`	Moves Graft usage to CLI transport.
`.github/workflows/daily-graft-intelligence.lock.yml`	Regenerates the workflow.
`.github/workflows/daily-compiler-quality.md`	Aligns tools and native MCP transport.
`.github/workflows/daily-compiler-quality.lock.yml`	Regenerates the workflow.
`.github/workflows/code-scanning-fixer.md`	Moves mounted tools to CLI usage.
`.github/workflows/code-scanning-fixer.lock.yml`	Regenerates the workflow.

Copilot · 2026-10-07T14:27:35Z


 ## Workflow Steps

+Use native edit tools for writing cache records and report bodies rather than shell heredocs. Safe outputs are CLI-mounted: invoke `safeoutputs <tool>` through bash (use `--help` for parameters), not an invented native safe-output tool.


All safe-output actions now use the CLI-mounted safeoutputs command through bash, including create-pull-request, create-issue, and noop. Fixed in c20f9b3.

Copilot · 2026-10-07T14:27:36Z

 ### 2. List All Open Alerts

-Use `list_code_scanning_alerts` to list all open code scanning alerts.
+Run `github list_code_scanning_alerts` through bash to list all open code scanning alerts. The GitHub CLI wrapper is available; do not invent a `github-list_code_scanning_alerts` tool name.


All GitHub reads now use the CLI-mounted github wrapper, and checked-out files are read through allowed shell readers; native GitHub MCP and get_file_contents guidance was removed. Fixed in c20f9b3.

Copilot · 2026-10-07T14:27:35Z

+	if requirements, qualified := promptQualifiedRequirements(target); qualified {
+		return requirements


Qualified invocations are now collected across the full instruction, with a regression that denies only the second custom MCP tool. Fixed in c20f9b3.

Copilot · 2026-10-07T14:27:35Z

+	case "safeoutputs":
+		return HasSafeOutputsEnabled(data.SafeOutputs)


Safe-output availability now checks the exact enabled manifest tool set, including dynamic tools; regressions cover noop-only configuration and custom scripts. Fixed in c20f9b3.

gh-aw-bot · 2026-10-07T13:53:55Z

@copilot address the following outstanding work in one pass:

Update this branch with the latest main using make merge-main, resolving any conflicts and preserving the intended changes.
Review (.github/workflows/code-scanning-fixer.md:117): This new CLI-only instruction conflicts with the earlier mandatory guidance at lines 97–98, which explicitly says not to invoke safeoutputs through bash and to emit those tools directly. Because cli-proxy: true excludes the native safeoutputs server, following the earlier guidance still causes denied calls. Make the transport guidance consistent throughout the prompt. - Align nine workflow tool allowlists and warn on denied prompt tools #66554 (comment)
Review (.github/workflows/code-scanning-fixer.md:136): Switching these two calls to the github CLI is incomplete: the Tool Usage section at lines 87–91 still requires native GitHub MCP and forbids shell API access, while step 5 still requires native get_file_contents at line 173. With cli-proxy: true, the entire GitHub server is CLI-mounted, so the agent can still follow those instructions and hit the same denial. Update all GitHub-read guidance and calls to the CLI transport consistently. - Align nine workflow tool allowlists and warn on denied prompt tools #66554 (comment)
Review (pkg/workflow/prompt_tool_validation.go:207): Only the first qualified invocation on a line is returned, so an instruction such as the new Sighthound prompt's Call mcp__...findings and mcp__...summary never validates the second tool. A denied or misspelled second MCP tool therefore produces no warning. Parse all qualified invocations in the target (with corresponding regression coverage) instead of stopping after the first match. - Align nine workflow tool allowlists and warn on denied prompt tools #66554 (comment)
Review (pkg/workflow/prompt_tool_validation.go:365): This treats every safeoutputs tool as available whenever any safe output is enabled. For example, a workflow that only enables noop will not warn for Call mcp__safeoutputs__add_labels, even though that tool is absent at runtime. Check the requested tool against the complete enabled safe-output tool set, including dynamic tools, rather than the server-level boolean. - Align nine workflow tool allowlists and warn on denied prompt tools #66554 (comment)

Push the necessary fixes, reply to each listed review thread and resolve it when addressed. Ignore feedback already answered or resolved. Use the pr-finisher skill and stop when only human review or CI remains; do not trigger CI.

Sous-chef head: 6974129
Sous-chef work: 00d5db7e8ad7f1aa713db5d325afcdf024cc62da93c23ecccbfd29d6ed39b998 015af6facd1b04632082603a240ef7d1219a19d15e9b16afa6d2bb8223e9f4f9 1505756a2b092d7aec2370a4a90b47e28eb94420ab6cbfa5cab6d71e7f97dce5 32c6e0c8a9fbf5627c3a2cb92202a115c3680f14f794719a622124042ecf4ea3
Sous-chef state: 1a78b194d387c57c967c59791693ce20c6e34c8c791d69cfc36e5fb29a72ae8c

Generated by 👨‍🍳 PR Sous Chef · pi · haiku45 · 7.92 AIC · ⌖ 6.03 AIC · ⊞ 1K · ◷
_{Comment /souschef to run again}

github-actions · 2026-10-07T13:57:16Z

✅ Design Decision Gate 🏗️ completed the design decision gate check. See the comment below for the result and any generated ADR draft.

🏗️ ADR gate enforced by Design Decision Gate 🏗️

github-actions · 2026-10-07T13:58:25Z

🧠 Matt Pocock Skills Reviewer is reviewing this pull request using Matt Pocock's engineering skills...

github-actions · 2026-10-07T13:59:12Z

✅ Ponytail Reviewer completed successfully!

Generated by Ponytail Reviewer for #66554

github-actions · 2026-10-07T14:00:14Z

✅ PR Code Quality Reviewer completed the code quality review.

🔎 Code quality review by PR Code Quality Reviewer

github-actions · 2026-10-07T14:01:14Z

✅ Test Quality Sentinel completed test quality analysis.

Test Quality Sentinel skipped because pre-fetch PR data was unavailable: unable to fetch test file diff

🧪 Test quality analysis by Test Quality Sentinel

github-actions · 2026-10-07T14:03:13Z

🏗️ Design Decision Gate: ADR Required

This PR triggers ADR enforcement and no Architecture Decision Record was found.

Why this PR requires an ADR

Check	Result
`implementation` label	❌ not present
New lines in business-logic dirs (`pkg/`)	✅ 868 additions (threshold: 100)
Config	no `.design-gate.yml` — defaults used

Evidence: 23 changed files, including pkg/workflow/prompt_tool_validation.go (+464), pkg/workflow/prompt_tool_validation_test.go (+314), pkg/workflow/reported_tool_permissions_test.go (+89), and pkg/workflow/compiler.go.

ADR search results

PR body — no docs/adr/ link, no "ADR"/"Architecture Decision" section
PR branch — no docs/adr/66554-*.md present before this run
Linked issue (Fixes #66491) — not consulted; branch/body search was conclusive

Action taken

A draft ADR has been generated from the diff and committed to this branch:

docs/adr/66554-advisory-compile-time-validation-of-prompt-tool-requirements.md

It records the decision as: add an advisory, compile-time validator that extracts explicit tool requirements from prompt text (including imports) and emits warnings — never errors, never permission widening — when the merged permissions, engine capabilities, or transport cannot satisfy them.

Alternatives captured: hard compile error, and runtime-only detection via log mining.

Next step for you

Review the draft ADR and correct anything inferred incorrectly from the diff.
Replace any remaining assumptions with the real decision rationale (especially the Deciders line and the negative consequences).
Change Status from Draft to Proposed or Accepted when the team agrees.

Merging with the ADR still in Draft means the recorded rationale has not been human-reviewed.

🏗️ ADR gate enforced by Design Decision Gate 🏗️ · pi · opus50 · 53.6 AIC · ⌖ 44.7 AIC · ⊞ 1.7K · ◷
_{Comment /review to run again}

github-actions

pkg/workflow/prompt_tool_validation.go:L39-41: yagni: second recursive runtime-import collector. Reuse c.collectRuntimeImportMarkdownForCompilerAnalysis(data), adding comment stripping there if needed.

net: -25 lines possible.

Generated by ✂️ Ponytail Reviewer for #66554 · codex · gpt56 · 23.9 AIC · ⌖ 6.49 AIC · ⊞ 13.4K
_{Comment /ponytail to run again}

github-actions · 2026-10-07T14:03:49Z

+	for _, entry := range data.PromptImports {
+		contents = append(contents, entry.Markdown)
+	}
+	seed := runtimeImportValidationMarkdown(data) + "\n" + strings.Join(contents, "\n")


L39-41: yagni: second recursive runtime-import collector. Reuse c.collectRuntimeImportMarkdownForCompilerAnalysis(data), adding comment stripping there if needed.

Prompt validation now reuses collectRuntimeImportMarkdownForCompilerAnalysis; the shared collector strips XML comments and includes inline imported prompt markdown. Nested, commented, and confined import coverage was added. Fixed in c20f9b3.

github-actions · 2026-10-07T14:06:58Z

Comment Memory

Peek at saved memory (pr-code-quality-reviewer)

reviewed_at: 2026-10-07T14:02:13Z
review_event: REQUEST_CHANGES
top_themes:
- prompt/tool contract contradictions in code-scanning-fixer
- prompt validator misses secondary qualified MCP references
- prompt validator ignores bare safe-output aliases
files_reviewed:
- .github/workflows/code-scanning-fixer.md
- .github/workflows/daily-compiler-quality.md
- .github/workflows/daily-graft-intelligence.md
- .github/workflows/daily-regression-audit-kiro.md
- .github/workflows/daily-schema-audit-cursor.md
- .github/workflows/outcome-collector.md
- .github/workflows/sighthound-security-scan.md
- .github/workflows/smoke-copilot-aoai-apikey.md
- .github/workflows/smoke-workflow-call.md
- pkg/workflow/compiler.go
- pkg/workflow/prompt_tool_validation.go
- pkg/workflow/prompt_tool_validation_test.go
- pkg/workflow/reported_tool_permissions_test.go
comment_count: 1

Note

This comment is managed by comment memory.

Expand the saved memory block to view or edit the persistent context for this thread.
Edit only the text inside the backtick fences; workflow metadata and the footer are regenerated automatically.

Learn more about comment memory

🔎 Code quality review by PR Code Quality Reviewer · copilot · gpt54 · 52.3 AIC · ⌖ 5.51 AIC · ⊞ 19.7K · ◷
_{Comment /review to run again}

github-actions

Request changes

The new prompt-permission checks still miss explicit tool requirements, and code-scanning-fixer now gives conflicting instructions about native tools versus CLI wrappers.

Blocking themes

code-scanning-fixer tells the agent to use CLI-mounted github/safeoutputs commands in the workflow steps, but its earlier Tool Usage section still mandates native MCP reads and direct safe-output emission.
prompt_tool_validation.go only validates the first qualified tool reference on a line, so combined instructions can still hide a denied MCP tool.
The validator also ignores bare safe-output aliases such as create_issue and add_labels, so the main prompt style used in these workflows still bypasses the new warning entirely.

🔎 Code quality review by PR Code Quality Reviewer · copilot · gpt54 · 52.3 AIC · ⌖ 5.51 AIC · ⊞ 19.7K
_{Comment /review to run again}

github-actions · 2026-10-07T14:07:03Z

+	if strings.EqualFold(verb, "run") && promptLiteralShellCommand(target) {
+		return promptShellRequirements(strings.TrimSuffix(target, "."))
+	}
+	if strings.EqualFold(verb, "use") || strings.EqualFold(verb, "call") {


This validator still ignores the normal bare safe-output aliases, so prompts like Call create_issue or Use add_labels can require tools that were never configured and compile without any warning.

💡 Why this matters

promptInvocationRequirements only resolves bare names through knownGitHubTools, so the new availability checks never run for the safe-output spellings these workflows actually use. A missing or denied safe output can still slip through unchanged. Please map configured safe-output aliases into the safeoutputs server and add a regression that fails when Call create_issue is present but safe-outputs.create-issue is absent.

Bare safe-output names and aliases now resolve to safeoutputs requirements and use the same exact availability check; a regression verifies create_issue warns when only add-labels is enabled. Fixed in c20f9b3.

Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>

* Initial plan * Stop Codex resume retries on deterministic request-body rejection Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> * Export Codex request error code helper Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>

…66560) * Initial plan * Apply remaining changes Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>

Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>

done

pelikhan · 2026-10-07T15:50:04Z

@copilot run pr-finisher skill

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

github-actions · 2026-10-07T20:32:01Z

🎉 This pull request is included in a new release.

Release: v0.91.5

Initial plan

a26f800

Copilot AI assigned Copilot and pelikhan Oct 7, 2026

Copilot started work on behalf of pelikhan October 7, 2026 12:13 View session

Copilot AI linked an issue Oct 7, 2026 that may be closed by this pull request

[AW Top 10] 07 Fix tool allowlists that deny required tools #66491

Closed

2 tasks

Align nine workflow tool permissions and warn on denied prompt tools

6974129

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI changed the title ~~[WIP] Fix tool allowlists that deny required tools~~ Align nine workflow tool allowlists and warn on denied prompt tools Oct 7, 2026

Copilot AI requested a review from pelikhan October 7, 2026 13:07

Copilot finished work on behalf of pelikhan October 7, 2026 13:07

pelikhan marked this pull request as ready for review October 7, 2026 13:28

Copilot AI balanced review requested due to automatic review settings October 7, 2026 13:28

Copilot started reviewing on behalf of pelikhan October 7, 2026 13:28 View session

Copilot AI reviewed Oct 7, 2026

View reviewed changes

github-actions Bot mentioned this pull request Oct 7, 2026

[pr-sous-chef] 🍳 PR Sous Chef Summary: 4 PRs nudged toward completion #66387

Closed

Copilot started work on behalf of gh-aw-bot October 7, 2026 13:54 View session

docs(adr): add draft ADR-66554 for advisory prompt tool validation

d75287d

github-actions Bot reviewed Oct 7, 2026

View reviewed changes

github-actions Bot previously requested changes Oct 7, 2026

View reviewed changes

Copilot AI and others added 4 commits October 7, 2026 14:15

Classify stalled Codex MCP calls as transport wedges (#66303)

64b5fb7

Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>

Fix prompt tool validation review findings

299fbbd

Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>

Address follow-up prompt validation findings

c20f9b3

Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>

Copilot finished work on behalf of gh-aw-bot October 7, 2026 14:29

Copilot AI requested a review from gh-aw-bot October 7, 2026 14:29

Copilot started work on behalf of pelikhan October 7, 2026 15:50 View session

Merge latest main

077b9f1

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot finished work on behalf of pelikhan October 7, 2026 16:15

pelikhan merged commit 9bdf285 into main Oct 7, 2026
26 of 27 checks passed

pelikhan deleted the copilot/aw-top-10-07-fix-tool-allowlists branch October 7, 2026 16:30


		## Workflow Steps

		Use native edit tools for writing cache records and report bodies rather than shell heredocs. Safe outputs are CLI-mounted: invoke `safeoutputs <tool>` through bash (use `--help` for parameters), not an invented native safe-output tool.

		if requirements, qualified := promptQualifiedRequirements(target); qualified {
		return requirements

		case "safeoutputs":
		return HasSafeOutputsEnabled(data.SafeOutputs)

Conversation

Copilot AI commented Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Copilot AI Oct 7, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 7, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 7, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 7, 2026

Choose a reason for hiding this comment

Uh oh!

gh-aw-bot commented Oct 7, 2026

Uh oh!

github-actions Bot commented Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 7, 2026

Uh oh!

github-actions Bot commented Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 7, 2026

🏗️ Design Decision Gate: ADR Required

Why this PR requires an ADR

ADR search results

Action taken

Next step for you

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Oct 7, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 7, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Oct 7, 2026

Comment Memory

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Request changes

Uh oh!

github-actions Bot Oct 7, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 7, 2026

Choose a reason for hiding this comment

Uh oh!

pelikhan commented Oct 7, 2026

Uh oh!

Uh oh!

github-actions Bot commented Oct 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Copilot AI commented Oct 7, 2026 •

edited

Loading

github-actions Bot commented Oct 7, 2026 •

edited

Loading

github-actions Bot commented Oct 7, 2026 •

edited

Loading

github-actions Bot commented Oct 7, 2026 •

edited

Loading

github-actions Bot commented Oct 7, 2026 •

edited

Loading