🔍 Smoke Test Investigation - Run #18786022255

Summary

The Smoke Copilot workflow failed with the SAME permission denial issue as #2288. Despite being closed, this issue has recurred 4 hours later on the same branch. The Copilot agent receives "Permission denied" when attempting to use safe-outputs MCP tools, even after commit e8bde4f added tools: ["*"] field to MCP configs.

Failure Details

• Run: #18786022255
• Commit: e8bde4f
• Branch: copilot/update-copilot-agent-engine
• Trigger: workflow_dispatch
• Duration: 2.3 minutes
• Failed Jobs: create_issue (7s duration)

Recurrence Timeline

Time Between Occurrences: ~4.5 hours
Status: Issue recurred despite attempted fix

Root Cause Analysis

Primary Error (SAME AS #2288)

From /tmp/gh-aw/aw-mcp/logs/run-18786022255/agent-stdio.log/agent-stdio.log:13-14:

✗ create_issue
   Permission denied and could not request permission from user

Investigation Findings

MCP Server Status: ✅ WORKING CORRECTLY

• Safe-outputs MCP server running properly
• Tools registered: create_issue, missing_tool
• Config parsed successfully

Agent Behavior: ✅ FOUND TOOLS, ❌ DENIED PERMISSION

• Agent successfully listed merged PRs using GitHub MCP
• Agent attempted to use safe-outputs-create_issue tool
• Permission system blocked the tool call

Downstream Impact: ❌ CASCADING FAILURE

Error reading agent output file: ENOENT: no such file or directory, 
open '/tmp/gh-aw/safe-outputs/agent_output.json'

What Changed Between Occurrences

Commit e8bde4f: "Add tools field back to MCP configs with ["*"] value"

This commit attempted to fix MCP tool availability by:

• Adding tools: ["*"] field to all MCP server configs
• Using wildcard to allow all tools in Copilot CLI
• Updating GitHub, Playwright, safe-outputs, and other MCP configs

Result: The permission denial still occurs despite this change.

Why the Fix Didn't Work

The tools: ["*"] field may configure which MCP tools are registered, but it does NOT address the Copilot CLI's tool authorization/permission system that's blocking tool usage.

This is NOT:

• ❌ A tool registration issue (tools are registered)
• ❌ A config parsing issue (config is correct)
• ❌ An MCP server crash (server is running)

This IS:

• ✅ A Copilot CLI authorization system issue
• ✅ A tool permission/allowlist configuration problem
• ✅ A potential Copilot CLI version compatibility issue

Failed Jobs and Errors

Job Sequence

1. ✅ activation - succeeded (3s)
2. ✅ agent - succeeded (58s) - Agent ran but couldn't use safe-outputs
3. ✅ detection - succeeded (30s)
4. ❌ create_issue - failed (7s)
5. ⏭️ missing_tool - skipped

Error Summary

• Total Errors: 7
• Total Warnings: 4
• Key Error: Permission denied for safe-outputs-create_issue

Pattern Information

• Pattern ID: COPILOT_SAFE_OUTPUTS_PERMISSION_DENIED
• Category: Permission/Authorization Error
• Severity: High
• Is Flaky: No - Consistent behavior
• Total Occurrences: 2 (and counting)
• First Seen: 2025-10-24T12:03:06Z
• Last Seen: 2025-10-24T16:33:05Z

Comparison with Related Issues

This is a RECURRENCE, not a new pattern.

Recommended Actions

Immediate (Critical Priority)

• Reopen [smoke-detector] 🔍 Smoke Test Investigation - Smoke Copilot: Permission Denied for Safe-Outputs Tools #2288 or acknowledge this recurrence

  • Issue was closed but root cause remains unresolved
  • Why: Prevents issue from being marked as fixed when it's not

• Investigate Copilot CLI authorization beyond MCP registration

  • The tools: ["*"] field did not resolve the issue
  • Check if there's a separate permission/allowlist mechanism
  • Why: Root cause is in Copilot CLI, not MCP config

• Compare with Claude engine configuration

  • Claude smoke tests successfully use safe-outputs
  • Identify what Claude does differently
  • Why: Prove safe-outputs works with different engine

High Priority

• Check Copilot CLI 0.0.349 release notes

  • Look for tool permission changes
  • Check for known issues with MCP tool authorization
  • Why: May be a version-specific bug

• Test with explicit tool permissions in workflow

  # If Copilot supports workflow-level tool permissions:
  permissions:
    safe-outputs:
      - create_issue
      - missing_tool

  • Why: May need explicit permission grants

• Add pre-flight tool permission test

  • Check if safe-outputs tools are callable before running agent
  • Fail fast with clear error if permissions missing
  • Why: Better developer experience

Medium Priority

• Engage Copilot CLI team

  • File bug report with Copilot CLI project
  • Provide reproduction case
  • Why: Issue is in Copilot CLI, not gh-aw

• Consider workaround options

  • Use Claude engine for safe-outputs workflows until fixed
  • Implement alternative output mechanism for Copilot
  • Why: Unblock smoke tests

Prevention Strategies

1. Don't close issues until verified fixed

   • Add verification step: re-run smoke test after fix
   • Wait 24 hours to ensure issue doesn't recur

2. Automated recurrence detection

   • Pattern database now tracks occurrences
   • Smoke Detector will flag future recurrences

3. Root cause validation

   • Ensure fixes address actual root cause, not symptoms
   • The tools: ["*"] fix addressed registration, not authorization

Technical Details

Environment

• Copilot CLI: 0.0.349
• Node.js: v24.10.0
• Safe-Outputs MCP: v1.0.0
• MCP Protocol: 2025-06-18

Agent Session

• Task: Review last 5 merged PRs and create issue
• Tools Available: GitHub MCP (✅ used), safe-outputs MCP (❌ denied)
• Exit Code: 0 (success, but incomplete task)

MCP Config

{
  "tools": ["*"],
  "create_issue": {"max": 1, "min": 1},
  "missing_tool": {}
}

Historical Context

Related Investigations:

• Investigation rejig docs #1: /tmp/gh-aw/cache-memory/investigations/2025-10-24-18779136174.json (issue [smoke-detector] 🔍 Smoke Test Investigation - Smoke Copilot: Permission Denied for Safe-Outputs Tools #2288)
• Investigation Add workflow: githubnext/agentics/weekly-research #2: /tmp/gh-aw/cache-memory/investigations/2025-10-24-18786022255.json (this run)

Pattern File: /tmp/gh-aw/cache-memory/patterns/copilot_safe_outputs_permission_denied.json

Fix Attempts:

1. ✅ Fixed malformed JSON config ([smoke-detector] 🔍 Smoke Test Investigation - Smoke Copilot: Safe-Outputs MCP Crashes Due to Malformed Config JSON #2280)
2. ✅ Fixed string-spreading bug ([smoke-detector] 🔍 Smoke Test Investigation - Smoke Copilot Run #18778382550: Safe-Outputs MCP Treats Config as Character Array #2281)
3. ❌ Added tools: ["*"] field (e8bde4f) - DID NOT FIX ISSUE

Questions for Copilot Team

1. How does Copilot CLI's tool permission system work?
2. Is tools: ["*"] in MCP config sufficient to allow tool usage?
3. Are there additional permission configurations needed?
4. Is this a known issue in version 0.0.349?
5. Why does Claude engine work with safe-outputs but Copilot doesn't?

Related Issues

• [smoke-detector] 🔍 Smoke Test Investigation - Smoke Copilot: Permission Denied for Safe-Outputs Tools #2288 - First occurrence (closed prematurely)
• [smoke-detector] 🔍 Smoke Test Investigation - Smoke Copilot: Safe-Outputs MCP Crashes Due to Malformed Config JSON #2280 - Config parsing issues (fixed)
• [smoke-detector] 🔍 Smoke Test Investigation - Smoke Copilot Run #18778382550: Safe-Outputs MCP Treats Config as Character Array #2281 - String spreading bug (fixed)

Investigation Metadata:

• Investigator: Smoke Detector
• Investigation Run: #18786074249
• Pattern ID: COPILOT_SAFE_OUTPUTS_PERMISSION_DENIED
• Occurrence: 2 of 2+
• Severity: High
• Requires Reopen: Yes ([smoke-detector] 🔍 Smoke Test Investigation - Smoke Copilot: Permission Denied for Safe-Outputs Tools #2288)

AI generated by Smoke Detector - Smoke Test Failure Investigator