Skip to content

Classify silent agent exits and ensure terminal safe outputs - #65660

Merged
pelikhan merged 6 commits into
mainfrom
copilot/aw-top-10-ensure-terminal-safe-output
Oct 5, 2026
Merged

pelikhan merged 6 commits into
mainfrom
copilot/aw-top-10-ensure-terminal-safe-output

Conversation

Copilot AI commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

Agents can exit without a valid terminal safe output, leaving failure reports unable to distinguish driver termination from malformed CLI calls or agent silence.

  • Terminal output handling: Add a cause-aware report_incomplete signal for empty or diagnostic-only results. Preserve explicit noops and task outputs.
  • Failure classification: Distinguish driver exits, CLI errors, rejected outputs, and missing terminal results through issue titles and deduplication categories. Allow each category in reporting filters.
  • CLI validation: Reject stray positional arguments—even alongside valid flags—with actionable guidance, without echoing payloads.
safe-outputs:
  report-failure-as-issue:
    - engine_driver_failure
    - safeoutputs_cli_error

Copilot AI and others added 3 commits October 4, 2026 21:03
…ents

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copilot AI changed the title [WIP] Add post-agent check for terminal safe output errors Classify silent agent exits and ensure terminal safe outputs Oct 4, 2026
Copilot AI requested a review from pelikhan October 4, 2026 21:13
@github-actions

github-actions Bot commented Oct 4, 2026

Copy link
Copy Markdown
Contributor

Excellent work! Classifying silent agent exits and ensuring terminal safe outputs is well-scoped and tested. This PR looks ready for review.

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • github.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "github.com"

See Network Configuration for more information.

Generated by ✅ Contribution Check · copilot · auto · 52.8 AIC · ⌖ 7.51 AIC · ⊞ 9.1K · ◷

@pelikhan
pelikhan marked this pull request as ready for review October 4, 2026 21:48
Copilot AI balanced review requested due to automatic review settings October 4, 2026 21:48

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Agent-controlled reasons can currently spoof trusted driver and CLI failure classifications.

Review effort: Balanced
Findings: 1 Medium severity

Open (1)
What changed in this PR

Adds terminal safe-output detection and cause-specific failure reporting for silent or malformed agent exits.

Changes:

  • Classifies missing terminal outputs, driver exits, CLI errors, and rejected outputs.
  • Rejects stray safeoutputs CLI arguments without exposing payloads.
  • Updates schema, documentation, tests, and generated Codex workflow locks.
File Description
pkg/​parser/​schemas/​main_workflow_schema.json Allows new failure categories.
pkg/​parser/​schema_location_test.go Tests category validation.
docs/​src/​content/​docs/​reference/​safe-outputs.md Documents terminal-output categories.
actions/​setup/​js/​mcp_cli_bridge.cjs Rejects positional safeoutputs arguments.
actions/​setup/​js/​mcp_cli_bridge.test.cjs Tests CLI rejection and audit safety.
actions/​setup/​js/​handle_agent_failure.cjs Classifies and deduplicates failure causes.
actions/​setup/​js/​handle_agent_failure.test.cjs Tests cause-specific reporting.
actions/​setup/​js/​empty_output_outcome.cjs Builds cause-aware incomplete outcomes.
actions/​setup/​js/​empty_output_outcome.test.cjs Tests outcome classification.
actions/​setup/​js/​collect_ndjson_output.cjs Adds fallback for diagnostic-only output.
actions/​setup/​js/​collect_ndjson_output.test.cjs Tests terminal fallback collection.
.github/​workflows/​windows.lock.yml Refreshes generated Codex configuration.
.github/​workflows/​update-astro.lock.yml Refreshes generated Codex configuration.
.github/​workflows/​smoke-otel-backends.lock.yml Refreshes generated Codex configuration.
.github/​workflows/​smoke-github-codex.lock.yml Refreshes generated Codex configuration.
.github/​workflows/​smoke-codex-auto.lock.yml Refreshes generated Codex configuration.
.github/​workflows/​smoke-ci.lock.yml Refreshes generated Codex configuration.
.github/​workflows/​sighthound-security-scan.lock.yml Refreshes generated Codex configuration.
.github/​workflows/​schema-feature-coverage.lock.yml Refreshes generated Codex configuration.
.github/​workflows/​repo-audit-analyzer.lock.yml Refreshes generated Codex configuration.
.github/​workflows/​purelock.lock.yml Refreshes generated Codex configuration.
.github/​workflows/​poem-bot.lock.yml Refreshes generated Codex configuration.
.github/​workflows/​outcome-collector.lock.yml Refreshes generated Codex configuration.
.github/​workflows/​metrics-collector.lock.yml Refreshes generated Codex configuration.
.github/​workflows/​issue-triage-agent.lock.yml Refreshes generated Codex configuration.
.github/​workflows/​grumpy-reviewer.lock.yml Refreshes generated Codex configuration.
.github/​workflows/​github-remote-mcp-auth-test.lock.yml Refreshes generated Codex configuration.
.github/​workflows/​front-page-copy-guard.lock.yml Refreshes generated Codex configuration.
.github/​workflows/​example-permissions-warning.lock.yml Refreshes generated Codex configuration.
.github/​workflows/​eslint-monster.lock.yml Refreshes generated Codex configuration.
.github/​workflows/​dependabot-go-checker.lock.yml Refreshes generated Codex configuration.
.github/​workflows/​daily-windows-defender-scan.lock.yml Refreshes generated Codex configuration.
.github/​workflows/​daily-spending-forecast.lock.yml Refreshes generated Codex configuration.
.github/​workflows/​daily-regulatory.lock.yml Refreshes generated Codex configuration.
.github/​workflows/​daily-go-test-parallelizer.lock.yml Refreshes generated Codex configuration.
.github/​workflows/​daily-evals-report.lock.yml Refreshes generated Codex configuration.
.github/​workflows/​daily-doc-updater.lock.yml Refreshes generated Codex configuration.
.github/​workflows/​daily-cli-performance.lock.yml Refreshes generated Codex configuration.
.github/​workflows/​daily-awf-spec-compiler-surfacing.lock.yml Refreshes generated Codex configuration.
.github/​workflows/​copilot-centralization-optimizer.lock.yml Refreshes generated Codex configuration.
.github/​workflows/​commit-changes-analyzer.lock.yml Refreshes generated Codex configuration.
.github/​workflows/​cloclo.lock.yml Refreshes generated Codex configuration.
.github/​workflows/​changeset.lock.yml Refreshes generated Codex configuration.
.github/​workflows/​avenger.lock.yml Refreshes generated Codex configuration.
.github/​workflows/​audit-workflows.lock.yml Refreshes generated Codex configuration.
.github/​workflows/​api-consumption-report.lock.yml Refreshes generated Codex configuration.
.github/​workflows/​agentic-token-trend-audit.lock.yml Refreshes generated Codex configuration.
.github/​workflows/​ace-editor.lock.yml Refreshes generated Codex configuration.

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.


// Sanitize workflow name for title
const sanitizedWorkflowName = sanitizeContent(workflowName, { maxLength: 100 });
const emptyOutputCause = agentOutputResult?.items?.find(item => item.type === "report_incomplete" && EMPTY_OUTPUT_CAUSES.has(item.reason))?.reason;

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 250f0fa: the collector now stamps collectorEmptyOutputCause as root metadata and the failure handler classifies only that trusted field. Agent-authored report_incomplete.reason values remain generic, covered by regression tests.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 250f0fa: the collector now stamps collectorEmptyOutputCause as root metadata and the failure handler classifies only that trusted field. Agent-authored report_incomplete.reason values remain generic, covered by regression tests.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 250f0fa: the collector now stamps collectorEmptyOutputCause as root metadata and the failure handler classifies only that trusted field. Agent-authored report_incomplete.reason values remain generic, covered by regression tests.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 250f0fa: the collector now stamps collectorEmptyOutputCause as root metadata and the failure handler classifies only that trusted field. Agent-authored report_incomplete.reason values remain generic, covered by regression tests.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 250f0fa: the collector now stamps collectorEmptyOutputCause as root metadata and the failure handler classifies only that trusted field. Agent-authored report_incomplete.reason values remain generic, covered by regression tests.

@github-actions

github-actions Bot commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer

@github-actions

github-actions Bot commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

✅ Test Quality Sentinel completed test quality analysis.

Test Quality Sentinel skipped because pre-fetch PR data was unavailable: unable to fetch test file diff

🧪 Test quality analysis by Test Quality Sentinel

@github-actions

github-actions Bot commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

✅ PR Code Quality Reviewer completed the code quality review.

🔎 Code quality review by PR Code Quality Reviewer

@github-actions

github-actions Bot commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

✅ Ponytail Reviewer completed successfully!

Lean already. Ship.

Generated by Ponytail Reviewer for #65660

@github-actions

github-actions Bot commented Oct 4, 2026

Copy link
Copy Markdown
Contributor

🔍 Design Decision Gate 🏗️ is checking for design decision records on this pull request...

@github-actions

github-actions Bot commented Oct 4, 2026

Copy link
Copy Markdown
Contributor

Comment Memory

Peek at saved memory (pr-code-quality-reviewer)
reviewed_at: 2026-10-04T22:13:45Z
review_event: REQUEST_CHANGES
top_themes:
  - failure classification conflates CLI misuse with downstream MCP/tool failures
  - specific CLI-error classification is overwritten by generic driver-exit handling
files_reviewed:
  - actions/setup/js/collect_ndjson_output.cjs
  - actions/setup/js/collect_ndjson_output.test.cjs
  - actions/setup/js/empty_output_outcome.cjs
  - actions/setup/js/empty_output_outcome.test.cjs
  - actions/setup/js/handle_agent_failure.cjs
  - actions/setup/js/handle_agent_failure.test.cjs
  - actions/setup/js/mcp_cli_bridge.cjs
  - actions/setup/js/mcp_cli_bridge.test.cjs
  - docs/src/content/docs/reference/safe-outputs.md
  - pkg/parser/schema_location_test.go
  - pkg/parser/schemas/main_workflow_schema.json
comment_count: 2

Note

This comment is managed by comment memory.

Expand the saved memory block to view or edit the persistent context for this thread.
Edit only the text inside the backtick fences; workflow metadata and the footer are regenerated automatically.

Learn more about comment memory

🔎 Code quality review by PR Code Quality Reviewer · copilot · gpt54 · 60 AIC · ⌖ 7.42 AIC · ⊞ 19.4K · ◷
Comment /review to run again

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request changes

The new terminal-safe-output classification is heading in the right direction, but two parts of the implementation still mislabel failures badly enough to block merge.

Blocking themes
  • safeoutputs_cli_error currently catches downstream MCP/tool failures, not just malformed CLI invocations.
  • The later driver-exit branch overwrites that specific cause in the common non-zero-exit path, so the new category often never survives to reporting.

🔎 Code quality review by PR Code Quality Reviewer · copilot · gpt54 · 60 AIC · ⌖ 7.42 AIC · ⊞ 19.4K
Comment /review to run again

for (const line of fs.readFileSync(auditPath, "utf8").split("\n")) {
try {
const entry = JSON.parse(line);
if (["parse_args_error", "unrecognized_args", "tool_error", "call_error"].includes(entry?.event)) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Classifying tool_error and call_error as safeoutputs_cli_error is wrong, because those audit events mean the bridge got past argument parsing and the failure came from the MCP call or the safe-output tool itself.

💡 Limit this bucket to actual CLI invocation failures

Right now a real downstream failure like a rejected create_issue call, a JSON-RPC error from the safeoutputs server, or an isError=true tool result will be reported as "failed to invoke safeoutputs CLI". That sends triage in the wrong direction and defeats the new failure taxonomy this PR is adding.

A safer split is to reserve safeoutputs_cli_error for parse_args_error / unrecognized_args, and either leave tool_error / call_error under the generic missing-terminal-output path or introduce a separate category for MCP/tool execution failures.

if (["parse_args_error", "unrecognized_args"].includes(entry?.event)) {
  reason = "safeoutputs_cli_error";
}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 250f0fa: only parse_args_error and unrecognized_args classify as safeoutputs_cli_error. tool_error and call_error remain under the missing-terminal-output category; regression tests cover both.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 250f0fa: only parse_args_error and unrecognized_args classify as safeoutputs_cli_error. tool_error and call_error remain under the missing-terminal-output category; regression tests cover both.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 250f0fa: only parse_args_error and unrecognized_args classify as safeoutputs_cli_error. tool_error and call_error remain under the missing-terminal-output category; regression tests cover both.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 250f0fa: only parse_args_error and unrecognized_args classify as safeoutputs_cli_error. tool_error and call_error remain under the missing-terminal-output category; regression tests cover both.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 250f0fa: only parse_args_error and unrecognized_args classify as safeoutputs_cli_error. tool_error and call_error remain under the missing-terminal-output category; regression tests cover both.

// Ignore missing CLI audit evidence for workflows using MCP directly.
}
for (const event of events) {
if (event.type === "agent.execution" && event.provenance.component === "execution" && event.provenance.phase === "agent" && event.data.exitCode > 0) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This unconditional overwrite means the new safeoutputs_cli_error reason disappears as soon as the agent exits non-zero, which is the normal outcome of the bridge calling core.setFailed() for parse/tool/call errors.

💡 Preserve more specific failure causes instead of replacing them with the generic driver-exit bucket

The bridge marks malformed invocations and tool-call failures as failed steps, so the session will usually also record a non-zero agent.execution exit code. With the current ordering, the audit pass sets safeoutputs_cli_error and then this branch immediately replaces it with engine_driver_failure, making the new category effectively unreachable in real failures.

Only fall back to engine_driver_failure when the current reason is still the generic missing_terminal_safe_output, or introduce an explicit precedence table so more specific causes win.

if (reason === "missing_terminal_safe_output" && event.data.exitCode > 0) {
  reason = "engine_driver_failure";
}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 250f0fa: the CLI parse failure takes precedence over the nonzero exit it causes via core.setFailed(). A regression test combines the parse audit event with a nonzero exit and confirms the CLI classification is retained.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 250f0fa: the CLI parse failure takes precedence over the nonzero exit it causes via core.setFailed(). A regression test combines the parse audit event with a nonzero exit and confirms the CLI classification is retained.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 250f0fa: the CLI parse failure takes precedence over the nonzero exit it causes via core.setFailed(). A regression test combines the parse audit event with a nonzero exit and confirms the CLI classification is retained.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 250f0fa: the CLI parse failure takes precedence over the nonzero exit it causes via core.setFailed(). A regression test combines the parse audit event with a nonzero exit and confirms the CLI classification is retained.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 250f0fa: the CLI parse failure takes precedence over the nonzero exit it causes via core.setFailed(). A regression test combines the parse audit event with a nonzero exit and confirms the CLI classification is retained.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Impeccable mode applied: harden (bug-fix/edge-case PR classifying silent agent exits and ensuring terminal safe outputs).

Summary

This PR adds valuable classification for silent agent exits (engine driver failure, CLI errors, missing terminal output) and a safeoutputs CLI hardening fix (rejecting stray positional args). The implementation is well tested overall.

One blocking issue found: the new emptyOutputCause classification in handle_agent_failure.cjs trusts the public, agent-controlled report_incomplete.reason field as if it were server-verified runtime evidence, letting an agent spoof engine_driver_failure/safeoutputs_cli_error/etc. to control failure-issue titles and dedup categories. See inline comment for details and a suggested fix (tag the synthetic outcome so only server-generated signals are honored).

Everything else (CLI positional-argument rejection, schema category additions, doc updates) looks solid and well covered by tests.

🧵 Reviewed using Impeccable skills by Impeccable Skills Reviewer · copilot · sonnet50 · 95.3 AIC · ⌖ 13.3 AIC · ⊞ 8.1K


// Sanitize workflow name for title
const sanitizedWorkflowName = sanitizeContent(workflowName, { maxLength: 100 });
const emptyOutputCause = agentOutputResult?.items?.find(item => item.type === "report_incomplete" && EMPTY_OUTPUT_CAUSES.has(item.reason))?.reason;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Security/correctness: agent-controlled reason is trusted as runtime evidence of a driver/CLI failure.

report_incomplete is a public tool (actions/setup/js/safe_outputs_tools.json) whose reason field accepts any string from the agent. EMPTY_OUTPUT_CAUSES only gets populated with trusted values (engine_driver_failure, safeoutputs_cli_error, invalid_safe_outputs, missing_terminal_safe_output) inside buildEmptyOutputOutcome (synthetic, server-generated), but this line re-derives emptyOutputCause by scanning all report_incomplete items in agentOutputResult.items — including ones the agent itself emitted — and matching their reason against the same reserved set.

An agent can call safeoutputs report_incomplete '{"reason":"engine_driver_failure"}' directly and this code will treat it identically to a genuine driver crash: it changes the failure issue title (buildFailureIssueTitle), the dedup category (buildFailureMatchCategories), and triggers needsFailureDiagnostics (line ~4402), all based on untrusted agent input rather than actual runtime evidence (driver exit code, CLI audit log).

Suggested fix: tag the synthetic outcome from buildEmptyOutputOutcome with an internal marker (e.g. __synthetic: true or a separate field not in the public tool schema) and only honor EMPTY_OUTPUT_CAUSES classification when that marker is present, rather than matching on reason string equality alone.

@copilot please address this.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 250f0fa: collector-generated cause metadata is carried separately at the output root and passed through the output loader. Agent-provided reserved reason strings no longer affect issue titles or deduplication categories.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 250f0fa: collector-generated cause metadata is carried separately at the output root and passed through the output loader. Agent-provided reserved reason strings no longer affect issue titles or deduplication categories.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 250f0fa: collector-generated cause metadata is carried separately at the output root and passed through the output loader. Agent-provided reserved reason strings no longer affect issue titles or deduplication categories.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 250f0fa: collector-generated cause metadata is carried separately at the output root and passed through the output loader. Agent-provided reserved reason strings no longer affect issue titles or deduplication categories.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 250f0fa: collector-generated cause metadata is carried separately at the output root and passed through the output loader. Agent-provided reserved reason strings no longer affect issue titles or deduplication categories.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Skills-Based Review 🧠

Applied /diagnosing-bugs, /tdd, and /codebase-design — overall a well-tested, surgical fix for the silent-exit classification problem. Minor suggestions only; no blocking issues.

📋 Key Themes & Highlights

Key Themes

  • Precedence ambiguity: engine_driver_failure, safeoutputs_cli_error, and invalid_safe_outputs can theoretically all be true at once in buildEmptyOutputOutcome; the "last writer wins" ordering isn't documented or tested for the overlap case.
  • Scope asymmetry: the new rejectPositionalArguments strict-CLI validation only applies to the safeoutputs MCP server, not other bridged servers — likely intentional, but worth a one-line comment.
  • Four-site coupling: adding a new empty-output cause requires touching EMPTY_OUTPUT_CAUSES, buildFailureIssueTitle, buildFailureMatchCategories, the schema regex, and docs — a single lookup table would reduce future drift.

Positive Highlights

  • ✅ Addresses the prior reviewer's trust concern well: engine_driver_failure is sourced from a runtime-written exit-code file, and safeoutputs_cli_error from the CLI's own audit log — not the agent-controlled report_incomplete.reason string. EMPTY_OUTPUT_CAUSES allow-lists which reasons are trusted as categories, with an explicit regression test ("does not use arbitrary agent text as a failure category").
  • ✅ Good regression coverage: driver exit codes (1, 137, 139), CLI audit parsing without leaking payloads, diagnostic-only exits (missing_tool/missing_data) correctly not treated as terminal outputs, and a test that a failed tool call isn't mistaken for a driver crash.
  • ✅ Docs and JSON schema updated in lockstep with the new failure categories.

@copilot please address the review comments above.

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · copilot · sonnet50 · 106.9 AIC · ⌖ 13.6 AIC · ⊞ 10K
Comment /matt to run again

// Ignore missing CLI audit evidence for workflows using MCP directly.
}
for (const event of events) {
if (event.type === "agent.execution" && event.provenance.component === "execution" && event.provenance.phase === "agent" && event.data.exitCode > 0) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[/diagnosing-bugs] The engine_driver_failure classification loops over all events and keeps overwriting reason on every matching agent.execution entry, so if a later loop iteration also matches tool.execution_complete-style failure data it won't downgrade reason back — but conversely a driver exit recorded before a CLI audit error (both can coexist) means whichever block runs last wins silently, with no precedence test for the overlap case.

💡 Suggested fix

Add an explicit precedence test (e.g. both exit-code file and CLI audit log present) asserting which reason wins, and comment the intended priority order (driver failure > CLI error > invalid output > missing output) directly above the two assignment sites so future contributors don't accidentally invert it.

@copilot please address this.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 250f0fa: cause selection no longer depends on event ordering. A known CLI parse failure wins over the nonzero exit it triggers; otherwise a trusted driver exit supersedes invalid/missing-output causes. The overlapping CLI-audit/nonzero-exit case is tested.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 250f0fa: cause selection no longer depends on event ordering. A known CLI parse failure wins over the nonzero exit it triggers; otherwise a trusted driver exit supersedes invalid/missing-output causes. The overlapping CLI-audit/nonzero-exit case is tested.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 250f0fa: cause selection no longer depends on event ordering. A known CLI parse failure wins over the nonzero exit it triggers; otherwise a trusted driver exit supersedes invalid/missing-output causes. The overlapping CLI-audit/nonzero-exit case is tested.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 250f0fa: cause selection no longer depends on event ordering. A known CLI parse failure wins over the nonzero exit it triggers; otherwise a trusted driver exit supersedes invalid/missing-output causes. The overlapping CLI-audit/nonzero-exit case is tested.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 250f0fa: cause selection no longer depends on event ordering. A known CLI parse failure wins over the nonzero exit it triggers; otherwise a trusted driver exit supersedes invalid/missing-output causes. The overlapping CLI-audit/nonzero-exit case is tested.

@@ -902,6 +903,8 @@ function parseToolArgs(args, schemaProperties = {}, stdinContent = null) {
const canonicalKey = resolveSchemaPropertyKey(raw, schemaProperties, normalizedSchemaKeyMap, ambiguousNormalizedSchemaKeys);
result[canonicalKey] = true;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[/tdd] rejectPositionalArguments is only passed when serverName === SAFEOUTPUTS_SERVER_NAME, so other MCP CLI bridges silently keep the old "drop stray positionals" behavior this PR is trying to eliminate. If the intent is safeoutputs-specific tightening, a short comment explaining why non-safeoutputs servers are exempt would help; if it's an oversight, the gap deserves a regression test (e.g. a non-safeoutputs call with a positional arg + valid flags still succeeds today).

@copilot please address this.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 250f0fa: safeoutputs alone rejects stray positionals because its CLI arguments map to a strict tool schema. Other MCP bridges intentionally retain positional syntax; that scope is now documented beside the call.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 250f0fa: safeoutputs alone rejects stray positionals because its CLI arguments map to a strict tool schema. Other MCP bridges intentionally retain positional syntax; that scope is now documented beside the call.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 250f0fa: safeoutputs alone rejects stray positionals because its CLI arguments map to a strict tool schema. Other MCP bridges intentionally retain positional syntax; that scope is now documented beside the call.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 250f0fa: safeoutputs alone rejects stray positionals because its CLI arguments map to a strict tool schema. Other MCP bridges intentionally retain positional syntax; that scope is now documented beside the call.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 250f0fa: safeoutputs alone rejects stray positionals because its CLI arguments map to a strict tool schema. Other MCP bridges intentionally retain positional syntax; that scope is now documented beside the call.


// Sanitize workflow name for title
const sanitizedWorkflowName = sanitizeContent(workflowName, { maxLength: 100 });
const emptyOutputCause = agentOutputResult?.items?.find(item => item.type === "report_incomplete" && EMPTY_OUTPUT_CAUSES.has(item.reason))?.reason;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[/codebase-design] EMPTY_OUTPUT_CAUSES gates which reason strings are trusted as structured categories (good — this addresses the earlier reviewer concern about agent-controlled strings). Worth calling out in a short comment here that this .find() only trusts report_incomplete reasons present in the allow-list set, since the next maintainer adding a new cause needs to update both EMPTY_OUTPUT_CAUSES and buildFailureIssueTitle/buildFailureMatchCategories/docs/schema — four call sites for one concept. Consider a single lookup table (cause -> {category, title}) to avoid future drift between them.

@copilot please address this.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 250f0fa: cause keys and issue-title suffixes now share one lookup table, and those keys also gate failure categories. The comment calls out that schema and docs must remain aligned with the table.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 250f0fa: cause keys and issue-title suffixes now share one lookup table, and those keys also gate failure categories. The comment calls out that schema and docs must remain aligned with the table.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 250f0fa: cause keys and issue-title suffixes now share one lookup table, and those keys also gate failure categories. The comment calls out that schema and docs must remain aligned with the table.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 250f0fa: cause keys and issue-title suffixes now share one lookup table, and those keys also gate failure categories. The comment calls out that schema and docs must remain aligned with the table.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 250f0fa: cause keys and issue-title suffixes now share one lookup table, and those keys also gate failure categories. The comment calls out that schema and docs must remain aligned with the table.

@gh-aw-bot

Copy link
Copy Markdown
Collaborator

@copilot address the following outstanding work in one pass:

  1. Update this branch with the latest main using make merge-main, resolving any conflicts and preserving the intended changes.
  2. Review (actions/setup/js/handle_agent_failure.cjs:4277): emptyOutputCause trusts the agent-controlled report_incomplete.reason as runtime evidence. The public report_incomplete tool accepts any reason string, so an agent can emit engine_driver_failure or safeoutputs_cli_error despite exiting normally or producing task outputs; the resulting issue title, category, and deduplication key then misclassify agent behavior as a driver/CLI failure. Promote these reserved causes only from collector-stamped trusted metadata or an out-of-band runtime signal, and treat matching agent-authored reasons as ordinary report_incomplete. - Classify silent agent exits and ensure terminal safe outputs #65660 (comment)
  3. Review (actions/setup/js/empty_output_outcome.cjs:50): Classifying tool_error and call_error as safeoutputs_cli_error is wrong, because those audit events mean the bridge got past argument parsing and the failure came from the MCP call or the safe-output tool itself. - Classify silent agent exits and ensure terminal safe outputs #65660 (comment)
  4. Review (actions/setup/js/empty_output_outcome.cjs:63): This unconditional overwrite means the new safeoutputs_cli_error reason disappears as soon as the agent exits non-zero, which is the normal outcome of the bridge calling core.setFailed() for parse/tool/call errors. - Classify silent agent exits and ensure terminal safe outputs #65660 (comment)
  5. Review (actions/setup/js/handle_agent_failure.cjs:4277): Security/correctness: agent-controlled reason is trusted as runtime evidence of a driver/CLI failure. - Classify silent agent exits and ensure terminal safe outputs #65660 (comment)
  6. Review (actions/setup/js/empty_output_outcome.cjs:63): [/diagnosing-bugs] The engine_driver_failure classification loops over all events and keeps overwriting reason on every matching agent.execution entry, so if a later loop iteration also matches tool.execution_complete-style failure data it won't downgrade reason back — but conversely a driver exit recorded before a CLI audit error (both can coexist) means whichever block runs last wins silently, with no precedence test for the overlap case. - Classify silent agent exits and ensure terminal safe outputs #65660 (comment)
  7. Review (actions/setup/js/mcp_cli_bridge.cjs:904): [/tdd] rejectPositionalArguments is only passed when serverName === SAFEOUTPUTS_SERVER_NAME, so other MCP CLI bridges silently keep the old "drop stray positionals" behavior this PR is trying to eliminate. If the intent is safeoutputs-specific tightening, a short comment explaining why non-safeoutputs servers are exempt would help; if it's an oversight, the gap deserves a regression test (e.g. a non-safeoutputs call with a positional arg + valid flags still succeeds today). - Classify silent agent exits and ensure terminal safe outputs #65660 (comment)
  8. Review (actions/setup/js/handle_agent_failure.cjs:4277): [/codebase-design] EMPTY_OUTPUT_CAUSES gates which reason strings are trusted as structured categories (good — this addresses the earlier reviewer concern about agent-controlled strings). Worth calling out in a short comment here that this .find() only trusts report_incomplete reasons present in the allow-list set, since the next maintainer adding a new cause needs to update both EMPTY_OUTPUT_CAUSES and buildFailureIssueTitle/buildFailureMatchCategories/docs/schema — four call sites for one concept. Consider a single lookup table (cause -> {category, title}) to avoid future drift between them. - Classify silent agent exits and ensure terminal safe outputs #65660 (comment)
  9. Fix failing check conclusion (FAILURE): https://github.com/github/gh-aw/actions/runs/37237589559/job/111544969794.

Push the necessary fixes, reply to each listed review thread and resolve it when addressed. Ignore feedback already answered or resolved. Use the pr-finisher skill and stop when only human review or CI remains; do not trigger CI.

Sous-chef head: bd4d673
Sous-chef work: 0f42825218bb3274b5e6aaaf14e7c638c9641f5e947ea75e0ef7e20801a3080b 51e2287c3d1ee41b7e13769862add61dcd292a623cf943abc5968c984e246bcf 93564f6d4ab337ccca780e3a776af793f81520244acbd28c6436ab99d4013657 9790bfb58b6fad80b459ff2a02cb810519125cb343ea6926b195276e8a80275f c76b8dfa53188c098ff8e2afaf15456669b353013c9f575f4b0bb1c46bbf6373 ccb4182810bd52d1d58ff0c9b14698a67eb6a7262434629dff8d944f146cdee9 f2b852d618cd56ecab0bb70d5f365164ee144744f564e351d3befdb72d13147f f3879d4bb76c621d14283746990562b808d188503a8edeb25734b0a466e666e1
Sous-chef state: 8a67acfb5511585f9614fdd1e197bed747c243cd503566fe36bf5c8d14ed0ea6

Generated by 👨‍🍳 PR Sous Chef · pi · haiku45 · 7.05 AIC · ⌖ 7.12 AIC · ⊞ 3.2K · ◷
Comment /souschef to run again

Copilot AI and others added 2 commits October 4, 2026 23:05
…ure-terminal-safe-output

Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
Copilot AI requested a review from gh-aw-bot October 4, 2026 23:17
@pelikhan
pelikhan merged commit ba9e49e into main Oct 5, 2026
2 checks passed
@pelikhan
pelikhan deleted the copilot/aw-top-10-ensure-terminal-safe-output branch October 5, 2026 14:16
@github-actions

github-actions Bot commented Oct 5, 2026

Copy link
Copy Markdown
Contributor

🎉 This pull request is included in a new release.

Release: v0.91.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[AW Top 10] 07 Ensure a terminal safe output on silent agent exits

4 participants