Repository navigation
Conversation
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
This PR updates the JavaScript safe-outputs pipeline to better handle mixed-result batches (successes alongside item-level failures), improves accuracy of “skipped” semantics for non-fatal target resolution misses, and adds earlier emit-time validation so unsupported target configurations fail fast with actionable MCP errors.
Changes:
- Add emit-time target validation in
safe_outputs_handlersto rejecttarget: "triggering"(andtarget: "*"without an explicit target identifier) when the workflow event cannot imply an issue/PR target. - Mark non-fatal target resolution misses as
skippedin the shared processor result to align withresolveTarget(... shouldFail: false)behavior. - Tolerate fatal item failures as warnings when at least one safe output in the batch succeeded, avoiding job-fatal outcomes for mixed-result batches.
Show a summary per file
| File | Description |
|---|---|
| actions/setup/js/safe_outputs_handlers.test.cjs | Adds coverage for emit-time rejection/allow paths for triggering and * targets outside issue/PR events. |
| actions/setup/js/safe_outputs_handlers.cjs | Implements emit-time target validation and returns MCP errors before appending safe outputs. |
| actions/setup/js/safe_output_processor.test.cjs | Adds a test asserting unsupported triggering target is treated as skipped (non-fatal) in non-issue/PR events. |
| actions/setup/js/safe_output_processor.cjs | Returns skipped when resolveTarget indicates a non-fatal miss (shouldFail: false). |
| actions/setup/js/safe_output_handler_manager.test.cjs | Adds unit tests for the new “tolerate fatal failures when something succeeded” behavior. |
| actions/setup/js/safe_output_handler_manager.cjs | Adds shouldTolerateFatalFailures and adjusts job failure behavior to keep mixed-result batches green-with-warning. |
Copilot's findings
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Files reviewed: 6/6 changed files
- Comments generated: 4
| const eventName = context?.eventName || "unknown"; | ||
| if (isIssueOrPullRequestContext(eventName, context?.payload)) { |
| const fileInfo = writeLargeContentToFile(largeContent); | ||
| entry[largeFieldName] = `[Content too large, saved to file: ${fileInfo.filename}]`; | ||
| const targetValidationResponse = maybeRejectEmitBoundaryTarget(entry); | ||
| if (targetValidationResponse) return targetValidationResponse; | ||
| appendSafeOutput(entry); |
| return `Configured target for '${normalizedType}' is "*" but the current event '${eventName}' is not an issue or pull request event, and this MCP call did not include an explicit target number. Include item_number, issue_number, or pull_request_number in the tool call, use an explicit numeric target, or run the workflow from an issue or pull request event.`; | ||
| } | ||
|
|
||
| return `Configured target for '${normalizedType}' is "triggering" but the current event '${eventName}' is not an issue or pull request event. Use an explicit numeric target, switch to target "*" with an item_number/issue_number/pull_request_number in the tool call, or run the workflow from an issue or pull request event.`; | ||
| }; |
| if (tolerateFatalFailures) { | ||
| core.warning(`${failureCount} message(s) were skipped because ${successCount} other message(s) succeeded`); | ||
| } else { |
|
✅ PR Code Quality Reviewer completed the code quality review. |
|
🧪 Test Quality Sentinel completed test quality analysis. |
|
✅ Design Decision Gate 🏗️ completed the design decision gate check. No ADR enforcement needed: PR #36528 does not have the 'implementation' label and has 0 new lines of code in business logic directories (≤100 threshold). No custom design-gate config present. |
|
🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅ |
🧪 Test Quality Sentinel Report✅ Test Quality Score: 87/100 — Excellent
📊 Metrics & Test Classification (10 tests analyzed)
Test Classification Details
Language SupportTests analyzed:
Verdict
📖 Understanding Test ClassificationsDesign Tests (High Value) verify what the system does:
Implementation Tests (Low Value) verify how the system does it:
Goal: Shift toward tests that describe the system's behavioral contract — the promises it makes to its users and collaborators.
|
There was a problem hiding this comment.
Non-blocking observations — no correctness blockers, but four issues worth addressing.
### Findings summary
1. Redundant partitionFailureResults call + inconsistent null-safety (safe_output_handler_manager.cjs line ~580)
shouldTolerateFatalFailures recomputes partitionFailureResults that main() already computed, and uses r?.success while the sibling filter uses r.success. Pass pre-computed successCount and fatalFailures as parameters instead.
2. "were skipped" wording for fatal failures (safe_output_handler_manager.cjs line ~1520)
Fatal failures being described as "skipped" conflates them with intentionally-skipped items and will confuse users reading logs. Use "failed but were tolerated" or similar.
3. skippedHandlerResults swallows reason-carrying skips silently (safe_output_handler_manager.cjs line ~1479)
Dropping !r.reason means processor-level skips (with explicit error reasons) are now silently counted under "Skipped (no context or limit reached)" without surfacing the reason. The reasons should be logged.
4. comment_id/thread_id/review_id as bypass gates in hasExplicitTargetNumber (safe_outputs_handlers.cjs line ~160)
These are sub-resource IDs, not item/PR/issue numbers. Using them to bypass boundary validation can silently succeed at emit time but fail at routing time.
🔎 Code quality review by PR Code Quality Reviewer · sonnet46 199.3K
| function shouldTolerateFatalFailures(results) { | ||
| const successCount = results.filter(r => r?.success).length; | ||
| const { fatalFailures } = partitionFailureResults(results); | ||
| return successCount > 0 && fatalFailures.length > 0; |
There was a problem hiding this comment.
shouldTolerateFatalFailures re-invokes partitionFailureResults redundantly, creating a subtle consistency risk.
💡 Details
main() already calls partitionFailureResults(processingResult.results) at line 1469 and stores the result in fatalFailures. shouldTolerateFatalFailures then calls partitionFailureResults a second time on the same array. This is wasteful on large result sets, and more importantly, the outer successCount (computed on line 1468 with r.success) and the inner successCount (computed inside the function with r?.success) use different null-safety, meaning they could theoretically diverge on malformed results.
Pass the pre-computed values in instead:
function shouldTolerateFatalFailures(successCount, fatalFailures) {
return successCount > 0 && fatalFailures.length > 0;
}
// call site:
const tolerateFatalFailures = shouldTolerateFatalFailures(successCount, fatalFailures);| if (failureCount > 0) { | ||
| core.warning(`${failureCount} message(s) failed to process`); | ||
| if (tolerateFatalFailures) { | ||
| core.warning(`${failureCount} message(s) were skipped because ${successCount} other message(s) succeeded`); |
There was a problem hiding this comment.
"were skipped" is inaccurate language for fatal failures — it obscures the real problem from users reading logs.
💡 Details
These are fatalFailures — items that failed to process. Calling them "skipped" conflates them with intentionally-skipped items (e.g. standalone step handlers, limit-reached items) and makes it harder to diagnose a real failure. Suggested alternative:
core.warning(`${failureCount} message(s) failed but were tolerated because ${successCount} other message(s) succeeded`);This makes it clear that failure occurred, rather than suggesting the messages were benignly bypassed.
| const skippedHandlerResults = processingResult.results.filter( | ||
| r => r.skipped && !r.deferred && !r.cancelled && r.reason !== "Handled by standalone step" && r.reason !== "Handled by custom safe output job" | ||
| ); | ||
|
|
There was a problem hiding this comment.
skippedHandlerResults now silently absorbs items with non-null reason, mislabeling them as "no context or limit reached".
💡 Details
The old filter was r.skipped && !r.reason && !r.deferred && !r.cancelled. The new filter drops !r.reason and instead explicitly excludes two specific reason strings. This means any skipped item that carries a reason OTHER than those two strings — for example, items returned by safe_output_processor.cjs with { success: false, skipped: true, reason: targetResult.error } — now falls into skippedHandlerResults and is logged under "Skipped (no context or limit reached)". The actual error reason is never surfaced.
At minimum, the summary log should print the reasons for items in this bucket:
if (skippedHandlerResults.length > 0) {
core.info(`Skipped (no context or limit reached): ${skippedHandlerResults.length}`);
const reasonGroups = skippedHandlerResults.map(r => ` - ${r.type}: ${r.reason || 'no reason'}`);
core.info(reasonGroups.join('\n'));
}Alternatively, keep the !r.reason condition and add a separate bucket for reason-carrying skips.
| "discussion_number", | ||
| "comment_id", | ||
| "thread_id", | ||
| "review_id", |
There was a problem hiding this comment.
comment_id, thread_id, and review_id are entity IDs, not target/item numbers — including them in hasExplicitTargetNumber can produce false positives.
💡 Details
hasExplicitTargetNumber is used to decide whether a wildcard (*) target is safe to emit because "the agent provided an explicit target". But comment_id, thread_id, and review_id identify specific sub-resources within an issue or PR, not the issue/PR itself. An agent that passes only comment_id: 456 with no issue_number or pull_request_number will bypass boundary validation, but the downstream routing will still have no issue/PR context to resolve against.
Suggested fix: remove these three fields from the candidate list. If routing for comment-level tools genuinely needs comment IDs to resolve the parent, that should be an explicit lookup in the handler, not a bypass gate.
const candidateFields = [
"item_number",
"issue_number",
"pull_request_number",
"pr_number",
"pr",
"pull_number",
"discussion_number",
// comment_id, thread_id, review_id removed — not item/target numbers
];There was a problem hiding this comment.
Skills-Based Review 🧠
Applied /diagnose and /tdd — requesting changes on one correctness bug and a handful of observability/consistency gaps.
📋 Key Themes & Highlights
Blocking Issue
- Leaked temp file (
safe_outputs_handlers.cjsline 236–238):writeLargeContentToFileruns beforemaybeRejectEmitBoundaryTarget. If target validation fails the early return leaves an orphaned file on disk. Target validation must move before any side effects.
Non-Blocking but Worth Fixing
comment_id/thread_id/review_idinhasExplicitTargetNumber: child-resource IDs aren't top-level issue/PR targets; they allowtarget: "*"validation to be bypassed incorrectly, and the error message doesn't mention them.- Tolerated failures lose their detail lines:
failedItemLinesis only used in the hard-fail path — operators can't see which items failed when tolerance kicks in. skippedHandlerResultsfilter change broadens scope silently: processor-level target-miss skips (with areasonstring) now land in theSkipped (no context or limit reached)bucket, which is a misleading label.- Warning wording
"were skipped"conflicts with existingskipped: truesemantics: items that failed-but-were-tolerated are not the same as items that were intentionally skipped.
Positive Highlights
- ✅ Excellent test coverage for new paths: 115 net test lines across three files.
- ✅
validateEmitBoundaryTargetlogic is clean and the error messages are genuinely actionable for agent self-correction. - ✅ The
skipped: !targetResult.shouldFailpropagation fix insafe_output_processor.cjsis minimal and correct. - ✅ Good use of optional chaining (
r?.success,entry?.type) defensively throughout.
🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · sonnet46 2.2M
| "discussion_number", | ||
| "comment_id", | ||
| "thread_id", | ||
| "review_id", |
There was a problem hiding this comment.
[/diagnose] comment_id, thread_id, and review_id are child-resource identifiers, not top-level issue/PR targets. An agent emitting comment_id: 123 on a target: "*" create_issue call would bypass this validation even though no issue/PR target is present. The error message also only mentions item_number, issue_number, pull_request_number — the three extra fields are invisible to the agent trying to self-correct.
💡 Suggestion
Split the field set: keep item_number, issue_number, pull_request_number, pr_number, pr, pull_number, and discussion_number as valid bypass-fields, and remove comment_id, thread_id, and review_id. Update the error message to list every accepted field so agents know exactly what to include.
| } else { | ||
| core.warning(`${failureCount} message(s) failed to process`); | ||
| } | ||
| const failedItemLines = fatalFailures.map(r => ` - ${r.type}: ${r.error || "Unknown error"}`); |
There was a problem hiding this comment.
[/diagnose] failedItemLines is computed here but only used in the !tolerateFatalFailures branch below. When failures are tolerated, the warning on line 1520 shows only a count — the operator loses visibility into which safe outputs failed and why.
💡 Suggestion
Log the detail lines as a separate core.warning even in the tolerate path:
if (tolerateFatalFailures) {
core.warning(`${failureCount} message(s) were tolerated (${successCount} succeeded): \n${failedItems}`);
} else {
core.warning(`${failureCount} message(s) failed to process`);
core.setFailed(`${failureCount} safe output(s) failed:\n${failedItems}`);
}| const skippedNoHandlerResults = processingResult.results.filter(r => !r.success && !r.skipped && r.error?.includes("No handler loaded")); | ||
| const skippedHandlerResults = processingResult.results.filter(r => r.skipped && !r.reason && !r.deferred && !r.cancelled); | ||
| const skippedHandlerResults = processingResult.results.filter( | ||
| r => r.skipped && !r.deferred && !r.cancelled && r.reason !== "Handled by standalone step" && r.reason !== "Handled by custom safe output job" |
There was a problem hiding this comment.
[/diagnose] The updated filter now includes processor-level target-miss skips (which have reason set to the resolution error message). These will be logged under the Skipped (no context or limit reached) label — misleading for target-configuration issues.
💡 Suggestion
Add a dedicated bucket for processor-level skips so the summary label is accurate:
const skippedTargetResults = processingResult.results.filter(
r => r.skipped && !r.deferred && !r.cancelled && r.reason && ![
"Handled by standalone step",
"Handled by custom safe output job"
].includes(r.reason)
);
const skippedHandlerResults = processingResult.results.filter(
r => r.skipped && !r.deferred && !r.cancelled && !r.reason
);Then log skippedTargetResults with a distinct label like Skipped (target not resolvable).
| if (failureCount > 0) { | ||
| core.warning(`${failureCount} message(s) failed to process`); | ||
| if (tolerateFatalFailures) { | ||
| core.warning(`${failureCount} message(s) were skipped because ${successCount} other message(s) succeeded`); |
There was a problem hiding this comment.
[/tdd] The warning text says "were skipped" but these items genuinely failed — they're only tolerated, not skipped. Calling them "skipped" is inconsistent with how skipped: true is already used elsewhere (intentional non-fatal target misses). Consider wording like "failed but were tolerated" to keep the vocabulary consistent and avoid misleading operators who grep for skipped in logs.
| it("does not tolerate fatal failures when nothing succeeded", () => { | ||
| expect( | ||
| shouldTolerateFatalFailures([ | ||
| { type: "update_issue", success: false, error: "Target is invalid" }, |
There was a problem hiding this comment.
[/tdd] Two good cases, but the all-success batch (no failures at all) is not tested. shouldTolerateFatalFailures should return false when fatalFailures.length === 0, and a test would lock that in.
💡 Suggested test
it("does not tolerate when there are no failures", () => {
expect(
shouldTolerateFatalFailures([
{ type: "create_issue", success: true },
{ type: "update_issue", success: true },
])
).toBe(false);
});|
|
|
@copilot review all comments and review comments. Commit and apply copilot review skill. |
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Addressed the unresolved review feedback in commits Implemented fixes:
Validation run:
Remaining blockers after this pass: none identified in the in-scope review comments. The latest automated review only surfaced two non-blocking maintainability suggestions. |
|
Please also refresh the branch and rerun checks once the feedback is resolved.
|
safe_outputstreated any failed item as job-fatal, even when the agent produced other valid actions in the same batch. This change shifts mixed-result batches to success-with-warning and rejects unsupportedtarget: triggering/target: "*"configurations earlier at the MCP emit boundary for non-issue/non-PR events.Partial-failure tolerance
Skip semantics for non-fatal target misses
skipped, not generic failures.resolveTarget(... shouldFail: false)behavior and keeps summaries/counting accurate.Earlier MCP target validation
target: "triggering"outrighttarget: "*"unless the emitted item includes an explicit target numberSummary behavior
safe_outputsjob.Example of the new emit-time rejection path:
{ "result": "error", "error": "Configured target for 'update_issue' is \"triggering\" but the current event 'schedule' is not an issue or pull request event. Use an explicit numeric target, switch to target \"*\" with an item_number/issue_number/pull_request_number in the tool call, or run the workflow from an issue or pull request event." }