Repository navigation
Skip failure issues for invalidated PR merge-ref checkouts - #65408
Conversation
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Missing API permissions and ambiguous step-name matching can respectively prevent or incorrectly trigger suppression.
Review effort: Balanced
Findings: 1
Open (2)
What changed in this PR
Prevents failure issues when a closed PR invalidates its temporary merge-ref checkout.
Changes:
- Detects checkout failures occurring after PR closure.
- Skips failure-issue creation for classified runs.
- Adds regression tests for reportable and suppressed failures.
| File | Description |
|---|---|
actions/setup/js/handle_agent_failure.cjs |
Adds invalidated checkout classification and suppression. |
actions/setup/js/handle_agent_failure.test.cjs |
Covers checkout timing, PR state, and failure types. |
💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.
| job.name === "agent" && | ||
| job.conclusion === "failure" && | ||
| job.steps?.some(step => { | ||
| if ((step.name !== "Checkout repository" && step.name !== "Checkout actions folder") || step.conclusion !== "failure" || !step.completed_at) { |
There was a problem hiding this comment.
Implemented in cb5b92a: classification now requires compiler-derived default-checkout ref eligibility and matches only the uniquely named generated checkout; unrelated and actions-folder checkout steps are excluded.
|
|
||
| try { | ||
| const { owner, repo } = context.repo; | ||
| const { data: pr } = await github.rest.pulls.get({ owner, repo, pull_number: prNumber }); |
There was a problem hiding this comment.
Implemented in cb5b92a: conclusion jobs grant actions: read and pull-requests: read whenever failure-issue classification can run, with compiler permission coverage.
|
✅ Test Quality Sentinel completed test quality analysis. Test Quality Sentinel skipped because pre-fetch PR data was unavailable: unable to fetch test file diff
|
|
✅ PR Code Quality Reviewer completed the code quality review. Preparing final PR review submission after separating temp-file writes from safe-output calls.
|
|
✅ Ponytail Reviewer completed successfully! Lean already. Ship.
|
|
🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅
|
|
✅ Design Decision Gate 🏗️ completed the design decision gate check. See the comment below for the result and any generated ADR draft. Warning Threat Detection Engine Failure — The analysis engine could not complete. This is a tooling failure, not a security finding. What happenedThe threat detection engine failed to produce results. Review the workflow run logs for details.
|
Comment MemoryNote This comment is managed by comment memory.It stores persistent context for this thread in the code block at the top of this comment.
|
|
@copilot run pr-finished skill |
There was a problem hiding this comment.
Reviewed with Impeccable harden + audit modes (bug-fix/edge-case change). Logic is clear and well-tested, but two blocking issues were already flagged by a prior Copilot review and remain unresolved in the current diff:
- Missing
pull-requests: readpermission on the conclusion job —isInvalidatedPRMergeCheckoutcallsgithub.rest.pulls.get, which will 403/throw in real compiled workflows, always falling into thecatchand returningfalse. This makes the new skip logic a no-op in practice. - Step-name matching isn't sufficiently specific — "Checkout repository" / "Checkout actions folder" names are reused for checkout steps with custom refs/repos, so an unrelated failed checkout could be misclassified as an invalidated merge-ref checkout and suppress a genuine failure.
See inline comments for details and suggested fixes.
🧵 Reviewed using Impeccable skills by Impeccable Skills Reviewer · copilot · sonnet50 · 46.4 AIC · ⌖ 14.2 AIC · ⊞ 8.1K
|
|
||
| try { | ||
| const { owner, repo } = context.repo; | ||
| const { data: pr } = await github.rest.pulls.get({ owner, repo, pull_number: prNumber }); |
There was a problem hiding this comment.
This permissions gap means the new classifier will silently fail (and fall back to return false, i.e. still report failure as expected) in every compiled workflow, because github.rest.pulls.get and github.rest.actions.listJobsForWorkflowRun both require scopes the conclusion job doesn't currently request.
pulls.getneedspull-requests: read.listJobsForWorkflowRunneedsactions: read— this one is already granted whenconclusionReportFailedJobsEnabled(data)is true (the default), but is otherwise missing.
Since computeConclusionJobPermissions in pkg/workflow/notify_comment.go was not updated to add pull-requests: read (or actions: read unconditionally), this classifier will always hit the catch block in real compiled workflows with default/least-privilege permissions, defeating the purpose of the change.
Please add a conditional pull-requests: read grant (and ensure actions: read is present) in computeConclusionJobPermissions whenever this classifier can run, mirroring the existing pattern for conclusionReportFailedJobsEnabled.
@copilot please address this.
There was a problem hiding this comment.
Implemented in cb5b92a: the required actions: read and pull-requests: read scopes are now conditionally granted for failure-issue classification and covered by compiler tests.
| job.name === "agent" && | ||
| job.conclusion === "failure" && | ||
| job.steps?.some(step => { | ||
| if ((step.name !== "Checkout repository" && step.name !== "Checkout actions folder") || step.conclusion !== "failure" || !step.completed_at) { |
There was a problem hiding this comment.
The step-name match alone is not a reliable signal that this specific checkout targeted the PR merge ref. The generated "Checkout repository" step name is reused even when checkout.ref / checkout.repository overrides are configured elsewhere in the compiled workflow (see GenerateDefaultCheckoutStep in pkg/workflow/checkout_step_generator.go), and workflows can also emit custom checkout steps with overlapping names.
Combined with the missing pull-requests: read permission (see other comment), a failed checkout step named "Checkout repository" that is actually unrelated to the merge ref (e.g., a sparse/custom checkout of a different repo) could be misclassified as an invalidated merge-ref checkout and silently suppress a real, reportable failure.
Consider scoping the match more precisely — e.g., checking that the failed step belongs to the known default-checkout step and/or verifying the error output references the merge ref / "not found" — before suppressing failure reporting.
@copilot please address this.
There was a problem hiding this comment.
Implemented in cb5b92a: the handler checks compiler-derived checkout eligibility and only matches the uniquely named generated default checkout, not cross-repository or overlapping common-name steps.
There was a problem hiding this comment.
Skills-Based Review 🧠
Applied /diagnosing-bugs and /tdd — the test coverage is solid, but two unresolved review threads raise important risks that should be addressed before merge.
📋 Key Themes & Highlights
Key Themes
- Missing permission wiring:
isInvalidatedPRMergeCheckout()callsgithub.rest.pulls.getandgithub.rest.actions.listJobsForWorkflowRun, butcomputeConclusionJobPermissionsinpkg/workflow/notify_comment.gowas not updated to grantpull-requests: read. Most compiled conclusion jobs only getactions: read+issues: write(verified across the lock files) — so in practice these new API calls will 403 and silently fall through toreturn false, meaning the fix won't activate in deployed workflows. This is flagged in an existing Copilot review comment and confirmed by inspecting the compiled lock files. - Brittle step-name matching: matching on the literal strings
"Checkout repository"/"Checkout actions folder"doesn't verify the step actually targeted the PR merge ref —checkout.ref/checkout.repositoryoverrides reuse the same step name (percheckout_step_generator.go), so a genuine failure on an overridden checkout could be misclassified as "invalidated" and silently suppressed. Also flagged in an existing review comment. - Silent permission failures compound the gap: the
catchblock in the new function swallows 403s as a generic warning, making the permissions gap above hard to detect from workflow run logs (new comment posted on this PR).
Positive Highlights
- ✅ Clear, well-scoped guard conditions (event type, ref shape, PR state, closure ordering) before classifying a checkout as invalidated.
- ✅ Comprehensive regression suite — 6 cases covering happy path, pre-closure failures, non-checkout failures, both step names, non-PR context, and API errors — reads well as a spec.
- ✅ Fails safe: any ambiguity or API error defaults to still reporting the failure, so this change cannot newly suppress a real agent failure (modulo the permissions gap making it a no-op).
Recommend resolving the permissions wiring (grant pull-requests: read/confirm actions: read in computeConclusionJobPermissions when this check can run) and tightening the step-name check before merging, since the fix is currently likely to be a no-op in production due to missing permissions.
🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · copilot · sonnet50 · 112.7 AIC · ⌖ 14.6 AIC · ⊞ 10K
Comment /matt to run again
| } | ||
| const completedAt = Date.parse(step.completed_at); | ||
| return Number.isFinite(completedAt) && closedAt <= completedAt; | ||
| }) |
There was a problem hiding this comment.
[/diagnosing-bugs] If listJobsForWorkflowRun or pulls.get fails due to missing pull-requests: read/actions: read scope (as flagged in the other review comment), the catch here silently swallows it as core.warning and returns false — meaning the existing bug this PR fixes (spurious failure issues on invalidated checkouts) will keep happening silently whenever the conclusion job lacks the required permission, with no clear signal pointing back to a permissions problem.
💡 Suggested improvement
Consider distinguishing a 403/permission error from other failures so the warning message is actionable, e.g.:
} catch (error) {
const msg = getErrorMessage(error);
if (/403|Resource not accessible/i.test(msg)) {
core.warning(`Could not check PR merge-ref invalidation — missing permissions (grant pull-requests: read / actions: read on the conclusion job): ${msg}`);
} else {
core.warning(`Could not check PR merge-ref invalidation: ${msg}`);
}
return false;
}This also makes the gap called out in the other review thread (missing computeConclusionJobPermissions wiring) easier to diagnose from workflow run logs once it manifests in production.
@copilot please address this.
There was a problem hiding this comment.
Implemented in cb5b92a: 403 and Resource not accessible failures now warn with the required conclusion-job read scopes; other API failures retain the generic diagnostic.
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
|
@copilot add logging, update safe outputs specs |
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
…gate-merge # Conflicts: # .github/workflows/pr-sous-chef.lock.yml Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
|
@copilot resolve the merge conflicts on this branch. |
Ran the PR-finisher checks and replied to all five actionable review threads. Targeted checks and |
Ran PR-finisher checks and merged main with regenerated lock-file conflict resolution in
Added classifier debug diagnostics, logging regression coverage, and the safe-outputs/environment-variable contracts in |
Added the diagnostic logging and updated both safe-output specification files in
Ran the PR-finisher checks. |
|
🎉 This pull request is included in a new release. Release: |


PR gate runs can fail checkout when a PR closes and its temporary merge ref disappears. Those runs should not create failure issues that look like agent regressions.