Skip to content

Skip failure issues for invalidated PR merge-ref checkouts - #65408

Merged
pelikhan merged 7 commits into
mainfrom
copilot/aw-10-fix-pr-gate-merge
Oct 3, 2026
Merged

pelikhan merged 7 commits into
mainfrom
copilot/aw-10-fix-pr-gate-merge

Conversation

Copilot AI commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

PR gate runs can fail checkout when a PR closes and its temporary merge ref disappears. Those runs should not create failure issues that look like agent regressions.

  • Classification: Skip failure-issue creation only when the PR is closed, the agent’s merge-ref checkout failed, and closure preceded the failed checkout.
  • Regression coverage: Add cases for invalidated checkouts and for failures that must remain reportable, including agent failures and checkouts that failed before closure.

Copilot AI and others added 2 commits October 3, 2026 19:16
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix PR gate merge to handle invalidation cascade Skip failure issues for invalidated PR merge-ref checkouts Oct 3, 2026
Copilot AI requested a review from pelikhan October 3, 2026 19:19
@pelikhan
pelikhan marked this pull request as ready for review October 3, 2026 19:42
Copilot AI balanced review requested due to automatic review settings October 3, 2026 19:42

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Missing API permissions and ambiguous step-name matching can respectively prevent or incorrectly trigger suppression.

Review effort: Balanced
Findings: 1 High severity · 1 Medium severity

Open (2)
What changed in this PR

Prevents failure issues when a closed PR invalidates its temporary merge-ref checkout.

Changes:

  • Detects checkout failures occurring after PR closure.
  • Skips failure-issue creation for classified runs.
  • Adds regression tests for reportable and suppressed failures.
File Description
actions/​setup/​js/​handle_agent_failure.cjs Adds invalidated checkout classification and suppression.
actions/​setup/​js/​handle_agent_failure.test.cjs Covers checkout timing, PR state, and failure types.

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

job.name === "agent" &&
job.conclusion === "failure" &&
job.steps?.some(step => {
if ((step.name !== "Checkout repository" && step.name !== "Checkout actions folder") || step.conclusion !== "failure" || !step.completed_at) {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Implemented in cb5b92a: classification now requires compiler-derived default-checkout ref eligibility and matches only the uniquely named generated checkout; unrelated and actions-folder checkout steps are excluded.


try {
const { owner, repo } = context.repo;
const { data: pr } = await github.rest.pulls.get({ owner, repo, pull_number: prNumber });

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Implemented in cb5b92a: conclusion jobs grant actions: read and pull-requests: read whenever failure-issue classification can run, with compiler permission coverage.

@github-actions

github-actions Bot commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

✅ Test Quality Sentinel completed test quality analysis.

Test Quality Sentinel skipped because pre-fetch PR data was unavailable: unable to fetch test file diff

🧪 Test quality analysis by Test Quality Sentinel

@github-actions

github-actions Bot commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

✅ PR Code Quality Reviewer completed the code quality review.

Preparing final PR review submission after separating temp-file writes from safe-output calls.

🔎 Code quality review by PR Code Quality Reviewer

@github-actions

github-actions Bot commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

✅ Ponytail Reviewer completed successfully!

Lean already. Ship.

Generated by Ponytail Reviewer for #65408

@github-actions

github-actions Bot commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer

@github-actions

github-actions Bot commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

✅ Design Decision Gate 🏗️ completed the design decision gate check. See the comment below for the result and any generated ADR draft.

Warning

Threat Detection Engine Failure — The analysis engine could not complete. This is a tooling failure, not a security finding.

What happened

The threat detection engine failed to produce results.

Review the workflow run logs for details.

🏗️ ADR gate enforced by Design Decision Gate 🏗️

@github-actions

github-actions Bot commented Oct 3, 2026

Copy link
Copy Markdown
Contributor

Comment Memory

reviewed_at: 2026-10-03T20:00:57.711+00:00
review_event: REQUEST_CHANGES
top_themes:
  - missing conclusion-job pull-requests read permission for pulls.get
  - checkout-step name heuristic can suppress unrelated failures
  - workflow-job pagination can miss the agent checkout failure
files_reviewed:
  - actions/setup/js/handle_agent_failure.cjs
  - actions/setup/js/handle_agent_failure.test.cjs
comment_count: 1

Note

This comment is managed by comment memory.

It stores persistent context for this thread in the code block at the top of this comment.
Edit only the text inside the backtick fences; workflow metadata and the footer are regenerated automatically.

Learn more about comment memory

🔎 Code quality review by PR Code Quality Reviewer · copilot · gpt54 · 35 AIC · ⌖ 7.06 AIC · ⊞ 20.2K · ◷
Comment /review to run again

@pelikhan

pelikhan commented Oct 3, 2026

Copy link
Copy Markdown
Collaborator

@copilot run pr-finished skill

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed with Impeccable harden + audit modes (bug-fix/edge-case change). Logic is clear and well-tested, but two blocking issues were already flagged by a prior Copilot review and remain unresolved in the current diff:

  1. Missing pull-requests: read permission on the conclusion job — isInvalidatedPRMergeCheckout calls github.rest.pulls.get, which will 403/throw in real compiled workflows, always falling into the catch and returning false. This makes the new skip logic a no-op in practice.
  2. Step-name matching isn't sufficiently specific — "Checkout repository" / "Checkout actions folder" names are reused for checkout steps with custom refs/repos, so an unrelated failed checkout could be misclassified as an invalidated merge-ref checkout and suppress a genuine failure.

See inline comments for details and suggested fixes.

🧵 Reviewed using Impeccable skills by Impeccable Skills Reviewer · copilot · sonnet50 · 46.4 AIC · ⌖ 14.2 AIC · ⊞ 8.1K


try {
const { owner, repo } = context.repo;
const { data: pr } = await github.rest.pulls.get({ owner, repo, pull_number: prNumber });

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This permissions gap means the new classifier will silently fail (and fall back to return false, i.e. still report failure as expected) in every compiled workflow, because github.rest.pulls.get and github.rest.actions.listJobsForWorkflowRun both require scopes the conclusion job doesn't currently request.

  • pulls.get needs pull-requests: read.
  • listJobsForWorkflowRun needs actions: read — this one is already granted when conclusionReportFailedJobsEnabled(data) is true (the default), but is otherwise missing.

Since computeConclusionJobPermissions in pkg/workflow/notify_comment.go was not updated to add pull-requests: read (or actions: read unconditionally), this classifier will always hit the catch block in real compiled workflows with default/least-privilege permissions, defeating the purpose of the change.

Please add a conditional pull-requests: read grant (and ensure actions: read is present) in computeConclusionJobPermissions whenever this classifier can run, mirroring the existing pattern for conclusionReportFailedJobsEnabled.

@copilot please address this.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Implemented in cb5b92a: the required actions: read and pull-requests: read scopes are now conditionally granted for failure-issue classification and covered by compiler tests.

job.name === "agent" &&
job.conclusion === "failure" &&
job.steps?.some(step => {
if ((step.name !== "Checkout repository" && step.name !== "Checkout actions folder") || step.conclusion !== "failure" || !step.completed_at) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The step-name match alone is not a reliable signal that this specific checkout targeted the PR merge ref. The generated "Checkout repository" step name is reused even when checkout.ref / checkout.repository overrides are configured elsewhere in the compiled workflow (see GenerateDefaultCheckoutStep in pkg/workflow/checkout_step_generator.go), and workflows can also emit custom checkout steps with overlapping names.

Combined with the missing pull-requests: read permission (see other comment), a failed checkout step named "Checkout repository" that is actually unrelated to the merge ref (e.g., a sparse/custom checkout of a different repo) could be misclassified as an invalidated merge-ref checkout and silently suppress a real, reportable failure.

Consider scoping the match more precisely — e.g., checking that the failed step belongs to the known default-checkout step and/or verifying the error output references the merge ref / "not found" — before suppressing failure reporting.

@copilot please address this.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Implemented in cb5b92a: the handler checks compiler-derived checkout eligibility and only matches the uniquely named generated default checkout, not cross-repository or overlapping common-name steps.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Skills-Based Review 🧠

Applied /diagnosing-bugs and /tdd — the test coverage is solid, but two unresolved review threads raise important risks that should be addressed before merge.

📋 Key Themes & Highlights

Key Themes

  • Missing permission wiring: isInvalidatedPRMergeCheckout() calls github.rest.pulls.get and github.rest.actions.listJobsForWorkflowRun, but computeConclusionJobPermissions in pkg/workflow/notify_comment.go was not updated to grant pull-requests: read. Most compiled conclusion jobs only get actions: read + issues: write (verified across the lock files) — so in practice these new API calls will 403 and silently fall through to return false, meaning the fix won't activate in deployed workflows. This is flagged in an existing Copilot review comment and confirmed by inspecting the compiled lock files.
  • Brittle step-name matching: matching on the literal strings "Checkout repository"/"Checkout actions folder" doesn't verify the step actually targeted the PR merge ref — checkout.ref/checkout.repository overrides reuse the same step name (per checkout_step_generator.go), so a genuine failure on an overridden checkout could be misclassified as "invalidated" and silently suppressed. Also flagged in an existing review comment.
  • Silent permission failures compound the gap: the catch block in the new function swallows 403s as a generic warning, making the permissions gap above hard to detect from workflow run logs (new comment posted on this PR).

Positive Highlights

  • ✅ Clear, well-scoped guard conditions (event type, ref shape, PR state, closure ordering) before classifying a checkout as invalidated.
  • ✅ Comprehensive regression suite — 6 cases covering happy path, pre-closure failures, non-checkout failures, both step names, non-PR context, and API errors — reads well as a spec.
  • ✅ Fails safe: any ambiguity or API error defaults to still reporting the failure, so this change cannot newly suppress a real agent failure (modulo the permissions gap making it a no-op).

Recommend resolving the permissions wiring (grant pull-requests: read/confirm actions: read in computeConclusionJobPermissions when this check can run) and tightening the step-name check before merging, since the fix is currently likely to be a no-op in production due to missing permissions.

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · copilot · sonnet50 · 112.7 AIC · ⌖ 14.6 AIC · ⊞ 10K
Comment /matt to run again

}
const completedAt = Date.parse(step.completed_at);
return Number.isFinite(completedAt) && closedAt <= completedAt;
})

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[/diagnosing-bugs] If listJobsForWorkflowRun or pulls.get fails due to missing pull-requests: read/actions: read scope (as flagged in the other review comment), the catch here silently swallows it as core.warning and returns false — meaning the existing bug this PR fixes (spurious failure issues on invalidated checkouts) will keep happening silently whenever the conclusion job lacks the required permission, with no clear signal pointing back to a permissions problem.

💡 Suggested improvement

Consider distinguishing a 403/permission error from other failures so the warning message is actionable, e.g.:

} catch (error) {
  const msg = getErrorMessage(error);
  if (/403|Resource not accessible/i.test(msg)) {
    core.warning(`Could not check PR merge-ref invalidation — missing permissions (grant pull-requests: read / actions: read on the conclusion job): ${msg}`);
  } else {
    core.warning(`Could not check PR merge-ref invalidation: ${msg}`);
  }
  return false;
}

This also makes the gap called out in the other review thread (missing computeConclusionJobPermissions wiring) easier to diagnose from workflow run logs once it manifests in production.

@copilot please address this.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Implemented in cb5b92a: 403 and Resource not accessible failures now warn with the required conclusion-job read scopes; other API failures retain the generic diagnostic.

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
@pelikhan

pelikhan commented Oct 3, 2026

Copy link
Copy Markdown
Collaborator

@copilot add logging, update safe outputs specs

Copilot AI and others added 3 commits October 3, 2026 20:44
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
…gate-merge

# Conflicts:
#	.github/workflows/pr-sous-chef.lock.yml

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
@pelikhan

pelikhan commented Oct 3, 2026

Copy link
Copy Markdown
Collaborator

@copilot resolve the merge conflicts on this branch.

Copilot AI commented Oct 3, 2026

Copy link
Copy Markdown
Contributor Author

@copilot run pr-finished skill

Ran the PR-finisher checks and replied to all five actionable review threads. Targeted checks and make lint pass; make test still fails on the existing CLI assertion expecting gpt-5.4 while the workflow uses gpt-5.5. The merge with main was resolved locally, but publishing it failed during the progress tool’s rebase (upload-pack: not our ref); GitHub now reports head 00f0a47, so I can’t confirm the merge commit is included. Review threads remain open because no resolution action is available, and CI needs a maintainer re-trigger.

Copilot AI commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor Author

@copilot run pr-finished skill

Ran PR-finisher checks and merged main with regenerated lock-file conflict resolution in 55d851e. Formatting, full lint, focused Go tests, 410 JS tests, and all 319 workflow compilations pass. Remaining blockers: pre-existing custom-Go-linter findings in the progress gate, five answered review threads that available tools cannot resolve, unavailable automated review model, and CodeQL timeout. CI on the pushed HEAD remains unverified.

@copilot add logging, update safe outputs specs

Added classifier debug diagnostics, logging regression coverage, and the safe-outputs/environment-variable contracts in 00f0a47. All 410 failure-handler tests and targeted compiler tests pass.

@pelikhan
pelikhan merged commit c7f5b05 into main Oct 3, 2026
1 check passed
@pelikhan
pelikhan deleted the copilot/aw-10-fix-pr-gate-merge branch October 3, 2026 21:08

Copilot AI commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor Author

@copilot add logging, update safe outputs specs

Added the diagnostic logging and updated both safe-output specification files in 00f0a47.

@copilot run pr-finished skill

Ran the PR-finisher checks. make fmt/make lint, make test-unit, and the targeted checkout/handler tests passed; make test still fails on the existing assertion expecting copilot/gpt-5.4 while the workflow specifies copilot/gpt-5.5. The PR was merged at 55d851e; final CodeQL timed out and the code-review runner could not load its configured model.

@github-actions

github-actions Bot commented Oct 5, 2026

Copy link
Copy Markdown
Contributor

🎉 This pull request is included in a new release.

Release: v0.91.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[AW Top 10] 09 Fix PR gate merge or close invalidation cascade

3 participants