Skip to content
Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix: add retry/backoff to Fetch open non-draft PR queue step in pr-so…
…us-chef

Wrap the `gh pr list` GraphQL call with a bounded retry loop (4 attempts,
exponential backoff) that retries on transient 5xx/429/network errors and
fails hard on deterministic errors. This prevents a single transient HTTP 502
from killing the entire run before the agent starts.

Closes #43260

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
  • Loading branch information
Copilot and pelikhan authored Jul 4, 2026
commit e386bb60a70a496e0df6ab87e9735e0a4f86de4f
44 changes: 37 additions & 7 deletions .github/workflows/pr-sous-chef.lock.yml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

42 changes: 36 additions & 6 deletions .github/workflows/pr-sous-chef.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,12 +58,42 @@ steps:
filtered_last_comment_from_sous_chef=0
filtered_cooldown=0

gh pr list --repo "$EXPR_GITHUB_REPOSITORY" \
--state open \
--search "is:pr is:open -is:draft sort:updated-desc" \
--limit 30 \
--json number,title,url,headRefOid,headRefName,updatedAt,author,mergeStateStatus \
> "$candidate_file"
# Fetch PR queue with retry + exponential backoff to survive transient GraphQL 5xx errors.
pr_list_max_attempts=4
pr_list_attempt=0
pr_list_success=false
until [ "$pr_list_success" = "true" ] || [ "$pr_list_attempt" -ge "$pr_list_max_attempts" ]; do

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[/tdd] This fix directly addresses a production outage (15 consecutive failures over ~8h) but ships with no regression test. A future degradation event could silently break the retry logic with no automated signal.

💡 What to test

Key scenarios that would prevent regressions:

  1. Transient error → retries up to max_attempts — mock gh pr list to return HTTP 502 for N attempts then succeed; assert pr_list_success=true.
  2. All attempts exhausted — mock gh pr list to always return HTTP 503; assert the script exits non-zero and the error is reported.
  3. Deterministic error fails fast — mock gh pr list to return exit 1 with error: HTTP 422; assert exactly one attempt and immediate exit.
  4. Empty output fails fast (edge case from the grep guard) — mock gh pr list to return exit 1 with empty output.

Even a minimal shell-based test with bats or a mock gh binary would document the expected contract.

@copilot please address this.

pr_list_attempt=$((pr_list_attempt + 1))
set +e
pr_list_output=$(gh pr list --repo "$EXPR_GITHUB_REPOSITORY" \
--state open \
--search "is:pr is:open -is:draft sort:updated-desc" \
--limit 30 \
--json number,title,url,headRefOid,headRefName,updatedAt,author,mergeStateStatus \
2>&1)
pr_list_status=$?
set -e
if [ "$pr_list_status" -eq 0 ]; then
echo "$pr_list_output" > "$candidate_file"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2>&1 contaminates candidate_file on success: pr_list_output contains merged stdout+stderr, so any warning gh emits to stderr on a successful run silently corrupts the JSON written to candidate_file.

💡 Suggested fix

Capture stdout and stderr separately so only the clean JSON stdout lands in candidate_file:

set +e
pr_list_stderr_tmp=$(mktemp)
pr_list_output=$(gh pr list --repo "$EXPR_GITHUB_REPOSITORY" \
  --state open \
  --search "is:pr is:open -is:draft sort:updated-desc" \
  --limit 30 \
  --json number,title,url,headRefOid,headRefName,updatedAt,author,mergeStateStatus \
  2>"$pr_list_stderr_tmp")
pr_list_status=$?
pr_list_stderr=$(cat "$pr_list_stderr_tmp"; rm -f "$pr_list_stderr_tmp")
set -e

Then use "${pr_list_output}${pr_list_stderr}" for the transient-error grep, and $pr_list_output alone when writing to $candidate_file.

Why it matters: gh routinely writes upgrade notices, deprecation warnings, and experimental-feature advisories to stderr while still exiting 0. The original code (gh pr list ... > "$candidate_file") was safe because stdout alone went to the file. The new code merges stderr into the captured variable and then writes it unconditionally into the JSON file. Any non-empty stderr turns the JSON into garbage that jq will reject downstream — yet the step shows no error until then.

pr_list_success=true
elif echo "$pr_list_output" | grep -qE "HTTP (5[0-9]{2}|429)|context deadline exceeded|connection reset|EOF"; then

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[/diagnosing-bugs] The bare EOF token matches any error message containing that substring (e.g. Unexpected end-of-file, JSON parse errors, script errors), which could cause the code to silently retry what is actually a deterministic failure.

💡 Suggested fix

Anchor EOF to reduce false positives — for example:

elif echo "$pr_list_output" | grep -qE "HTTP (5[0-9]{2}|429)|context deadline exceeded|connection reset|unexpected EOF|read: EOF"; then

The risk: an error message that incidentally contains EOF would be retried up to 3 more times before failing, wasting ~14s of CI time and obscuring the root cause.

@copilot please address this.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[/diagnosing-bugs] If gh pr list exits with a non-zero status but produces no output (e.g., hard kill, OOM, runner timeout), $pr_list_output is empty — the grep -qE match fails silently and the code takes the else branch, exiting immediately with a generic error instead of retrying.

💡 Suggested fix

Add an explicit guard for empty output:

elif [ -z "$pr_list_output" ] || echo "$pr_list_output" | grep -qE "HTTP (5[0-9]{2}|429)|context deadline exceeded|connection reset|EOF"; then

An empty output from a network command is almost always a transient connectivity/timeout issue, not a deterministic error, so it deserves a retry.

@copilot please address this.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bare EOF match is too broad: the pattern EOF will match any output line containing that substring (e.g., "EOF in configuration", "PROOF", "SIZEOF"). This could cause legitimate non-transient errors to be silently retried rather than fail-fast.

Consider tightening to:

  • \bEOF\b (word boundary) — avoids substring false positives
  • or unexpected EOF — matches only the Go net/http error text that actually indicates a truncated response

@copilot please address this.

if [ "$pr_list_attempt" -lt "$pr_list_max_attempts" ]; then
backoff=$((2 ** pr_list_attempt))
echo "Transient error on attempt $pr_list_attempt/$pr_list_max_attempts (exit $pr_list_status); retrying in ${backoff}s..." >&2
echo "$pr_list_output" >&2
sleep "$backoff"
fi

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[/diagnosing-bugs] When the final attempt hits a transient error, the inner if [ "$pr_list_attempt" -lt "$pr_list_max_attempts" ] is false — so no log message is printed and no sleep is needed. But the fi just falls through, the loop condition fails, and control lands on the post-loop failure message. This is correct but the last transient error output is only shown in the post-loop message, which makes it hard to tell whether the last attempt was transient or not.

💡 Suggested improvement

Log the transient error even on the last attempt so the failure message makes sense in context:

elif echo "$pr_list_output" | grep -qE "..."; then
  echo "Transient error on attempt $pr_list_attempt/$pr_list_max_attempts (exit $pr_list_status); ${backoff}s..." >&2
  echo "$pr_list_output" >&2
  if [ "$pr_list_attempt" -lt "$pr_list_max_attempts" ]; then
    backoff=$((2 ** pr_list_attempt))
    sleep "$backoff"
  fi

Move the logging outside the inner if so it fires on every transient hit, including the last.

@copilot please address this.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Silent exit on final transient failure: when the 4th attempt hits a transient error, the inner guard [ "$pr_list_attempt" -lt "$pr_list_max_attempts" ] is false, so the if block is skipped — no log is emitted, no sleep, the loop simply ends, and the post-loop failure handler fires with no diagnostic output showing whether the last error was transient or not.

💡 Suggested fix

Log the final transient failure unconditionally, outside the if:

elif echo "$pr_list_output" | grep -qE "HTTP (5[0-9]{2}|429)|context deadline exceeded|connection reset|EOF"; then
  echo "Transient error on attempt $pr_list_attempt/$pr_list_max_attempts (exit $pr_list_status)" >&2
  echo "$pr_list_output" >&2
  if [ "$pr_list_attempt" -lt "$pr_list_max_attempts" ]; then
    backoff=$((2 ** pr_list_attempt))
    echo "Retrying in ${backoff}s..." >&2
    sleep "$backoff"
  fi

Without this, an operator looking at a 4-attempt-exhausted failure sees the post-loop message but zero context about what the last error actually was.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Observability gap on the last attempt: when pr_list_attempt equals pr_list_max_attempts, the transient-error branch is entered but the inner if [ "$pr_list_attempt" -lt "$pr_list_max_attempts" ] guard is false, so no diagnostic is logged and no exit occurs. The loop silently ends and the post-loop if catches it. This makes log triage harder because the retry count message and the final error output are separated, with nothing indicating the last attempt was also transient.

Suggestion — always log, only gate the sleep:

elif echo "$pr_list_output" | grep -qE '...'; then
  echo "Transient error on attempt $pr_list_attempt/$pr_list_max_attempts (exit $pr_list_status)" >&2
  echo "$pr_list_output" >&2
  if [ "$pr_list_attempt" -lt "$pr_list_max_attempts" ]; then
    backoff=$((2 ** pr_list_attempt))
    echo "Retrying in ${backoff}s..." >&2
    sleep "$backoff"
  fi
fi

@copilot please address this.

else
echo "gh pr list failed (exit $pr_list_status):" >&2
echo "$pr_list_output" >&2
exit 1
fi
Comment on lines +68 to +97
done
if [ "$pr_list_success" != "true" ]; then
echo "gh pr list failed after $pr_list_max_attempts attempts:" >&2
echo "$pr_list_output" >&2
exit 1
fi

jq -n '[]' > "$eligible_file"

Expand Down
Loading