Repository navigation
fix: add retry/backoff to Fetch open non-draft PR queue in pr-sous-chef
#43383
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 1 commit
3ab5fa7
f4605aa
e386bb6
c464e6f
054fcfd
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
…us-chef Wrap the `gh pr list` GraphQL call with a bounded retry loop (4 attempts, exponential backoff) that retries on transient 5xx/429/network errors and fails hard on deterministic errors. This prevents a single transient HTTP 502 from killing the entire run before the agent starts. Closes #43260 Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
- Loading branch information
There are no files selected for viewing
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -58,12 +58,42 @@ steps: | |
| filtered_last_comment_from_sous_chef=0 | ||
| filtered_cooldown=0 | ||
|
|
||
| gh pr list --repo "$EXPR_GITHUB_REPOSITORY" \ | ||
| --state open \ | ||
| --search "is:pr is:open -is:draft sort:updated-desc" \ | ||
| --limit 30 \ | ||
| --json number,title,url,headRefOid,headRefName,updatedAt,author,mergeStateStatus \ | ||
| > "$candidate_file" | ||
| # Fetch PR queue with retry + exponential backoff to survive transient GraphQL 5xx errors. | ||
| pr_list_max_attempts=4 | ||
| pr_list_attempt=0 | ||
| pr_list_success=false | ||
| until [ "$pr_list_success" = "true" ] || [ "$pr_list_attempt" -ge "$pr_list_max_attempts" ]; do | ||
| pr_list_attempt=$((pr_list_attempt + 1)) | ||
| set +e | ||
| pr_list_output=$(gh pr list --repo "$EXPR_GITHUB_REPOSITORY" \ | ||
| --state open \ | ||
| --search "is:pr is:open -is:draft sort:updated-desc" \ | ||
| --limit 30 \ | ||
| --json number,title,url,headRefOid,headRefName,updatedAt,author,mergeStateStatus \ | ||
| 2>&1) | ||
| pr_list_status=$? | ||
| set -e | ||
| if [ "$pr_list_status" -eq 0 ]; then | ||
| echo "$pr_list_output" > "$candidate_file" | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
💡 Suggested fixCapture stdout and stderr separately so only the clean JSON stdout lands in set +e
pr_list_stderr_tmp=$(mktemp)
pr_list_output=$(gh pr list --repo "$EXPR_GITHUB_REPOSITORY" \
--state open \
--search "is:pr is:open -is:draft sort:updated-desc" \
--limit 30 \
--json number,title,url,headRefOid,headRefName,updatedAt,author,mergeStateStatus \
2>"$pr_list_stderr_tmp")
pr_list_status=$?
pr_list_stderr=$(cat "$pr_list_stderr_tmp"; rm -f "$pr_list_stderr_tmp")
set -eThen use Why it matters: |
||
| pr_list_success=true | ||
| elif echo "$pr_list_output" | grep -qE "HTTP (5[0-9]{2}|429)|context deadline exceeded|connection reset|EOF"; then | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [/diagnosing-bugs] The bare 💡 Suggested fixAnchor elif echo "$pr_list_output" | grep -qE "HTTP (5[0-9]{2}|429)|context deadline exceeded|connection reset|unexpected EOF|read: EOF"; thenThe risk: an error message that incidentally contains @copilot please address this.
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [/diagnosing-bugs] If 💡 Suggested fixAdd an explicit guard for empty output: elif [ -z "$pr_list_output" ] || echo "$pr_list_output" | grep -qE "HTTP (5[0-9]{2}|429)|context deadline exceeded|connection reset|EOF"; thenAn empty output from a network command is almost always a transient connectivity/timeout issue, not a deterministic error, so it deserves a retry. @copilot please address this.
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Bare Consider tightening to:
@copilot please address this. |
||
| if [ "$pr_list_attempt" -lt "$pr_list_max_attempts" ]; then | ||
| backoff=$((2 ** pr_list_attempt)) | ||
| echo "Transient error on attempt $pr_list_attempt/$pr_list_max_attempts (exit $pr_list_status); retrying in ${backoff}s..." >&2 | ||
| echo "$pr_list_output" >&2 | ||
| sleep "$backoff" | ||
| fi | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [/diagnosing-bugs] When the final attempt hits a transient error, the inner 💡 Suggested improvementLog the transient error even on the last attempt so the failure message makes sense in context: elif echo "$pr_list_output" | grep -qE "..."; then
echo "Transient error on attempt $pr_list_attempt/$pr_list_max_attempts (exit $pr_list_status); ${backoff}s..." >&2
echo "$pr_list_output" >&2
if [ "$pr_list_attempt" -lt "$pr_list_max_attempts" ]; then
backoff=$((2 ** pr_list_attempt))
sleep "$backoff"
fiMove the logging outside the inner @copilot please address this.
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Silent exit on final transient failure: when the 4th attempt hits a transient error, the inner guard 💡 Suggested fixLog the final transient failure unconditionally, outside the elif echo "$pr_list_output" | grep -qE "HTTP (5[0-9]{2}|429)|context deadline exceeded|connection reset|EOF"; then
echo "Transient error on attempt $pr_list_attempt/$pr_list_max_attempts (exit $pr_list_status)" >&2
echo "$pr_list_output" >&2
if [ "$pr_list_attempt" -lt "$pr_list_max_attempts" ]; then
backoff=$((2 ** pr_list_attempt))
echo "Retrying in ${backoff}s..." >&2
sleep "$backoff"
fiWithout this, an operator looking at a 4-attempt-exhausted failure sees the post-loop message but zero context about what the last error actually was.
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Observability gap on the last attempt: when Suggestion — always log, only gate the sleep: elif echo "$pr_list_output" | grep -qE '...'; then
echo "Transient error on attempt $pr_list_attempt/$pr_list_max_attempts (exit $pr_list_status)" >&2
echo "$pr_list_output" >&2
if [ "$pr_list_attempt" -lt "$pr_list_max_attempts" ]; then
backoff=$((2 ** pr_list_attempt))
echo "Retrying in ${backoff}s..." >&2
sleep "$backoff"
fi
fi@copilot please address this. |
||
| else | ||
| echo "gh pr list failed (exit $pr_list_status):" >&2 | ||
| echo "$pr_list_output" >&2 | ||
| exit 1 | ||
| fi | ||
|
Comment on lines
+68
to
+97
|
||
| done | ||
| if [ "$pr_list_success" != "true" ]; then | ||
| echo "gh pr list failed after $pr_list_max_attempts attempts:" >&2 | ||
| echo "$pr_list_output" >&2 | ||
| exit 1 | ||
| fi | ||
|
|
||
| jq -n '[]' > "$eligible_file" | ||
|
|
||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[/tdd] This fix directly addresses a production outage (15 consecutive failures over ~8h) but ships with no regression test. A future degradation event could silently break the retry logic with no automated signal.
💡 What to test
Key scenarios that would prevent regressions:
gh pr listto returnHTTP 502for N attempts then succeed; assertpr_list_success=true.gh pr listto always returnHTTP 503; assert the script exits non-zero and the error is reported.gh pr listto return exit 1 witherror: HTTP 422; assert exactly one attempt and immediate exit.gh pr listto return exit 1 with empty output.Even a minimal shell-based test with
batsor a mockghbinary would document the expected contract.@copilot please address this.