Fix Bun backtest flake: retry cells on vex-step transport errors - #565
Merged
Mikola Lysenko (mikolalysenko) merged 1 commit intoOct 2, 2026
Merged
Conversation
The Bun backtest retries a failed cell from a clean tree when the row carries an explicit transport error, but the manifest-less VEX steps dropped the evidence: each vex run kept only exit/skip/attested, so a "Could not fetch patch <uuid>: ... error sending request" warning was discarded, and the checkout's `bun install --frozen-lockfile` output (bun's "error: ConnectionRefused downloading tarball ...") never reached the row. A patch-API or patch.socket.dev blip during these steps failed the whole job (vexLedgersDeleted, vexCheckoutPatchedBytes) instead of retrying the cell. Keep each vex run's warning details and the checkout install's transport-error lines in the row so the existing, narrowly scoped retry sees them. Non-transport failures (404, frozen-lock drift, attestation mismatches) still fail at once. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WLdPQLaBvre5kBJ8iD148S
Mikola Lysenko (mikolalysenko)
added
the
ci-janitor
Collaborator
Author
|
bugbot run Generated by Claude Code |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 61ece5f. Configure here.
This was referenced Oct 2, 2026
Tanmay Singla (Tanmay182003)
approved these changes
Oct 2, 2026
Mikola Lysenko (mikolalysenko)
deleted the
ci-janitor/bun-vex-transport-retry
branch
October 2, 2026 16:14
Mikola Lysenko (mikolalysenko)
added
the
Ready for review
Collaborator
Author
|
[burn-down agent] Labeled Ready for review at
Slack announcement not sent: the Slack connector in this session has no send-message tool. Generated by Claude Code |
This was referenced Oct 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Bun patch compatibilityis the slowest compatibility workflow (median ~22 min wall-clock per run) and had 3 re-runs of the same SHA in the last ~22 h (out of 68 runs). In each one, the first attempt'snativeleg failed one cell on a manifest-less VEX check:already-vendored-workspace hosted→vexLedgersDeleted(same run: ~15 cells hiterror sending request for url (https://patches-api.socket.dev/patch/batch) ... Connection reset by peer)workspace-get-search vendored→vexLedgersDeletedcrlf hosted→vexCheckoutPatchedBytesNone of these cells was retried, even though the harness already has a fresh-cell retry (
retry_network_cell, 3 tries) for explicit transport failures. Every re-run of the same SHA passed.Root cause
retry_network_cellonly retries whenhas_transport_failure(row)finds a transport error string in the row. The VEX block threw away the evidence:vex()stored onlyexit/skip/attested. A record fetch that fails in transport surfaces only in the envelope'swarnings[](Could not fetch patch <uuid>: Network error: error sending request for url (...)), and the skip code (record_unavailable) is the same for every cause.vexLedgersDeletedis the step that has to fetch the record from the patch API, because its ledgers were deleted.bun install --frozen-lockfileoutput (bun reportserror: ConnectionRefused downloading tarball <spec>/error: GET <url> - 5xx) went tovex-install.logand never reached the row.So a patch API or patch.socket.dev blip during these steps failed the job for good, and the only way out was a manual re-run of the whole ~22-minute workflow.
Fix
scripts/backtest-bun.pyonly:warnings[].detailstrings inrow['vex'][label]['warnings'].row['vexInstallTransport']).has_transport_failurealso recognises bun's own fetch-failure lines. These are connection errors (Connection*,FailedToOpenSocket,Timeout"downloading …") andGET <url> - 5xx.The retry stays as narrow as before. It only runs on an explicit transport error or a 5xx. A 404, a frozen-lock drift, or an attestation mismatch still fails the cell immediately. No timeouts were raised and no checks were removed or weakened.
Proof
All checks were run locally against the CLI built from this tree, with
patches-apipointed at a closed port:vex --jsonon a hosted-wired checkout exits 1 withwarnings: ["Could not fetch patch 3b1f…: Network error: error sending request for url (http://127.0.0.1:1/patch/view/3b1f…): client error (Connect): tcp connect error: Connection refused (os error 111)"]. Under the old row shapehas_transport_failurereturned False; under the new one it returns True.error: ConnectionRefused downloading tarball minimist@http://127.0.0.1:9/..., andbun_transport_failurespicks it up. AGET … - 503line matches;GET … - 404andlockfile had changes, but lockfile is frozendo not.retry_network_cellwith a row that failsvexLedgersDeletedon a connection reset and then passes. It retried once and returnedpassed=True, keepingnetworkRetryAttempts.ruff check --select F,E9shows the same single pre-existing finding as on main.This PR touches
scripts/backtest-bun*.py, so it triggers the full Bun workflow; that run is the end-to-end check.Where tests run
Unchanged. No test, cell or job was removed or moved, and required-check names are untouched.
🤖 Generated with Claude Code
https://claude.ai/code/session_01WLdPQLaBvre5kBJ8iD148S
Generated by Claude Code
Note
Low Risk
Changes only the Bun backtest harness’s failure detection and result row shape; no product CLI or runtime behavior.
Overview
Reduces flaky failures in the long-running Bun compatibility backtest by letting the existing fresh-cell retry (
retry_network_cell) see transport errors that used to be dropped during manifest-less VEX checks.has_transport_failurenow treats bun install fetch failures the same as CLI patch-API errors (connection/timeouts while downloading tarballs,GET … - 5xx). A newbun_transport_failureshelper extracts those lines from install output.The VEX checkout
bun installoutput is captured intorow['vexInstallTransport'], and eachvex()sub-step keepswarnings[].detailonrow['vex'][label]so patch-record fetch blips (e.g. “Could not fetch patch …”) are visible in the row. Retry behavior stays narrow: only explicit transport/5xx signals trigger a retry; 404s, lock drift, and attestation mismatches still fail immediately.Reviewed by Cursor Bugbot for commit 61ece5f. Configure here.
Generated by Claude Code