Fix Poetry matrix flakes on PyPI/patch API transport blips - #596
Mikola Lysenko (mikolalysenko) wants to merge 2 commits into
Conversation
The Poetry matrix runs against production PyPI and patch.socket.dev with no transport retry, unlike the Pipenv, vlt and Bun harnesses. In the last ~8.5 days 33 runs failed on just one or two random legs (different versions, OSes, modes and checks each time), twice on main today, and the one failure whose log shows its cause is a Poetry ConnectionError fetching urllib3. Re-run a case from a fresh directory, at most three attempts, when its error text or logs show a transport failure: requests/urllib3 connection errors, pip's "too many 5xx", or the CLI's request error, patch API 5xx or exhausted 429 retry. Functional failures are never retried. Failed attempts' logs are kept under attempts/ and uploaded. Most failures leave no cause in the job log (the notes live only in the artifact), so a failing case now also prints each failed check's recorded detail. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014tWUsNuPcS5ByUDhzx9xh8
Mikola Lysenko (mikolalysenko)
added
the
ci-janitor
|
bugbot run Generated by Claude Code |
Mikola Lysenko (mikolalysenko)
added
the
Ready for review
|
[burn-down agent] Labeled Ready for review at
Generated by Claude Code |
|
Review updated for The original retry classifier let unrelated informational errors, expected negative probes, recovered warnings, or a later exception erase an earlier required functional failure. The fix associates transport evidence with the failing operation and preserves partial check results. Every failed required check must have causal transport evidence; an independent functional invariant vetoes retry. Structured CLI transport errors still qualify, and required setup failures stop dependent probes after preserving file invariants. Validation: the Python suite ran 148 tests: 147 passed and one platform-specific skip. Six false-green flow scenarios fail on the original PR head and pass with this fix. A separate exact-commit review passed nine flow scenarios and 14 classifier assertions. Fresh directory recreation, three-attempt bound, backoff, and archived attempt evidence remain covered. The commit merges cleanly with current main. No remaining code finding from this review. The Ready label has been restored after all checks completed on the fixed commit. |
Mikola Lysenko (mikolalysenko)
removed
the
Ready for review
|
Agreed, this is a real gap. The classifier is case-wide: it scans every log, including steps that succeed after a retried request (pip prints The fix you describe is the right one: tie the transport evidence to the failing required check's own operation (its log, or the structured CLI output), and never retry when an independent functional failure is present. I won't push to this branch while you prepare the correction, so we don't collide. I'll pick up CI and Bugbot on your push. Generated by Claude Code |
|
bugbot run |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit ef3beaf. Configure here.
Mikola Lysenko (mikolalysenko)
added
the
Ready for review
This branch has not been deployed
2 participants
Final-head CI is complete: 241 successful checks, 4 skipped; no failures or pending checks. Bugbot passed, there are no unresolved review threads, and the PR is mergeable.
Problem
Transient PyPI and patch API failures can interrupt the native Poetry compatibility matrix. A retry must distinguish those failures from functional regressions: for example, an informational relock timeout must not cause a failed rescan byte invariant to be retried and replaced by a passing result.
Change
Retry a case from a fresh directory at most three times, with 10s/20s backoff, only when every failed required check has transport evidence from its own operation. Typed command failures and HTTP/transport exceptions retain partial check results. Any earlier independent functional failure vetoes a retry. Successful recovered warnings, informational checks, and expected negative probes cannot trigger one.
CLI transport errors returned in structured output are recognized even when the process exits successfully. Lock and pyproject byte invariants remain independent. A required setup transport failure stops dependent probes after recording those invariants. Persistent transport failures remain red after the attempt limit.
Failed attempts retain their logs under
attempts/<case>/<n>/, which the workflow uploads, and the final row recordstransportRetries. Job output includes failed check details, including checks retained when a later operation raises. Production CLI behavior and other ecosystem harnesses are unchanged.Validation
ef3beaf0. Local flow regressions additionally verify retry causality with external commands stubbed.The original flake sample established a transport cause for only one of 12 inspected failures; other failures may be functional and must remain red.
Original implementation: Claude Code session. Review correction:
ef3beaf0.Note
Low Risk
Changes are limited to the Poetry backtest harness, CI artifacts, and tests; production CLI behavior is unchanged.
Overview
Adds transport-aware retries to the native Poetry compatibility backtest so transient PyPI/patch API blips do not flake the matrix, without masking real regressions.
backtest-poetry.pynow classifies terminal transport failures (Poetry/pip connection errors, patch API 5xx/429, CLI JSON errors on exit 0), records per-checktransportFailuresfrom the failing operation, and wraps each case inretry_transport(up to 3 attempts from a clean case dir, 10s/20s backoff). Retries run only when every failed required check has transport evidence; informational checks, recovered retry warnings, independent byte invariants, and earlier functional failures block retry. Failed attempts are archived underattempts/<case>/<n>/, surfaced intransportRetries, and the job log printsfailure_detailsfor failing checks.The Poetry CI workflow uploads attempt logs; docs describe the policy. Unit tests cover the classifier, harness retry behavior, and full case-flow scenarios (including false-green guards).
Reviewed by Cursor Bugbot for commit ef3beaf. Configure here.
Generated by Claude Code