Repository navigation
CI red on all PRs since 2025-12: three independent causes (tracking) #1135
Description
Summary
Every PR Build since 2025-12-19 has failed, on all five matrix jobs, regardless of what the pull request changes. A one-line Dependabot version bump (#1132) fails identically to a large feature branch. Nobody can currently get a green check, and a genuine regression would be indistinguishable from the existing noise.
There are three independent causes, now tracked separately as sub-issues. Two of them are bugs in our own code that affect anyone using CathInstallation or EcodInstallation at runtime, not only CI.
| Cause | Sub-issue | Fixed by | |
|---|---|---|---|
| 1 | CathInstallation downloads over http, which now 301s to https, and the redirect body is cached as classification data |
#1138 | #1133 (ed36d3a) |
| 2 | ECOD changed its distribution format; neither the version header nor the column layout is recognised | #1139 | #1141 |
| 3 | Master Build fails on an expired or absent SONAR_TOKEN |
#1140 | needs credentials, no PR possible |
Correction to the original diagnosis
This issue originally recorded cause 2 as "EcodInstallationTest.testVersion downloads 657 MB from a slow host", and described it as a judgement call rather than a bug. That was wrong.
The download completes. The file is read. The assertion then fails because ECOD changed its version comment from #ECOD version develop291 to # Version: v295, so getVersion() falls back to the literal string "latest" — exactly what the test asserts against. The column layout changed at the same time, from 15 columns to 23 and then 25, which means getAllDomains() on the current release returns an empty list, silently, for every user of the library.
The 657 MB is real and is still worth addressing — it is pulled once per matrix job, five times per pull request — but it is a cost, not the failure, and making the download faster would not have fixed anything. Full detail in #1139.
Evidence
Last green PR Build |
2025-12-14, run 20214548025 |
Last green Master Build |
2025-12-10, run 20115022310 |
PR Build runs since |
every one failed |
| Same failures on a one-line dependency bump | #1132 |
| After the CATH fix, ECOD only | run 31916362199 |
| With integration tests excluded, all five green | run 31916395882 |
Logs older than GitHub's 90-day retention have expired, so the cause is only confirmable for recent runs. The failure streak itself is visible in run metadata back to 2025-12-19; whether the earliest ones share this cause cannot now be checked.
Reproducing
mvn verify -pl biojava-integrationtest -Dtest='CathDomainTest+EcodInstallationTest#testVersion'Keeping this from recurring
Both code causes are the same shape: an upstream service changed something, and we found out months later through a failure that named neither the service nor the change.
pull_request.ymlrunsmvn verifywith no module exclusion, so every pull request executesbiojava-integrationtest, which by design depends on CATH, ECOD, RCSB and EBI being up and fast.master.ymlalready excludes it. Run integration tests nightly rather than on every pull request #1137 proposes applying the same exclusion to pull requests and running the full suite nightly — an upstream break then reaches us within a day, as a failure whose meaning is unambiguous, instead of as a red check on somebody else's pull request.- Support the new ECOD distribution format #1141 adds offline unit tests pinning every ECOD file layout the project has ever read. Those run in under a second and would have caught the format change without downloading anything, which is the coverage that was actually missing.
Activity
aalhossary commentedon Aug 16, 2026
Pull requests are now open for the two causes that are ours to fix:
http://redirect) — fixed by commited36d3ain Fix FileDownloadUtils validation and implement checksum verification #1133, alongside the download hardening it depends on. Fix FileDownloadUtils validation and implement checksum verification #1133 also pre-emptively closes the same hole inDownloadChemCompProvider, which is not broken today only because the RCSB ligand endpoint still serves plain http.SONAR_TOKEN) — not something I can fix; it needs whoever holds the SonarCloud organisation credentials.Edited. This comment originally said that cause 2 was "deliberately untouched … a judgement call rather than a bug: the download works, it is just very large and served slowly". That was wrong, and I would rather correct it than leave it standing. The download does work — the test fails because ECOD changed its distribution format, which also means
getAllDomains()has been returning an empty list to every user of the library. Detail in #1139, fix in #1141.aalhossary commentedon Aug 16, 2026
Update: CATH is confirmed fixed in CI, and ECOD was the only remaining blocker
The matrix now says what previously only my local run did.
Before
ed36d3a, every job on every pull request failed on two tests. With the CATH fix in #1133, all five jobs fail on one:CathDomainTestno longer appears, across JDK 11/17/21 on both Linux and Windows (run31916362199).Meanwhile #1137, which takes integration tests off the pull request path, passed all five jobs (run
31916395882) — the first greenPR Buildsince 2025-12-14. That is also a useful control: the same tree passes with the integration module excluded and fails with it included, so nothing in the main modules is broken.That ECOD failure is now diagnosed and fixed in #1141, so with #1133 and #1141 merged the build should be green with the integration tests still running.
Edited. This comment originally read "the download is not broken: it works, it is simply 657 MB served slowly and pulled once per matrix job", and left the choice of fix open. That framing was wrong. The failure is a bug — ECOD changed its version comment and its column layout — and the size, while a genuine CI cost, had nothing to do with it. See #1139.
aalhossary commentedon Aug 16, 2026
The three causes are now split into sub-issues and linked under this one, since they have different owners and different lifetimes — one is fixed and awaiting review, one turned out to be a bug and has a fix, one needs credentials nobody else can supply.
http://URL. Fixed byed36d3ain Fix FileDownloadUtils validation and implement checksum verification #1133, closable as soon as that merges.Master Buildfails on an expired or absentSONAR_TOKEN. Open; needs whoever holds the SonarCloud credentials.If you would rather not have them fragmented, say so and I will close the three and keep everything here.
Edited. This comment originally offered to create the sub-issues and described item 2 as "downloads 657 MB from a slow host — open, needs a decision". It needed neither a decision nor a policy change: it was a bug, and it is fixed.
[-]CI red on all PRs since 2025-12: our CATH downloader follows a stale http URL, and the ECOD test pulls 657 MB[/-][+]CI red on all PRs since 2025-12: three independent causes (tracking)[/+]aalhossary commentedon Aug 27, 2026
With all 3 subissues merged, this issue can be safely closed now.