All notable changes to RunBoth. Dates are the day the work was measured, not the day it was released.
- Progress output during adjudication, on stderr. A run on a real repository takes
minutes and used to print nothing at all, which is indistinguishable from a hang. stdout
is untouched, so
--jsonconsumers see exactly what they saw before. - Control suite for the generated-
_version.pystub ([6/6]inrunboth selftest), including the cases that must NOT fire. RUNBOTH_NO_VERSION_STUB=1to disable that stub.
-
Async functions were silently dropped.
async deffunctions and methods never reached the report: not changed, not no_change, not abstained. And had they been listed, calling one runs none of its body, so both sides would have returned a coroutine object and compared equal. RunBoth now finds them, runs each coroutine (and drains each async generator) to completion, and compares what it returned or raised. On Windows the sandbox's network block also stopped the event loop from starting; it is lifted only while the loop is built, so the code under test still cannot open a connection. Found 2026-10-18 on a NeMo Gym environment pull request whosereset,stepand HTTP helpers were all missing. Tests:tests/test_async.py(a changed async function ischangedwith the boundary input, an async method is reported, an equivalent async rewrite isno_change). -
A missing
_version.pymade whole packages unadjudicable. setuptools_scm and similar tools generatepkg/_version.pyat build time and gitignore it, sogit archivenever carries it, the package import raisesModuleNotFoundError, and every function in it abstains for a reason that has nothing to do with its behaviour.materialise()now writes the file the build would have written.Measured 2026-09-14 on humanize: 13.0% of functions adjudicated before, 82.6% after, and the run went from zero findings to correctly catching
fractional(0)changing from'0/1'to'0'. urllib3 could not be imported at all before, and imports now.Deliberately narrow: only a directory that is already a package, only when
_version.pyis absent, and only when a sibling imports it outside atry/except. A package that guards the import and falls back is already working, and stubbing it would change an answer that was never broken. dateutil does exactly that, and is left alone. -
scripts/hunt_regressions.pycould not see past 60 commits.previous_changewalked the last 60 commits touching a path and compared unparsed sources. On a file that changes constantly that window never reaches the commit that moved the function, so it returned "nothing found" rather than "I ran out of room". It now asks git directly with-L <lines>:<path>, taking line ranges fromast, so it never depends on git's funcname regex and needs no.gitattributeschange in someone else's repository.Measured on more-itertools (more.py, 5,633 lines): the walk found nothing for five functions;
-Lfound the real commit for all five and was 25x faster (12.18s to 0.49s). One of them,only, was changed 211 commits back along that path. -
A timed-out suspect was reported as a clean one. The hunt counted attempts, not verdicts, and
if not recs: continueswallowed timeouts and empty results identically. A run could print "adjudicated 10 suspects, 0 leads" when nine of them never produced a verdict. It now names each one and separates attempted from adjudicated, because a count that hides its own failures is the thing this project exists to refuse.
funcs_atunparsed every function in a file to answer a question about one, and cached nothing. It now takes anonlyargument and memoises. Isolated benchmark on more-itertools: 66.01s to 28.92s for three lookups.
First public release.