Differential execution for Python
RunBoth runs both versions of your code on generated inputs and tells you what actually behaves differently, including the functions nobody touched. No test suite required.
The gap
A refactor is supposed to look different. That is exactly why nobody catches the one that behaves different.
Reviewers read intent, not behaviour. A clean rename and a changed comparison operator look the same at a glance.
Your suite covers what you thought to cover. Prototype code written fast usually has no suite at all.
One line moved in a helper. A function three files away now returns something else, and its own source never changed.
How it works
Both commits are checked out at once. For every changed function, inputs are
generated from its signature and from the constants mined out of its own bytecode, so a
function containing units >= 100 gets tested at 99, 100 and 101. Both versions
then run in separate sandboxed subprocesses on the same inputs, in the same order.
The before version runs a second time, in a third fresh process, against itself. If it
disagrees with itself the comparison is meaningless, so RunBoth abstains with that reason
rather than reporting a difference. That is what catches uuid4,
os.getpid and unseeded randomness.
From what changed, it walks outward through the call graph and executes the callers too, carrying the values already proven to differ. This is the part with no equivalent in a diff-based reviewer: the caller's source did not change, so there is nothing to review, and it still broke.
The clearest one it found
__dir__ in packaging/tags.py.
Its source never changed a character. The commit added a new public name
to the module, so the function returns a different list now. There was no diff to read and
nothing for a reviewer to look at. It was found by running it.
Out of 2,548 functions across eight libraries, the false-positive oracle flagged exactly one thing, and on inspection the tool was right and the oracle was wrong.
The exact arguments, what it used to do, what it does now. A fact you can re-run.
N generated inputs found no difference across seven channels. Evidence, never proof.
It could not be checked, and here is why. Never counted as passing.
Three verdicts instead of two is the deliberate part. "Cannot tell" and "no difference" are different claims, and collapsing them into a green check is how tools end up lying. RunBoth never says safe.
What it caught
toolz, and said it hadn't.toolz is a real library you have probably installed.
tail(n, seq) returns the last n elements. A frontier model was
asked to refactor it and to preserve behaviour. It added a guard that reads like exactly
the kind of defensive check a careful engineer writes.
+ if n <= 0:
+ return type(seq)() if hasattr(seq, '__getitem__') else tuple()
It looks obviously correct. It is not. In the
original, seq[-n:] with n = 0 is seq[0:], the whole
sequence.
| call | toolz | after the refactor |
|---|---|---|
| tail(0, [1, 2, 3]) | [1, 2, 3] | [] |
| tail(-1, [1, 2, 3]) | [2, 3] | [] |
| tail(2, [10, 20, 30, 40, 50]) | [40, 50] | [40, 50] |
A correct answer became a silently empty one. No exception, no failing test: the docstring example still passes, which is why the doctest stays green and a reviewer waves it through. Then RunBoth ran both versions.
$ git commit -m "refactor(itertoolz): simplify tail, behaviour unchanged" BLOCKED: your commit message says the behaviour did not change. It did. tail(-2, [1, 2, 3]) used to: return [3] now: return [] That call is the proof. Anything relying on the old result behaves differently now, and your test suite did not stop this commit.
No test was written and no annotation was added. The inputs came from the function's own signature and the constants in its own bytecode.
Across 36 functions from toolz and
markupsafe. One model and one prompt style, so it is a measurement rather than a
law, and the before and after source of every failure is published with it.
Independent work puts that in range. Dristi and Dwyer used differential fuzzing to check functional equivalence in LLM-generated refactorings across six models, three datasets and two refactoring types, and found them functionally non-equivalent 19 to 35% of the time (arXiv:2602.15761, February 2026). The 44% above is higher because it is narrower: one model, one prompt style, 36 functions. Their range is the published figure; mine is a replication that landed above it.
Found in the wild
Everything above is a break we went looking for. This one we did not. It surfaced while running the regression hunt across six public repositories, and the project confirmed it themselves.
On 7 January 2025 a commit landed in
humanize titled
“Update pre-commit (#226)”. It was a tooling configuration
change. It also changed what naturalsize() returns.
$ git checkout fb61d5a~1 # before naturalsize(1.5) '1 Bytes' $ git checkout fb61d5a # "Update pre-commit (#226)" naturalsize(1.5) '1.5 Bytes' $ git checkout 11e62ee # six weeks later naturalsize(1.5) '1 Bytes'
That last commit is titled “Fix regression in naturalsize for float”. The project found it six weeks after it shipped. RunBoth reported the function as changed at the commit that introduced it, from the code alone, with no test and no issue report.
No reviewer reads a pre-commit config change looking for altered output. No diff shows it, because the diff is in a YAML file. The test suite passed. This is the exact shape the tool exists for, and we did not have to construct it.
The control is what makes either number mean
anything. The same instrument, run over 29 real human commits across
toolz and markupsafe, found zero witnessed
behaviour changes. Without that, a number like 44% invites “maybe it just flags
everything”. With it, the number is about the refactors and not about the tool.
Proof
Real historical commits from public projects, with an automated oracle built to catch the tool lying: for every function reported as changed, check whether its source actually changed, and whether a changed callee or constructor explains it. Anything else is a probable false positive.
| Repository | Layout | Functions | Abstained | Median | False pos. |
|---|---|---|---|---|---|
| boltons | flat | 268 | 0.0% | 7.9s | 0 |
| sqlparse | flat | 230 | 0.9% | 14.3s | 0 |
| arrow | flat | 283 | 1.4% | 118s | 0 |
| cachetools | src/ | 325 | 1.8% | 60.9s | 0 |
| more-itertools | flat | 843 | 4.4% | 102.5s | 0 |
| packaging | src/ | 78 | 5.1% | 0.2s | 0 |
| pluggy | src/ | 157 | 6.4% | 33.5s | 0 |
| tenacity | async | 364 | 14.3% | 192.8s | 0 |
arrow is the one that matters most: a datetime library is the worst case for a differential tester, and it produced no false positives across 283 functions. tenacity is the honest other end, a retry library whose functions genuinely sleep, so checking it is slow and it abstains more.
What it is not
Translate code into logic, let inputs be unconstrained symbols, ask a solver whether a bad state is reachable. You get a proof, and you pay in modelling effort and a limited language subset.
Damage your code on purpose to see whether your tests notice, and score the suite. Needs a test suite to be worth anything.
Executes both real versions on concrete inputs and compares seven channels. Mutates nothing, needs no suite, and hands you a witness you can paste into a REPL.
Start
Installs a commit-msg hook. It stays silent unless
behaviour moved, and it never blocks on an abstention.
pip install runboth runboth install-hook
Runs on your own runners, so there is no service to trust and nothing leaves your CI. Comments once, edits itself after.
- uses: runboth/runboth@v0.1.0
with:
budget: 60No repository and no checkout. It fetches both released versions, builds each in its own environment and runs them against each other. For the minor bump that broke something in production last year.
runboth versions requests 2.31.0 2.32.0
An MCP server ships in the package, so a coding agent can check its own edit before it tells you it is done. Stdlib only, no extra install.
python -m runboth.mcp_server
The CLI, the commit hook and the Action are free, and they stay free. Source available under FSL-1.1: free for every use except building a competing product, converting to plain Apache 2.0 two years after each release.
FAQ
The mechanism is differential testing and it is decades old. Nothing here is claimed as invented, and anyone telling you this idea is novel is selling something. What is different is the shape it is packaged in: no oracle to write, no properties to state, no approvals to capture, and it runs the callers you did not touch. The work was not thinking of it, it was making it quiet enough to leave switched on.
If you already have tests covering the behaviour an agent is about to change, use them, and this buys you less. It exists for the enormous amount of code that has no such tests, and for the case tests structurally cannot reach: a test asserts what somebody thought of in advance, and nobody writes a test for the function they did not edit.
They are not random. Inputs come from the signature and from the literal constants mined
out of that function's own bytecode, so a function comparing against 0 and
-1 gets tried on 0 and -1. That is not a
refinement, it is the whole reason the toolz case above fell out on the
first run.
No. It generates its own inputs from each function's signature and from the literal constants already present in that function. That is the point for prototype code written fast, which usually has no suite at all.
No. The shipped package makes no network calls, declares no dependencies and contains no AI model. Every verdict is backed by an execution you can re-run yourself.
It blocks only on a measured behaviour change, never on an abstention, and it reconciles against your commit message: if you said what you were changing, it lets it through. A gate that argues with work you did on purpose gets uninstalled the same afternoon.
No, and it will never say so. Sampling finds differences; it cannot prove their absence.
The verdict is worded no_change at budget 60 for exactly that reason.
The before version is run twice in separate processes and compared against itself. If it disagrees with itself, RunBoth abstains rather than reporting a difference. An adversarial corpus of 22 functions built to trigger false positives produced none.
Function-level checking is Python. Changed files in other languages are named explicitly rather than passed over quietly, because silence is how this tool reports no difference found, and silence must never cover something it did not look at.
Have it run on your repository
The tool checks the commit in front of you. A drift audit is the other direction: RunBoth run backwards across your history to find the behaviour changes that shipped while a commit message said nothing moved, written up with the exact call that proves each one.
This one is paid work and I do it by hand. Leave an address and I will tell you what it would cost for your repository. No newsletter, and the address is never passed on.