refactoring-oracle

A small fixture of Python refactoring cases and a deterministic grader that says whether a candidate refactoring is correct, with no model and no human in the loop.

It exists because a thread claimed frontier coding agents do common refactorings "quite reliably", a reply said "quite reliably is not enough", and nobody had a number. Nobody had a number because nobody had a grader. This is the grader, plus five things it grades: Claude through the API, bare and with a refactoring skill; Claude Code run headless, bare and with the same skill; and rope, the deterministic refactoring library for Python.

grade(case, candidate_dir) runs four checks, in order, and names the first one that fails:

The verdict's failure_class splits the usual argument into measurable parts: changed behavior (the fear), did not refactor (the shortcut), touched things it should not have (the reason people review every line).

What it does not judge: style, naming beyond the named interface, or whether the refactoring was a good idea.

ro-selfcheck and tests/test_selfcheck.py enforce, for every case:

• the reference after/ passes all four checks;
• the untouched before/ fails the shape check (so the assertions bite);
• a hand-written mutants/behavior_* candidate, a subtly wrong version of the changed function, fails on behavior and nothing earlier;
• a synthetic collateral mutant (one byte appended to the never-allowed __init__.py) fails on collateral, and a synthetic compile mutant fails on compiles.

If any of those stops holding, the fixture is wrong, not the thing being graded.

Six refactorings, two fresh samples each, no examples from Fowler or from anyone's skill library. Each case's instruction names the target interface exactly, the way a person would ask, so grading needs no interpretation.

Case layout:

cases/<refactoring>/<sample>/
  case.json      id, instruction, allowed_files, protected, allowed_imports,
                 deterministic_tool, shape assertions
  before/        what an arm is given: the module, __init__.py (never allowed), tests/
  after/         the reference solution; never shown to an arm
  tests_after/   the held-out suite
  mutants/       behavior_<name>/ candidate trees that must fail on behavior

The whole cases/ tree is frozen by ro-freeze into cases.sha256, and ro-grade refuses to run if it has drifted.

An arm gets a fresh copy of a case's before/ tree and the instruction, produces a candidate tree, and the oracle grades it. The arm is the only variable.

Jason Gorman's format from the thread: a name, a one-line description, a one-paragraph summary, and one before/after example. One file per refactoring in skills/, with examples in domains that are not in the fixture. The rename one, complete:

---
name: rename
description: Rename refactoring for Python. Use when asked to rename a function, parameter, or variable everywhere it is used.
---
# Rename

Change a name at its definition and at every use, including call sites, keyword arguments, docstrings, and tests that reference it. Only the name changes.

Example:

def calc(amt, pct):
    """Apply pct to amt."""
    return amt * pct / 100

def tax(amt):
    return calc(amt, 8)

Rename `calc` to `apply_percent` and `amt` to `amount`.

def apply_percent(amount, pct):
    """Apply pct to amount."""
    return amount * pct / 100

def tax(amount):
    return apply_percent(amount, 8)

The prompt and the skills are hashed into every artifact's meta. ro-run runs each arm over each case N times, sequentially, copies before/ per trial, applies the returned files, grades, keeps the unified diff, and rewrites the artifact after every trial so a killed run resumes. --dry-run swaps in a reference arm that copies after/ and needs no key.

Artifacts in runs/: api-results.json, agent-results.json, rope-results-v2.json (and rope-results.json, the first rope run, see below), all from 2026-09-24. Model claude-sonnet-4-6 for the four model arms. Every trial starts from a fresh copy of before/.

Across 144 model trials: zero behavior changes, zero files or functions touched outside the instruction, zero compile errors. Every miss by a model is the same one case.

The one case. guard-clauses/discount-eligibility asks for guard clauses "so that no if statement is nested inside another if". Both API arms produced, on every run, a byte-identical version with one nested if under if order_total >= 100:. The tests pass and it is a reasonable guard-clause rewrite; it violates the letter of the instruction, and the reference met the constraint with a conditional expression. The bare agent passed all three runs by writing if order_total >= 100 and coupon == "SAVE20": return 0.2 followed by if order_total >= 100: return 0.1. The agent with the skill passed twice and produced the API arms' nested version once.

The skill. No effect on the API arm (same outputs, same miss, about 200 more input tokens) and one extra miss on the agent arm. On this fixture the model already knew all six refactorings; the before/after example added nothing it needed.

The harness. Same model, same instruction, different result: the one-shot rewrite missed the constraint every time and the agent met it every time. The agent can read the whole tree, run the before tests, and revise; it used about five turns per case and cost three times as much.

The deterministic tool. Within what it implements, rope is perfect: both renames, both inlines, both extracts. It is reported as two numbers on purpose, because pooling them into 6 of 12 would score rope on the catalog I chose rather than on its work. The other six cases (parameter object, magic literal, guard clauses) have no rope refactoring at all; that is the coverage number.

Two things about how rope was driven. It takes offsets and refactoring classes, not an instruction, so a person wrote a spec per case, and the comparison is "rope, driven by someone who read the instruction." And the first rope run (runs/rope-results.json) scored 4 of 6, because ExtractMethod chooses its own parameter order, evening_discount(entered_hour, base) and validate(zone, weight_kg), and offers no way to set it; the held-out tests call the new function in the instruction's order. Adding the step a person would take next, ChangeSignature with the named order, is what the v2 specs do. Both artifacts are kept.

What this is not: twelve cases, one model, small files, and instructions written to be gradable. It is a shape, not a benchmark.

uv sync --group dev
cp .env.example .env              # ANTHROPIC_API_KEY, only needed for live arms
uv run pytest -q                  # oracle unit tests plus the selfcheck over every case
uv run ro-selfcheck               # the same, as a table
uv run ro-grade extract-function/parking-fee cases/extract-function/parking-fee/after
uv run ro-grade extract-function/parking-fee cases/extract-function/parking-fee/before
uv run ro-grade <case-id> <candidate-dir> --json
uv run ro-freeze
uv run ro-run --dry-run                          # reference arm, no network
uv run ro-run --cases rename/csv-cleaner --runs 1 --out runs/smoke.json
uv run ro-run --runs 3 --out runs/api-results.json
uv run ro-run --arms agent-bare,agent-skill --runs 3 --out runs/agent-results.json   # needs the claude CLI
uv run ro-run --arms rope --runs 1 --out runs/rope-results.json                      # local, free
uv run ro-run --report-only --out runs/api-results.json

defines_function (name, optional exact params), not_defines, defines_class (optional dataclass flag and exact field order), calls (callee inside a function), no_call (file-wide or inside one function), defines_constant (module-level, exact value and type), no_literal (inside a function; 1 and True are different literals), starts_with_guard, max_if_depth, name_absent, name_present. Methods are addressed as Class.method.

src/refactoring_oracle/case.py         Case and ShapeAssertion models, loading
src/refactoring_oracle/shape.py        the assertion kinds
src/refactoring_oracle/collateral.py   byte, AST, and import comparisons
src/refactoring_oracle/behavior.py     compile check and the pytest subprocess
src/refactoring_oracle/oracle.py       grade() and the Verdict
src/refactoring_oracle/freeze.py       hash of the cases tree
src/refactoring_oracle/cli.py          ro-grade, ro-selfcheck, ro-freeze, ro-run
src/refactoring_oracle/prompts.py      the frozen system prompt, tool, and skill loading
src/refactoring_oracle/arms/api.py     the two API arms and the dry-run reference arm
src/refactoring_oracle/arms/agent.py   the two Claude Code headless arms
src/refactoring_oracle/arms/rope_arm.py  rope, with per-case offset specs
src/refactoring_oracle/runner.py       trials, checkpointed artifact, summary
src/refactoring_oracle/report.py       stdout tables
skills/                                one Gorman-style skill per refactoring
runs/                                  committed artifacts and captured output
tests/                                 unit tests, selfcheck, house rules