feat(loop-drill): fire drills that prove guardrails actually fire - #600
Conversation
docs/failure-modes.md names ten ways loops fail. loop-audit scores whether the
mechanical counterparts are *present*. Nothing checked whether they fire.
The sharpest case is the verifier. loop-audit awards its joint-largest signal
(14 points) for one, and gates L3 on it, via a filename check:
if (base.includes('verifier') || base === 'loop-verifier') // auditor.ts:181
An empty file passes. On a scratch repo, `touch .claude/agents/verifier.md`
moves the score 34 (L0) -> 55 (L1). docs/failure-modes.md calls the resulting
failure Verifier Theater and rates it S2; docs/primitives.md calls maker/checker
"the single most important structural pattern for reliable loops". The scorer
meant to certify that pattern was performing it.
loop-drill injects a known fault and asserts the guardrail responds. Each drill
is tagged with the failure mode it exercises, so the report reads as coverage of
docs/failure-modes.md rather than a list of anonymous assertions.
Every guardrail is drilled in both directions, because only one is easy:
sensitivity (the fault is caught) and specificity (benign input is not). A
denylist of ["**"] catches every seeded fault and still fails, because it blocks
an ordinary docs change too.
The verifier canary borrows mutation testing: apply a mechanical,
behaviour-changing edit to real source and see whether the verifier notices.
Deterministic and model-free, so it costs only what the verifier costs.
Three properties make the score trustworthy:
- each run is an ephemeral git worktree, removed even on throw
- a second control runs in a clean worktree; without it a missing
node_modules fails every mutant for the wrong reason and a broken setup
reports a perfect score, so mutants are skipped rather than scored
- a verifier that rejects a clean tree bails before mutants run, so
reject-everything cannot report 100%
Operators are limited to edits that change behaviour in any C-family language
and read as bugs. `<` -> `>` and `+` -> `-` are excluded: they break TS generics
and string concatenation, and would make a correct verifier look broken.
The gate and breaker drills are offline and deterministic -- no agent, no
tokens -- so the tool is useful before anyone wires up a verifier.
CI runs it against this repo, tolerating exit 1 (a skipped drill) and failing on
exit 2. The guard is an explicit if rather than `cmd || [ $? -eq 1 ]`, because
under set -e a failing command on the right of || does not exit the shell --
that shorter form silently passed an exit-2 drill when first written.
|
This PR changes paths that must run the real Fork PRs from first-time contributors start with those workflows waiting for approval. A maintainer needs to open the Checks tab and click Approve and run workflows. Until that happens, branch protection will show the PR as blocked even after a review. Content-only PRs ( — loop-engineering fork-pr-gate |
cobusgreyling
left a comment
There was a problem hiding this comment.
Thanks for this — the gate/breaker drill design (sensitivity + specificity, exit 0/1/2 matching loop-gate/loop-context, worktree-isolated canary) is the right shape for proving guardrails actually fire.
Requesting changes on one test that currently cannot fail, plus a smaller typing issue in the canary runner.
…ary spawn Review feedback from @cobusgreyling on cobusgreyling#600. 1. test/drill.test.mjs: the unreachable-threshold test was a passing no-op. Its name, comment, values and assertion disagreed, and no config could make it fail. Chasing that down surfaced a real bug behind it: the breaker drills asserted only `decision.escalate`, not *which* rule fired. Escalating for another reason leaves the rule under test unproven -- the same standard the gate drills already applied via `trigger !== 'denylist'`. breaker.stagnation, breaker.no-progress and breaker.token-budget now assert their own trigger. That exposed breaker.no-progress passing for the wrong reason. It built its ledger from errors differing only by a number ("module 1 not found", "module 2 not found"), and errorSignature() collapses every number to '#', so all five normalized to one signature and tripped *stagnation*. The no-progress rule was never exercised. It now uses errors that stay distinct after normalization. The no-op is replaced by tests that fail when the assertion is weakened, verified by reverting each trigger check in turn: - similarityThreshold as a percentage (95 instead of 0.95) -> identical errors never match, the breaker never fires, stagnation unproven - noProgressThreshold 1 -> the breaker escalates via no-progress, so stagnation is unproven even though the loop did stop - maxIterations 2 -> escalates via max-iterations, same conclusion - similarityThreshold 0 -> every error counts as the same error, so stagnation swallows the no-progress drill breaker.token-budget has no wrong-trigger test: its ledger is one successful attempt, so no other trigger is reachable. Noted in the test file rather than faked. 2. src/canary.ts: promisify(execFile) was called with an options object in the `args` position behind an `as never` cast. It worked only because Node detects a non-array second argument, and the cast hid the real signature. Caller-supplied strings (--verifier-cmd, --setup) now go through promisify(exec), the shell form, which is what those flags mean. git keeps execFile with an argv array and no shell, so repo paths are never re-parsed by a shell. No casts remain. Canary re-verified end to end after the spawn change: 100% mutation score against loop-gate's real suite, exit 2 for a rubber-stamp verifier, and timeout handling still reports "timed out" rather than a spawn error. 57 tests pass.
|
@cobusgreyling I have made the changes, please verify it now |
cobusgreyling
left a comment
There was a problem hiding this comment.
Re-reviewed 09a50bd. Both requested changes are addressed:
- canary spawn:
promisify(exec)for--verifier-cmd/--setupandpromisify(execFile)+ argv for git. Noas neverleft. - breaker tests: drills now assert the specific trigger. The old no-op is gone; wrong-trigger cases (
similarityThreshold: 95+noProgressThreshold: 1,maxIterations: 2,similarityThreshold: 0) fail when the wrong rule fires, and no-progress uses signatures that stay distinct after#normalization.
CI is green (audit/gate/validate). Approving.
The problem
docs/failure-modes.mdnames ten ways loops fail. Several have mechanical counterparts —loop-gatefor path scope,loop-context's circuit breaker for runaway retries — andloop-auditawards points for having them.Nothing checked whether they fire.
The sharpest case is the verifier.
loop-auditawards its joint-largest signal (14 points) for a verifier, and gates L3 on it, via a filename check:An empty file passes. On a scratch repo:
.claude/agents/verifier.mdtouch verifier.mdis worth 21 points and a readiness level.docs/failure-modes.mdcalls the resulting failure Verifier Theater (S2).docs/primitives.mdcalls maker/checker "the single most important structural pattern for reliable loops." The scorer meant to certify that pattern was performing it.What this adds
loop-drillinjects a known fault and asserts the guardrail responds. 19 drills across three guardrails, each tagged with thedocs/failure-modes.mdentry it exercises, so the report reads as coverage of that document rather than a list of anonymous assertions.Both directions, always
Every guardrail is drilled twice, because only one direction is easy:
A
denylist: ["**"]catches every seeded fault and still fails the suite, because it blocks an ordinary docs change too.The verifier canary
Borrows mutation testing: apply a small mechanical, behaviour-changing edit to real source and check whether the verifier notices. Deterministic and model-free — no agent invented the defect, so it costs only what your verifier costs.
Three properties make the score trustworthy:
git worktree, removed even on throw, so a verifier that writes or builds cannot touch the checkout.node_modules, sonpm testfails there for reasons unrelated to code quality. Without this second control every mutant scores as "caught" and a broken setup reports a perfect score. When it fails, mutants are skipped, not scored, and you are pointed at--setup.Operators are limited to edits that change behaviour in any C-family language and read as bugs:
===↔!==,<=→<,>=→>,&&→||,return true↔return false.<→>and+→-are deliberately excluded — they break TS generics and string concatenation, and would make a correct verifier look broken. Comments, tests,dist/andnode_modules/are never mutated.Verified end-to-end
gate.yaml+ breakerloop-gate's own test suite)echo LGTM)A rubber-stamp verifier, caught:
52/52 tests pass.
loop-gate(32) andloop-context(53) are unaffected. No worktrees leaked.Design notes for review
checkGateandcheckCircuitBreakerare already pure functions, so the gate and breaker drills are offline and deterministic — no agent, no tokens. The tool is useful before anyone wires up a verifier.loop-gate/loop-context— 0 proceed, 1 warnings, 2 escalate — so control scripts chain all three.file:links toloop-gateandloop-context, followingloop-swarm's existing dependency onloop-sandbox.Two bugs I hit building it — both this tool's own failure mode
Worth surfacing because they are the argument for the tool:
node cli.js . || [ $? -eq 1 ]. Underset -e, a failing command on the right of||does not exit the shell, so an exit-2 drill sailed through. Now an explicitif, verified against both exit 1 and exit 2.Limits (also in the README)
Escalation FailureandNotification Fatiguehave no drills yet — they need a notification sink to observe.--mutants 1and--scope.One call to flag
I wired the dogfood run into
scripts/ci-validate-gates.sh, so this repo's CI now fails if its own guardrails stop firing. That is the strongest form of the idea, but it is also the most intrusive part of this PR — happy to drop it to a non-blocking step, or remove it entirely, if you would rather land the package first and adopt it separately.