docs: accuracy sweep — misattributed DOIs, a guard that could not fail, and count drift - #323

Reopened against main after #321 merged (the original #322 auto-closed when its base branch was deleted). Same commit, rebased.

Asked to make the repo accurate and current. Three classes of defect turned up, all live in a public repository.

1. Two DOIs belonged to other researchers

Verified twice — Zenodo record API and doi.org content negotiation.

They appeared in four files each, including docs/proposals/attestation-schema-proposal.md, which is written for submission to a standards venue. Citing another researcher's mass-spectrometry dataset as your own normalization-of-deviance paper, in a document arguing for a shared evidence schema, is the kind of thing that ends a conversation with a standards body.

All nine DOIs in the repo were re-verified. Seven resolve to Saleme records and are retained. The two above were replaced with verified records rather than re-pointed at a guessed identifier — no Zenodo record under either title by this author was located, and inventing one would repeat the original error. The README carries a standing correction naming the real authors.

2. A guard that could not fail

cli_counts = re.findall(r'(\d+)\s+security tests', f.read())   # cli.py has no such string
if counts_in_crosswalk and cli_counts:                          # -> always False
    ...                                                         # -> never ran

test_test_count_consistent_in_crosswalk passed while the crosswalk said 595 and the canonical count was 603. Two faults worth naming: it compared two documents to each other instead of to the source of truth, and it guarded the comparison behind a truthiness check, so the guard's own failure looked like a pass.

Same defect class as the HITL bug in #321 — a check that reports safety it never performed. Different subsystem, same root.

3. 595 tests / 43 modules in twelve live files

Including both AIUC-1 submission documents, the docs index, QUICKSTART, STRATEGY, the launch posts, CLAUDE.md and free_scan.py. README, SKILL.md and TEST-INVENTORY were guarded and correct. Guarding three files did not guard the repository — and the ones that drifted are what outside readers see.

Also found: EVALUATION_PROTOCOL.md claimed a 130-test suite and carried a Framework version: 3.1 (189 tests) footer.

The new guards, and why they are trustworthy

test_no_stale_test_count_anywhere and test_no_stale_module_count_anywhere sweep every live document. Dated snapshots — CHANGELOG, evaluation reports, archived roadmaps, blog posts — are excluded so history is not rewritten to today's number.

The first version of the matcher was itself broken. It missed the parenthetical (603 tests) form used in QUICKSTART, so a planted stale count sailed through. Caught by planting one rather than assuming. test_the_count_guard_can_actually_fail now pins that the matcher fires on six real phrasings and stays silent on x402 tests, L402 tests and Ed25519.

Verified by planting stale counts in three real files:

planted '595 tests'       in docs/QUICKSTART.md  -> 1 failed
planted 'tests-595-'      in README.md           -> 1 failed
planted '43 test-bearing' in docs/README.md      -> 1 failed
restored                                         -> 2 passed

Also corrected

• The comparison table listed the same repo twice. invariantlabs-ai/mcp-scan now redirects to snyk/agent-scan; Invariant and Snyk were counted as two separate competitors. Merged, star counts re-verified (Snyk 2.9K, Cisco 1.0K, Garak 8.7K).
• MCP coverage restated as 46 (protocol 32 + supply-chain 4 + tool-poisoning repro 10); it said 31. Enterprise platforms corrected from "20" to 58 (core 31 + extended 27).
• Citation honesty. README and ROADMAP now state that a 2026-08-02 OpenAlex audit found 30 citation edges and 0 qualifying independent citations — every edge a self-citation. ROADMAP had called the foundation "peer-cross-citing DOIs", asserting exactly the external citation the audit disproves, and claimed "independent, reproducible adversarial evidence" where adjudication is author-performed.
• ROADMAP release history ran v3.9 → v4.4 with "Next" in progress, seven minor versions behind. Now v3.9 → v4.13.1 with dates reconciled against CHANGELOG, plus a stated known-gaps section (T16-S1, T10-S1/S3, one guidance-only control, author-performed adjudication).
• The research-frontier list named intent-contract, multi-agent and memory work that shipped as modules. Replaced with the open research question for each, and the honest frontier: human-oversight measurement needs a study design, not more tests.
• README: added a Human Oversight layer, an OWASP v1.1 row, and an Independent Reproduction entry crediting @VrtxOmega — scoped as one external party reproducing one pinned artifact, explicitly not endorsement or adoption.
• Sample CLI output said v4.2; now v4.13.1, with a note that unreachable and unserviced targets report INCONCLUSIVE.

Verification

• 361 tests pass (was 358)
• count_tests.py → 603 across 44 modules, unchanged
• validate_owasp_agentic_mapping.py → PASS
• Every README link checked to resolve
• No product behaviour changes; documentation, one dead test repaired, three added

🤖 Generated with Claude Code