Repository navigation
feat(eval): AACR-Bench replay harness - #26
Merged
Merged
Conversation
…luation ocra-eval selects a seeded subset of AACR-Bench PRs, reviews each one by running the real ocra CLI on a blobless clone, and scores findings with a port of the benchmark's four-stage matching: path, side, line proximity and an LLM judge. Runs resume per PR and stop starting new PRs at a spend cap. Reports cover precision, recall, F1, line-level metrics, breakdowns by language, category and context, and cost and latency. The judge treats answers opening with no as no, fixing a false positive in the official parser.
jma49
added a commit
that referenced
this pull request
Sep 28, 2026
The site's design rules now live in its DESIGN.md (site #26), with a token migration plan and a decision on the terminal's warn colour. Two site traps that cost time are added to pitfalls: blank full-page screenshots caused by Reveal, and builds failing without access to Google Fonts.
jma49
added a commit
that referenced
this pull request
Sep 28, 2026
The site's design rules now live in its DESIGN.md (site #26), with a token migration plan and a decision on the terminal's warn colour. Two site traps that cost time are added to pitfalls: blank full-page screenshots caused by Reveal, and builds failing without access to Google Fonts.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #12 (the harness; the baseline follows once bundling and off-bundle filtering land).
What
ocra-eval(docs indocs/evaluation.md):list: free preview of a seeded selection (--limit,--seed,--languages,--max-change-lines,--ids).run: for each PR, blobless-clone the repo (cached), check out the head, run the realocra review --from <base> --to <head> --format json, and save the result. Runs resume per PR and stop starting new PRs at--max-cost-usd.score <run-dir>: re-score with cached judge answers (free).Scoring is a port of AACR-Bench's
evaluate_comments, so numbers are comparable:The report covers precision, recall, F1, line-level variants, breakdowns by language / category / context level, tokens, cost and latency.
Judge: the official prompt at temperature 0, via any OpenAI-compatible endpoint (
JUDGE_*variables, the official names) or a Gemini key. One deliberate fix: the official parser counts "No, they are not the same issue" as a match; an answer opening with "no" is a no here.Also: Biome skips git-ignored files (local eval output had broken the check).
Smoke run (3 PRs ≤150 lines, flash-lite for review and judge, $0.17)
The sample is too small to judge quality, but it exposed the next fixes:
breakbug was reported by 3 tasks reviewing other files, which cuts precision.module.builtinModulesnodejs/node#56185's head commit is no longer fetchable (force-pushed); such PRs should be excluded as unavailable rather than counted as failures.How verified
npm run verify(215 tests: dataset parsing and cache, seeded selection, matching stages including the one-line tolerance and one-to-one counting, metrics, judge request/parse/cache, runner resume and budget with a stand-in CLI, scoring, report)