Skip to content

Add unreviewed-merge metric to the reviewer table - #2

Merged
jonahgeorge merged 1 commit into
masterfrom
claude/pr-unreviewed-merge-metric-a28133
Aug 26, 2026
Merged

jonahgeorge merged 1 commit into
masterfrom
claude/pr-unreviewed-merge-metric-a28133

Conversation

@jonahgeorge

Copy link
Copy Markdown
Owner

What

Adds an Unreviewed column (with a Requested denominator) to the Reviewers table: how often a PR was merged while that reviewer's requested review was still outstanding.

│ Name          ┆        p50 ┆ ... ┆ Count ┆ Requested ┆ Unreviewed │
│ babakks       ┆  69h10m12s ┆ ... ┆   101 ┆        80 ┆   16 (20%) │
│ BagToad       ┆  23h55m41s ┆ ... ┆    61 ┆        73 ┆   23 (32%) │
│ williammartin ┆   1h18m47s ┆ ... ┆    22 ┆        37 ┆   16 (43%) │

Why

The existing turnaround percentiles only measure reviews that were actually submitted, so a review that was asked for and then overtaken by the merge is invisible — it silently drops out of the stats rather than showing up as a slow one. This makes that case countable.

How

The GraphQL query now pulls each PR's REVIEW_REQUESTED_EVENT and REVIEW_REQUEST_REMOVED_EVENT timeline items. review_requests replays those events chronologically per reviewer to determine whether the request was still open at merge time, and reviewer_stats counts it as unreviewed when the PR merged with the request outstanding and no review from them submitted by mergedAt.

Three judgment calls worth a reviewer's attention:

  • Withdrawn requests don't count — a request removed before the merge isn't a missed review.
  • Requests added after the merge don't count — GitHub permits this, and it isn't a case of merging out from under someone.
  • Team and mannequin requests are skipped — they name no individual to attribute the miss to. Bot requests follow the existing --include-bots flag.

Side effect: reviewers who were requested but never reviewed now appear in the table, which previously was keyed only on submitted reviews. Their percentile columns render as -.

Verification

cargo build and cargo clippy are clean.

Ran against cli/cli over a two-month window and cross-checked every reviewer against an independent Python implementation over the same raw API data — all six matched exactly (babakks 80/16, BagToad 73/23, williammartin 37/16, tidy-dev 21/7, sergiou87 2/0, niik 1/1). That data exercised bot requests (76, correctly excluded by default and included under --include-bots), team requests (231, skipped), and removal events (4).

Hand-verified cli/cli#14019: niik was requested at 11:23:58 and it merged at 11:31:05 with no review — correctly flagged.

No live PR exercised the "request added after merge" branch, so that path is reasoned-through rather than observed.

Notes

  • cargo fmt --check still reports one diff at src/main.rs:594. That is pre-existing on master (confirmed via git show master:src/main.rs) and was left alone rather than adding unrelated churn.
  • Unrelated and not fixed here: the README's Authors example table is stale from an earlier change — it is missing the Size (p75)/Size (p99) columns the code emits today.

🤖 Generated with Claude Code

Surfaces how often a PR was merged while a reviewer's requested review
was still outstanding — the reviews that were asked for but overtaken by
the merge, which the existing turnaround percentiles cannot show because
they only measure reviews that were actually submitted.

The query now pulls each PR's REVIEW_REQUESTED_EVENT and
REVIEW_REQUEST_REMOVED_EVENT timeline items. Those events are replayed
chronologically per reviewer to decide whether the request was still open
at merge time, so:

  - a request withdrawn before the merge is not a missed review, and
  - a request added after the merge is not counted either.

Team and mannequin requests are skipped since they name no individual to
attribute the miss to; bot requests follow the existing --include-bots
flag.

Reviewers who were requested but never reviewed now appear in the table,
which previously was keyed only on submitted reviews. Their percentile
columns render as "-".

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@jonahgeorge
jonahgeorge merged commit 8625ad0 into master Aug 26, 2026
@jonahgeorge
jonahgeorge deleted the claude/pr-unreviewed-merge-metric-a28133 branch August 26, 2026 23:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant