GitHub Built a Benchmark That Tells You When Your AI Code Reviewer Is Lying to You

Every AI code review tool on the market can show you a demo where it catches a bug a human missed. None of them can show you the pull requests where it missed something a junior reviewer would have caught, or flagged forty nitpicks nobody asked for. On October 5, GitHub released ReviewBench, an open benchmark built specifically to surface that second half of the story: 219 pull requests, pulled from 187 public repositories across 19 languages, with ground-truth findings good enough that GitHub trusts the numbers it produces about its own product.

The corpus itself isn’t the interesting part. Plenty of benchmarks have a pile of pull requests. What makes ReviewBench worth reading closely is the five-step process GitHub used to decide what counts as a correct finding — because that process is the actual template for building any benchmark you’d trust enough to act on.

Start from the real distribution, not a convenient sample

GitHub analyzed 103.9 million pull requests before picking the 219 that made the cut, and deliberately weighted the selection toward multi-file, genuinely reviewable changes rather than the trivial single-file diffs that dominate raw PR volume. That matters more than it sounds. A benchmark built from whatever PRs are easiest to scrape skews toward simple changes, which is exactly where every AI review tool already looks competent. The hard cases — the ones that actually separate a useful reviewer from a noisy one — live in the multi-file changes that touch more than one concern at once.

Don’t trust one source for ground truth

For each PR, GitHub gathered findings from four separate sources — human reviewers, frontier LLMs, static analysis tools, and the author’s own follow-up commits — then semantically deduplicated the results and validated them against a shared rubric, using Claude Sonnet 5 as the grading model. That last detail is worth sitting with: an AI model is doing part of the work of deciding what counts as a correct answer in a benchmark that will be used to grade other AI models. GitHub’s answer to the obvious objection is the next step.

Audit the auditor

Before release, senior engineers independently re-labeled the ground-truth findings by hand and compared their labels against ReviewBench’s — 96.6% agreement. That’s the number that makes the Claude Sonnet 5 grading step defensible instead of circular. It’s also the number most teams skip when they build an internal eval set: they pick a grading model, trust its output, and never check it against a human doing the same task cold. A 96.6% agreement rate earned that way is a far stronger claim than a benchmark that simply asserts its labels are correct.

ReviewBench also reports six separate metrics rather than one aggregate score: grounded precision, recall, and F1 measured against the fixed golden set, plus augmented versions of each that credit a tool for finding real issues the golden set missed. Results filter by severity — Critical, Medium, Low — and by category: Correctness, Security, Reliability, Maintainability, Testing. A tool that scores well on Low-severity Maintainability catches and says nothing about whether it catches the Critical-severity Security bug that actually matters. Collapsing all of that into one leaderboard number is how benchmarks end up measuring the wrong thing well.

Check it against production, not just itself

The step most benchmarks never take: GitHub ran a multi-model ensemble experiment and compared the offline ReviewBench numbers against what actually happened when the same change shipped to production. The addressed rate — a rough proxy for precision, since it tracks how often a flagged issue actually got fixed — rose 8.0% in both the offline and online measurements. Recall improved 13.6%. Comment volume rose 61%, and critical comments rose 227% offline versus 262% in production — not an exact match, but pointed in the same direction, which is the bar that matters. Cost per review dropped 8.0%. GitHub’s own framing is blunt about what this is for: “offline changes have consistently pointed in the same direction as what we see later in production.” That’s a benchmark claiming to be predictive, with the comparison published to back it up, rather than a benchmark claiming to be definitive.

What to actually do with this if you’re choosing a code review tool

Don’t take any vendor’s headline accuracy number at face value, including GitHub’s own Copilot Code Review, which the company evaluated internally using this same benchmark across iterations — that’s useful context, not proof the tool is unbiased by the benchmark it was tuned against. Ask whoever built the eval you’re being sold four questions: what’s the real-world distribution the test set was sampled from, how many independent sources fed the ground truth, who audited the labels and what was the agreement rate, and has anyone checked the offline score against what shipped. ReviewBench answers all four in public. Most vendor benchmarks answer none of them, and the gap between those two states is the actual thing you’re buying when you pick a code review tool.