An independent, reproducible audit of the major public LLM leaderboards — is each
printed #1 a statistically real ranking, or a saturated coin flip? Same method certifies and debunks.
6
SWE-bench boards audited
5
print a #1 the data can't resolve
3
have a genuinely real #1
Leaderboards
SWE-bench Test
1,424-task full split — the resolving-power control
REAL #1
P(printed #1 is the true #1)100%
P(#1) since 2023-10-28 · flipped 2025-02-28
submissions × items24 × 1424
tied for #11
#1 survives split100%
resolvable tiers17 / 24
current #1 · sonar-foundation-agent_claude-opus-4-5
Real #1 — separates from #2 and survives random task splitsCoin flip — several submissions statistically tied; the title flips on re-splitUnresolved — only partly separated
The Leaderboard Confidence Standard
A leaderboard prints an ordinal next to each row, and we read it like a stopwatch at a
finish line. On a saturated board that ordinal is often an accident of which tasks were in the set.
The fix costs nothing — four lines any leaderboard can adopt:
Show a rank confidence interval, not a bare ordinal — the 95% CI of each row's rank under a bootstrap over the evaluation units.
Mark the significance group tied for #1 (paired McNemar on the shared items) instead of a single crowned #1.
Report the resolving power — how many statistically-distinguishable tiers the board actually supports.
Only print a lone #1 when it survives a split-half re-test. Where it does — like SWE-bench Test — say so loudly; it's earned.
Watch a leaderboard, or audit one
Leaderboard Monitor
Pick a benchmark and I'll watch it — you get an alert the moment its #1 flips between
real and coin-flip, plus a monthly confidence check. The board above, kept honest for you.
This page is the free summary. The full report — for a public board or your own —
adds what a headline can't show, as a citeable PDF for a paper, launch, or diligence:
a 95% confidence interval on every submission's rank & score
the exact significance group, with paired-test p-values
reliability, frontier resolving-power & the latent-skill breakdown
the winner's-curse correction and per-finding remediation
Put the confidence verdict on your model's README or paper — a live badge that updates when the
ranking does. One per board, plus a summary:
[](https://ipezygj.github.io/eval-audit-site/status.html)
Or consume the verdicts as JSON:
status.json — a public API for agents and tools.
Method. Bootstrap rank confidence intervals over the evaluation units; paired McNemar for the
top significance group; 50× split-half stability; Bradley-Terry + Condorcet for pairwise data. Pure
standard-library Python, fixed seeds, reproducible from the public per-instance results. Verdicts are
computed from the numbers, never asserted.
Ilpo Väätäinen — independent measurement & eval-integrity audits ·
ipezygj2@gmail.com ·
Measured, Not Believed Last audited 2026-07-23 16:49 UTC · recomputed on schedule from live data
