Is the #1 real?
An eval-integrity audit that only ever debunks is a naysayer, not an auditor. The test of the method is whether it can also certify — say "this number holds" and show why. So I pointed it at the most-watched leaderboard in the field, the one whose #1 gets cited in every launch post: LMArena. The #1 holds. Decisively. And the reason it holds is the useful part.
I have been auditing whether leaderboard #1s are statistically real. On SWE-bench Lite/Verified the answer is often no — the top rank is a coin flip among a saturated tie, because the frontier submissions sit within a task or two of each other on a fixed, finite test set. LMArena is the natural counter-test: unlike a static benchmark it keeps accumulating human votes, and it already publishes confidence intervals. Does its #1 survive an independent recomputation?
What I did
I took the public arena battle data — lmarena-ai/arena-human-preference-140k: model_a, model_b, winner per human vote — kept the decisive votes, fit my own Bradley-Terry ranking, and bootstrapped over battles to get, for every model, a 95% rank confidence interval and P(this model is truly #1). No dependence on LMArena's published numbers. The fit and bootstrap are pure standard-library; only reading the parquet needs a data lib.
Result: the #1 is not a coin flip
| Audit slice | Decisive battles | Models | #1 | Rank CI | P(#1 truly #1) |
|---|---|---|---|---|---|
| Full arena | 98,088 | 51 | gemini-2.5-pro | [1, 1] | 1.00 |
| Coding votes only | 28,287 | 52 | gemini-2.5-pro | [1, 2] | 0.96 |
| One 19k shard | 13,872 | 48 | gemini-2.5-pro | [1, 3] | 0.83 |
On the full ~98k decisive battles, gemini-2.5-pro is the top model in 100% of bootstrap resamples, with a rank confidence interval of exactly [1, 1]. The runner-up (a gemini-2.5-pro preview build) sits at rank [2, 3], clearly below. The #1 holds when you restrict to coding votes — still first, P = 0.96. This is the opposite of a saturated tie: the lead is far larger than sampling noise.
The point: a #1 is only as real as the resolving power behind it
Here is the two-sided lesson, and it lives in that third row. On a single 19k-battle shard the very same audit gave gemini-2.5-pro only P = 0.83 with a three-way tie at the top — it looked much less certain. Five times the battles collapse that ambiguity to a clean [1, 1]. Nothing changed about the models; the ranking just crossed from under-powered to resolved.
Why a certify matters as much as a debunk
If you are about to hire someone to check your evaluation, your real fear is that they will simply trash it — that "auditor" means "person who finds a reason to say no." This case is the answer. The honest display in both directions is the same object: a rank plus the confidence interval or tie-group around it. When the interval is wide, I say the ordinal isn't supported yet. When it's [1, 1], I say so, and show the 98,088 comparisons that make it so. LMArena's practice — publishing CIs, gathering enough battles — is exactly what makes its ordinal defensible, and the audit certifies it.
Data:
lmarena-ai/arena-human-preference-140k (public). Read model_a/model_b/winner; build decisive battles (winner_model, loser_model).Fit Bradley-Terry (MM iteration, pure-stdlib); bootstrap B resamples over battles for each model's rank CI and P(#1).
Full set: 98,088 decisive battles, 51 models (≥300 games) → rank 1
gemini-2.5-pro, CI [1,1], P(#1)=1.00; runner-up rank [2,3].Coding-only: 28,287 battles → still #1, P=0.96. One 19k shard → P=0.83, 3-way tie. Deterministic given seed.