The model shown second wins more
Human preference is the gold standard. It's what we use to check whether an AI judge is any good. So it's worth asking the uncomfortable question about the human vote itself: is it fair to both sides? On the most-watched LLM leaderboard, the answer is almost — but not quite. The model shown second wins a little more often, and it isn't because it's the better model.
LMArena ranks models from blind, side-by-side human votes: you see two answers, A and B, and pick the one you prefer. The order is randomized, so over enough votes any left/right quirk should wash out. I checked whether it actually does, using the public battle data (lmarena-ai/arena-human-preference-140k).
The second-shown model wins 1.25 points more
| Measure | Decisive votes | Position A wins | Position B (second) wins | p |
|---|---|---|---|---|
| Raw | 98,348 | 49.38% | 50.62% | 9.8 × 10⁻⁵ |
| Within-matchup (controlled) | 98,332 | 49.38% | 50.62% | 9.0 × 10⁻⁵ |
Across ~98,000 decisive votes, the model in the second slot wins 50.62% of the time versus 49.38% for the first — a 1.25 percentage-point edge, with a p-value near 10⁻⁴. Small, but far too consistent to be chance at this scale.
It's the position, not the model
The obvious objection: maybe the stronger models just happen to land in the second slot more often. If so, this is about model strength, not order. So I controlled for it the clean way — the within-matchup test.
Does this break the leaderboard? No — and that's the interesting part
Here is the honest, two-sided reading. Because LMArena randomizes A/B, each model spends about equal time in both slots, so the bias mostly averages out in the aggregate ranking. (In a separate audit I found the arena's #1 holds up decisively — so this doesn't overturn it.) The bias is real but largely laundered by the randomization.
What it does mean:
- Any ranking built on human votes that doesn't randomize order, or doesn't average both orders, silently inherits a ~1-point thumb on the scale.
- For models with few or lopsided battles, the averaging is incomplete — a close call between two near-tied models can be decided by presentation, not quality.
- It's a clean caution for the booming practice of LLM-as-judge: AI judges have a well-known, much larger position bias — and here we see that even the human gold standard we compare them to isn't immune.
The fix is one design choice
Randomize order and average both presentations before you score — or, better, show each matchup both ways and combine. It costs nothing and removes the thumb from the scale. The general rule is the one this whole series keeps arriving at: the measurement has its own biases, and you only find them by measuring the measurement.
Data:
lmarena-ai/arena-human-preference-140k (public) — columns model_a, model_b, winner.Keep decisive votes (drop
tie/both_bad): 98,348. Position-B win rate = 50.62% (z = −3.90, p = 9.8e−5).Within-matchup control: for each unordered pair present in both orders (1,197 pairs), pool wins by slot — B = 50.62%, +1.25 pp, p = 9.0e−5. Pure-stdlib, deterministic.