Check the number before you publish it
Four of the cheapest checks that catch an overstated eval claim, free and in-browser — nothing is sent anywhere. The same checks behind the case studies and the open-source evalgate.
1 · Does your subset win survive correction?
Report the subset/metric/checkpoint where a model looks best and you’re reporting the max of many noisy tests. Correct for how many you could have picked.
2 · Is the judge winning, or just biased?
Count the verdicts a side won — the longer answer, the first-listed one, the judge’s own family — and test it against chance.
3 · Does one point flip your slope?
Paste your x y points (one per line). Fits a power-law exponent on log-log axes, then refits leaving each point out — and flags if dropping one crosses your threshold.
4 · Can your test set tell the models apart?
Two models on a fixed test set. This gives the two-proportion significance of the gap and the minimum detectable effect — the smallest gap this sample could reliably see. Below the MDE, the ranking is a coin flip.
These run the same math as evalgate. They’re the cheap DIY version — the value of an audit is running all of them, adversarially, on a number you’re too close to.