Measured, Not Believed
Why AI benchmark scores and trading backtests overpromise — and how to catch them. Twenty chapters of real, reproducible audits of the numbers people compute and then believe.
A benchmark score is not a model’s quality. A backtest return is not a strategy’s profit. A win rate is not an edge. Each is a number produced by a procedure, standing in for a thing in the world it resembles but is not — and the entire craft, in AI evaluation and in trading alike, is holding onto that distinction under the constant pressure to forget it. This book is a field guide to the handful of ways that distinction breaks, each shown on a real, named number, each with the cheap check that catches it.
What’s inside
Part I turns the lens on AI evaluation — the leaderboards and LLM-judge results that decide which model “wins”:
- The judge rewards length. An LLM judge that prefers the longer answer 68% of the time, and its own model family 71.5% — computed on the benchmark’s own file. (read the audit)
- The luckiest of 23 tries. A “we lead on subset X” win that evaporates once you correct for how many subsets it could have been picked from. (read the audit)
- The exponent that came free. A “super-linear” scaling law that a constant substitution already produces, and one point can flip. (read the audit)
- Length as proven cause, not correlation; contamination and self-preference; and the toolkit that runs these checks.
Part II turns the same discipline on money — a real trading system taken live, and what the backtest hid: fees that beat the edge, a queue that can’t be simulated, the false negative, and the regime that ended it. Part III is the method as a repeatable audit — reproduce, control, measure the thing the model skipped — applied in the wild to published papers, and turned back on the author’s own work.
Who it’s for
Anyone who computes a number and then has to decide whether to believe it: ML researchers and eval-tooling teams shipping benchmark claims, quants and traders reading backtests, and technical readers who want to stop being fooled by a tidy summary statistic — including their own.
By Ilpo Väätäinen, who also runs an independent eval-integrity audit practice and maintains the open-source evalgate checks. The book is the fuller story behind both.