Eval Integrity Audit — sample report

Every engagement ends in one document like this: which claims hold, which don’t, and the one-line statistical fix for each that doesn’t. Nothing is hand-waved — each verdict is reproducible from your data.

Client

Reward-model company (fictional — “Acme RM”)

Engagement

Full Eval Audit — pre-launch leaderboard claims

Scope

3 headline claims from a draft launch post + model card

Turnaround

5 business days

Auditor

Ilpo Väätäinen — independent eval-integrity practice

1 · Executive summary

Of the three headline claims audited, one is certified as stated, and two require revision before publication — each is a specific, reproducible over-statement, not a judgment call.

2 · Findings

“We lead on the Reasoning subset” is the best of 18 noisy tests.

The leaderboard is scored across 18 subsets, and the draft post reports the single subset where Acme RM leads. That win looks significant on its own (raw p = 0.012), but it is the maximum of 18 comparisons. Corrected for the family it was chosen from (Šidák over 18 subsets), it lands at p = 0.19 — above the 0.05 bar. Between two equally-good models at 18 subsets you expect ~0.9 “significant” subset wins from noise alone.

The “91% human agreement” is entangled with answer length.

The judge’s headline agreement number is real, but on the pairwise verdicts where the two answers differ in length, the judge picks the longer one 63% of the time (252 / 400, p ≈ 2 × 10⁻⁷). A large part of “agreement with humans” may be “agreement with length,” which the raw number can’t separate. This does not prove length causes the agreement — but a proxy claim can’t be read cleanly while a presentation variable moves the verdict this much.

“Model B leads on the Safety subset” survives correction.

This subset win holds up: raw p = 0.0003, and after the same 18-way Šidák correction it is p = 0.005 — comfortably below 0.05. This is a genuine subset-level result and can be stated as published. Note the audit certifies it independently of the overall ranking; it earns the claim on its own evidence.

3 · Method & scope

Each claim was reproduced from the client’s released per-subset scores and pairwise judgments. Subset comparisons use a per-subset two-proportion test with Šidák and Bonferroni family-wise correction over the 18 subsets; the judge check is an exact binomial test on length-differing pairs. All computations are deterministic and were returned with the client’s report. This audit examined only the three claims in scope; it does not assess the models’ overall quality.