Editorial illustration for Is the #1 AI Model Really #1 Across Eight Draws with Replacement?
Top AI Models Statistically Indistinguishable, Study Finds
Is the #1 AI Model Really #1 Across Eight Draws with Replacement?
A leaderboard puts a 71.8% score above a 70.0% score and calls it a ranking. Eight small open models, tested on the same 2,000 questions pulled from MMLU, say otherwise. Run the top two contenders through a standard significance test at the usual 5% threshold, and they can't be told apart. Shrink the test to 200 questions, the size plenty of benchmarks actually use, and the model in second place would have won about three times out of ten, just from which questions happened to get picked.
The problem gets worse on smaller benchmarks. AIME has 30 problems, so a single question is worth 3.3 points. Two models can't score 1.8 points apart on it.
They're separated by zero points, or 3.3, or 6.7, nothing in between. That's an extreme case, but every benchmark built from a finite list of questions has some version of the same wobble built in. Evan Miller at Anthropic laid out the fix for this in 2024: error bars on eval scores, head-to-head comparisons on shared questions, and a rule for how many questions a test needs before the numbers mean anything.
That approach, applied to a leaderboard built from scratch, is where this starts.
With 30 questions, the model that is #1 on the full set is #1 again in 45% of draws. With 100 questions it's 60%, with 200 it's 70%, with 500 it's 82%, and with 1,000 it's 91%. Even using all 2,000 questions it's 96%, not 100%.
Why this matters
A 1.8-point lead on 2,000 questions sounds decisive until you run the bootstrap and watch it dissolve into noise at the standard 5% threshold. At 200 questions, the "best" model loses its crown in roughly three of ten resamples. That's not a rounding error, that's the leaderboard itself being unstable.
For developers picking an API based on a single-point score, or founders citing a benchmark rank in a pitch deck, this is a warning about precision theater: 71.8% versus 70.0% reads as settled fact when it's closer to a coin flip with weighted odds. Researchers building these evals should be reporting confidence intervals or resampling distributions alongside headline numbers, not just a static table. Until that becomes standard practice, treat any "#1 model" claim as provisional, especially on test sets under a few thousand items.
The next thing worth watching: whether major leaderboard maintainers like LMSYS or Hugging Face start publishing uncertainty bands by default, or whether this stays a niche complaint from people willing to rerun the math themselves.
Related Reading
Research & Benchmarks Microsoft AI Chief: "If It Isn't Safe We Shouldn't Build It
Research & Benchmarks Google's FACTS benchmark shows 70% factuality ceiling across four tests
Research & Benchmarks Nvidia's DLSS 4.5 beta adds 6x Multi Frame Generation for RTX 50 GPUs
From Archives Block Unveils Dessa, a Full-Duplex Speech Model Trained Only on Speech
From Archives Google DeepMind releases Gemini 4 Carbon; independent evaluations back gains
Common Questions Answered
Why can't a 1.8-point difference between two AI models be considered statistically significant?
When the top two contenders are run through a standard significance test at the 5% threshold, they cannot be statistically distinguished from each other despite the 71.8% versus 70.0% score difference. This means the 1.8-point gap falls within the margin of error and could easily be due to random variation rather than a true performance difference.
How does test size affect the stability of AI model rankings according to the bootstrap analysis?
The article demonstrates that with only 30 questions, the top-ranked model remains #1 in just 45% of resampled draws, but this stability increases dramatically with larger test sets: 60% at 100 questions, 70% at 200 questions, 82% at 500 questions, and 91% at 1,000 questions. This shows that smaller benchmarks like the 200-question tests commonly used produce highly unstable rankings where the second-place model would win about three times out of ten.
What is the practical risk of relying on single-point benchmark scores when selecting an AI model API?
Developers and founders who make decisions based on a single leaderboard ranking are engaging in what the article calls 'precision theater,' trusting in scores like 71.8% that appear decisive but are actually unstable and unreliable. The bootstrap analysis reveals that even with 2,000 questions, the top model only maintains its ranking 96% of the time, meaning apparent winners can be misleading when test sizes are smaller.
Why is the MMLU benchmark of 2,000 questions still not sufficient to guarantee a stable model ranking?
Even when using all 2,000 questions from MMLU, the top-ranked model only retains its position 96% of the time across resampled draws, not 100%. This indicates that random variation in which specific questions are selected still introduces enough noise to occasionally flip the rankings, demonstrating that benchmark stability is a fundamental challenge rather than a rounding error.
Further Reading
- Data Efficient Evaluation of Large Language Models and Text-to-Text Generation Models - arXiv
- A Reliability Audit of Cybersecurity LLM Benchmarks - arXiv
- LLM Benchmark Methodology 2026: Reading Leaderboards - Digital Applied
- Technical Performance - Stanford HAI AI Index
- Lexical and Syntactic Sensitivity in LLM Evaluation - ACL Anthology
