Editorial illustration for New Analysis Index Widens Gap Between Open and Closed AI Models
Open AI Models Fall Behind in New Benchmark Index
New Analysis Index Widens Gap Between Open and Closed AI Models
Kimi K3.1 Pro launched in fourth place on Artificial Analysis's Intelligence Index. Version 5 of that same index puts it ninth, behind eight closed models. The firm rebuilt its benchmark from the ground up, and the fallout lands hardest on open-weight models that looked competitive under the old rules.
The headline number is the gap at the top of the leaderboard: 12 points between open and closed models now, up from just 3 before the overhaul. Grok 4.7 lost 12 points too. The main change driving this is methodological. For the first time, 60 percent of the index draws on private test sets, a direct response to "benchmaxxing," where labs train models on questions that closely mirror what graders will later ask.
Kimi K3.1 Pro's slide suggests the previous version flattered certain models rather than measuring them accurately. But closing one gap opens another. Private test sets make it harder to game the benchmark, and also harder for outsiders to catch errors buried in the questions themselves. Artificial Analysis's own AA-Omniscience test needed revisions for ambiguous and flawed items before this release, a sign that opacity cuts both ways.
Artificial Analysis overhauled its Intelligence Index, causing Kimi K3.1 Pro to drop five spots in the rankings. The model originally launched in fourth place, but it falls to ninth behind eight closed models in version 5. Artificial Analysis now puts the gap between open models and the top of the leaderboard at 12 points, up from 3 points previously.
Why this matters
A five-spot drop for Kimi K3.1 Pro tells us more about the previous index than about the model itself. If 40 percent of a benchmark was vulnerable to benchmaxxing, every leaderboard screenshot shared on social media or cited in a pitch deck over the past year deserves a second look. For developers choosing between open and closed models, the practical question isn't which one topped a chart last quarter, it's which benchmark that chart was built on and how much of it the model's training data could have touched.
Artificial Analysis moving 60 percent of its index to private test sets is a tacit admission that public benchmarks have become a training target rather than a measurement tool. Nathan Lambert's skepticism is worth sitting with: if closing the measurement gap widens the open-closed gap from 3 points to 12, that's not noise, that's a correction. Founders betting roadmaps on open models performing near parity with GPT-class systems should treat this as a prompt to rerun their own evaluations, not wait for the next index revision to do it for them.
Related Reading
Research & Benchmarks Microsoft AI Chief: "If It Isn't Safe We Shouldn't Build It
Research & Benchmarks Google's FACTS benchmark shows 70% factuality ceiling across four tests
Research & Benchmarks Nvidia's DLSS 4.5 beta adds 6x Multi Frame Generation for RTX 50 GPUs
From Archives FEMA training may fade as AI emergency alerts approach and readiness lags
From Archives Meta quietly launches Waddle World, a 16-bit AI pet journal app.
Common Questions Answered
How did Kimi K3.1 Pro's ranking change after Artificial Analysis rebuilt its Intelligence Index?
Kimi K3.1 Pro dropped five spots in the rankings, falling from fourth place in the previous index to ninth place in version 5. This significant drop reflects changes in how Artificial Analysis evaluates and benchmarks AI models rather than changes to the model itself.
What is the new gap between open and closed models on Artificial Analysis's Intelligence Index?
The gap between open and closed models has widened to 12 points in version 5 of the index, up from just 3 points in the previous version. This substantial increase demonstrates how the index overhaul disproportionately affected open-weight models that previously appeared competitive.
Why does the article suggest previous Intelligence Index rankings deserve a second look?
According to the article, approximately 40 percent of the previous benchmark was vulnerable to benchmaxxing, meaning models could be optimized specifically to perform well on those metrics. This vulnerability means that leaderboard screenshots and rankings cited over the past year may not accurately reflect true model performance across diverse real-world applications.
What should developers prioritize when choosing between open and closed AI models according to this analysis?
Rather than selecting models based on which one topped a chart in the previous quarter, developers should focus on understanding which benchmark the chart was built on and how much of it the model was optimized for. This approach helps ensure they select models that will perform well on their specific use cases rather than just on particular benchmarks.
Further Reading
- Artificial Analysis Intelligence Index v5 Widens Open-vs-Closed Model Gap to 12 Points - AI Weekly
- Claude Opus 5.5 Takes Top Spot as Artificial Analysis Launches Intelligence Index v5 - Startup Fortune
- Kimi K3.1 Pro launches at 4th on the Intelligence Index and will be the leading open-weights model once its weights are published - Artificial Analysis
- Chinese Open-Weight Model Kimi K3.1 Pro Closes In on U.S. Leaders - Heise
