Editorial illustration for Karpathy's LLM Council Ranks GPT-5.1 Top Amid AI Model Comparison
LLM Council Ranks GPT-5.1 Top in Revolutionary AI Evaluation
Gemini 3.0, Claude, and Grok rank GPT-5.1 top in Karpathy’s LLM Council
Silicon Valley loves a bake-off, a cage match, some ranking where one model can be declared winner. Usually it's a team of humans with a scoring rubric. Andrej Karpathy tried something else. He made the models judge each other.
His method is brutally simple. Pose a question. Get answers from all of them, stripped of any identifying marks.
Then show each model the anonymous responses and ask for a ranking. The results are stark. GPT-5.1 keeps winning.
Not by a little. Its direct competitors, Gemini 3.0, Claude, and Grok, all keep picking it as the best.
They even agree on who's worst. It's Claude.
"Quite often, the models are surprisingly willing to select another LLM's response as superior to their own, making this an interesting model evaluation strategy more generally," said Karpathy. "For example, reading book chapters together with my LLM Council today, the models consistently praise GPT 5.1 as the best and most insightful model, and consistently select Claude as the worst model, with the other models floating in between." Karpathy's experiment setup is a three-step loop. First, the user's query is sent to all models separately, and their answers are shown side-by-side without revealing who wrote what. Next, each model sees the others' responses, still anonymised, and ranks them based on accuracy and insight.
Forget the leaderboards. This is a different kind of truth. It's one thing for OpenAI to claim its model is smarter.
It's another for its rivals, in a blind test, to quietly endorse that claim while demoting one of their own. The humility Karpathy notes is eerie. These systems are designed to project confidence, to answer.
Here, they are programmed to compare and, often, to reject their own work in favor of another's.
The real sting is for Claude. Being voted weakest by a jury of your peers is a harsh review. It suggests a qualitative gap the other models perceive, whether that's a lack of depth or a style that simply reads as less insightful.
Karpathy's council cuts through the noise. It creates a closed loop where the only opinion that counts is the model's. And right now, that opinion is unanimous.
Related Reading
Browse all OpenAI news →
LLMs & Generative AI ChatGPT's 'Nerdy' tweak rewards goblin metaphors in answers, study finds
LLMs & Generative AI Google tests visual 'magazine-style' UI for Gemini 3 Pro users
LLMs & Generative AI AI Engineers Face Rising Costs, Need New Strategies for Efficiency
Related Topic OpenAI hires 630 ex-Meta staff as ChatGPT memory may turn data into ads
Business & Startups
Related Topic Sen. Warren urges Trump admin to expand AI credit after OpenAI letter
Business & Startups
From Archives Gemini 3 Pro builds screenshot-to-code app in two prompts, fixes bugs
From Archives Google's nested learning, based on brain's fast-slow circuits curbs LLM forgetting
From Archives Gemini 3 Pro and GPT-5 stumble on graduate-level physics benchmark
From Archives OpenAI and Foxconn to co-design multiple generations of AI data-centre racks
Business & Startups
Common Questions Answered
How does Andrej Karpathy's LLM Council approach model evaluation differently from traditional benchmarking?
Karpathy's LLM Council involves language models directly assessing and ranking each other's performance, creating a unique peer-review dynamic. Unlike traditional benchmarking, this method allows AI models to collaboratively evaluate responses, with models often willing to acknowledge another model's superior performance.
Which AI model was ranked top in Karpathy's LLM Council experiment?
GPT-5.1 emerged as the top performer in the LLM Council experiment, consistently being praised by other models as the most insightful. The models participating in the evaluation unanimously selected GPT-5.1 as the best model in the comparative assessment.
What surprising outcome did Karpathy observe during the LLM Council evaluation?
Karpathy noted that the models were surprisingly willing to select another language model's response as superior to their own, creating an innovative self-assessment approach. Additionally, the models consistently ranked Claude as the worst performer in the evaluation, while other models were ranked in between.
Further Reading
- Gemini 3 vs GPT-5 vs Claude 4.5 vs Grok 4.1 - The Ultimate Reasoning Performance Battle — Vertu
- Grok 4.1 vs Gemini 3.0 : 2025 Frontier AI Showdown — Skywork.ai
- Claude vs. GPT-4.5 vs. Gemini: A Comprehensive Comparison — Evolution.ai
- AI Models Comparison 2025: Claude, Grok, GPT & More - Collabnix — Collabnix
- The Best AI in October 2025? We Compared ChatGPT, Claude, Grok, Gemini & Others — FelloAI
