LLM Council Shows Three Models Deliver Separate Answers in First Stage
Why does a trio of language models answering the same prompt matter? The LLM Council, a project credited to Andrej Karpathy, aims to surface more reliable answers by letting three separate models speak first, then evaluate each other's output. In practice, users submit a query and watch as each model posts its own reply—no aggregation, just raw, unfiltered text.
After that, the system shuffles the responses, stripping away any label that reveals which model produced which answer, and asks the models to rank the set. The idea is that collective judgment, stripped of bias toward a single engine, could surface the most accurate or useful answer. It’s a two‑step dance: speak, then critique.
The design hopes to expose divergences early, then let the models sort the results without knowing the source. This structure sets the stage for the next part of the experiment, where you can actually see the individual answers and the subsequent anonymous rankings.
Then we can see that the first stage is completed and all the three LLM has stated their individual responses. We can see the individual responses by clicking on the LLM names
In the second stage we can see the LLM response rankings by each other without knowing who generated this response. It also
Then we can see that the first stage is completed and all the three LLM has stated their individual responses. We can see the individual responses by clicking on the LLM names In the second stage we can see the LLM response rankings by each other without knowing who generated this response. It also shows the combined ranking of all the council members Now comes the final stage in which the Chairman LLM selects the best answer and presents it before you.
And this is how the LLM Council by Andrej Karpathy works. We tested the installation by asking the Council a complex question: "What is the future of jobs with AI? Will AI make everyone unemployed?" The interface displayed the workflow in real-time as models like Grok, ChatGPT and Llama debated and ranked each other's predictions.
The first stage ends with three models posting their own answers, each visible by clicking the model name. In the second stage those answers are anonymously ranked by the other models, creating a peer‑review loop. By forcing the models to critique one another, the system aims to weed out hallucinations and bias.
Yet the article does not provide metrics on how often the highest‑ranked response proves correct. It remains unclear whether the ranking process consistently filters out weak answers or simply reflects the quirks of the participating models. The council’s design mirrors a human board meeting, where dissenting views are aired before a consensus emerges.
Critics might ask if three perspectives are enough to capture the full range of possible errors. Supporters will note that the anonymous cross‑evaluation removes the temptation to favor a single source. Ultimately, the LLM Council offers a structured approach to multi‑model debate, but its real‑world reliability still needs independent verification.
Results are pending.
Further Reading
More in LLMs & Generative AI
Gemini 3 lets users upload sport videos, using long‑context and vision
Why does this matter for anyone who’s ever tried to improve a swing, a serve, or a sprint?
OpenAI says GPT‑5.2 will debut next week, ahead of Google’s Gemini 3 in internal tests
OpenAI is pulling the plug on its year‑end rollout plan and moving GPT‑5.2 up to next week.
Gemini 3 Pro delivers strongest spatial understanding and reasoning yet
Why does a model’s grasp of space matter to anyone building AI today? Because most vision systems...
Google's Gemini 3 showcases vibe coding power with single‑prompt apps
Google just dropped Gemini 3, its newest large‑language model, and the buzz isn’t about incremental...
Reddit warns AI content loops are degrading its human‑focused community
Reddit’s moderators are sounding the alarm on a feedback loop that’s turning the site’s...
Common Questions Answered
What is the purpose of the first stage in the LLM Council project created by Andrej Karpathy?
In the first stage, three separate language models each generate their own unfiltered answer to the user's query. This stage allows users to see each model's raw response by clicking on the model names, establishing a baseline before any peer review occurs.
How does the second stage of the LLM Council ensure anonymity while ranking responses?
During the second stage, the three models anonymously rank each other's answers after the responses are shuffled and stripped of any identifying labels. This peer‑review loop lets the models critique one another without knowing which model produced which answer, aiming to reduce bias.
What role does the Chairman LLM play in the final stage of the LLM Council process?
In the final stage, the designated Chairman LLM reviews the ranked responses and selects the single best answer to present to the user. This selection consolidates the peer‑reviewed insights into a final output that is intended to be the most reliable.
Why does the article claim the LLM Council could help weed out hallucinations and bias?
The article suggests that by forcing models to critique each other's answers, the system creates a peer‑review mechanism that can identify and suppress hallucinated or biased content. However, it also notes that no concrete metrics are provided to confirm how consistently this process filters out weak answers.