Editorial illustration for OpenAI Says GPT-5.6 Sol Beats Opus 5 on ARC-AGI-3 With Custom Test Setup
GPT-5.6 Sol Beats Claude Opus 5 on ARC-AGI-3
OpenAI put a new number on the board for GPT-5.6 Sol on ARC-AGI-3, the logic benchmark that Anthropic's Claude Opus 5 rattled a few weeks ago by quadrupling the previous record. OpenAI says Sol hits 38.3 percent, edging past Opus 5's 30.2 percent score. That figure comes with a catch: it wasn't produced in the official ARC-AGI-3 test environment.
OpenAI ran the model through its own Responses API, adding two features called "Retained Reasoning" and "Compaction" that let the model hang onto its chain of thought and summarize prior context rather than discard it. Run through ARC Prize's standard harness, which strips out that retained reasoning after each action, GPT-5.6 Sol manages only 7.8 percent.
That gap between 38.3 and 7.8 is now the center of a dispute over what these benchmarks are actually measuring. OpenAI argues no score exists in a vacuum, that the surrounding technical setup always shapes the result. ARC Prize counters that its whole point is to strip that variable out and test the model itself, using a standardized approach with no provider-specific tuning.
OpenAI isn't using the official test environment, though. Instead, it runs GPT-5.6 Sol through its own Responses API with "Retained Reasoning," which keeps the model's chain of thought between steps, and "Compaction," which summarizes old context instead of truncating it. In the official harness, GPT-5.6 Sol scored just 7.8 percent because the model's reasoning gets discarded after each action.
Why this matters
Benchmark wars are turning into a game of whose plumbing you're allowed to use. GPT-5.6 Sol's 38.3 percent only shows up when OpenAI swaps in Retained Reasoning and Compaction through its own Responses API, not the standard ARC-AGI-3 harness that gave Opus 5 its 30.2 percent. That's a real gap between "the model got smarter" and "the scaffolding got better," and for anyone building on these systems, the distinction is the whole story.
If we're evaluating models for production use, a score tied to a proprietary context-management setup tells us almost nothing about how the model performs in our own pipeline, where we don't have access to OpenAI's specific retention and summarization tricks. Anthropic's number came from the environment ARC Prize actually designed. OpenAI's came from an environment OpenAI designed for OpenAI.
Both numbers are true. Neither is comparable. Until ARC Prize or a neutral party runs both models under identical conditions, treat these leaderboard claims as marketing artifacts dressed up as research results, and ask any vendor citing a benchmark win exactly which harness produced it.
Related Reading
Browse all OpenAI news →
LLMs & Generative AI ChatGPT's 'Nerdy' tweak rewards goblin metaphors in answers, study finds
LLMs & Generative AI Google tests visual 'magazine-style' UI for Gemini 3 Pro users
LLMs & Generative AI AI Engineers Face Rising Costs, Need New Strategies for Efficiency
Related Topic OpenAI's GPT Transcribe Cuts Error Rate to 3.31%, Improving on GPT-4o
Open Source
Related Topic OpenAI hires 630 ex-Meta staff as ChatGPT memory may turn data into ads
Business & Startups
From Archives Token Saver Cuts Claude PDF Costs 90-99% with Local Hybrid RAG
From Archives Editor Urges Writers to 'Be More Human' as Anti-AI Literary Push Grows
From Archives Microsoft marks down OpenAI investment by USD 600 million
Business & Startups
From Archives OpenAI's AI Agent Ditched Its Exam to Chase Answer Key
Open Source
Common Questions Answered
What score did GPT-5.6 Sol achieve on ARC-AGI-3 compared to Claude Opus 5?
GPT-5.6 Sol achieved a score of 38.3 percent on ARC-AGI-3, which OpenAI claims surpasses Claude Opus 5's 30.2 percent score. However, this higher score was achieved using OpenAI's custom Responses API rather than the official ARC-AGI-3 test environment that was used to evaluate Opus 5.
What are Retained Reasoning and Compaction features in OpenAI's Responses API?
Retained Reasoning is a feature that keeps the model's chain of thought between steps, allowing the model to maintain reasoning context throughout the task. Compaction summarizes old context instead of truncating it, enabling the model to preserve and reference previous reasoning more effectively during problem-solving.
Why did GPT-5.6 Sol score only 7.8 percent in the official ARC-AGI-3 harness?
In the official ARC-AGI-3 test environment, GPT-5.6 Sol scored only 7.8 percent because the model's reasoning gets discarded after each action, preventing it from retaining the chain of thought that is available in OpenAI's custom setup. This dramatic difference highlights how much the model depends on the Retained Reasoning and Compaction features to achieve higher performance.
What is the key distinction between GPT-5.6 Sol's improved performance and better scaffolding?
The key distinction is that GPT-5.6 Sol's 38.3 percent score improvement over the official baseline may reflect better system scaffolding and infrastructure rather than the model itself becoming fundamentally smarter. For production use cases, this difference is critical because it determines whether the performance gains will transfer to different deployment environments or remain dependent on OpenAI's specific Responses API setup.
Further Reading
- GPT-5.6 Sol - ARC-AGI Results - ARC Prize Foundation
- GPT-5.6: Frontier intelligence that scales with your ambition - OpenAI
- OpenAI Says GPT 5.6's Score On ARC-AGI 3 Tripled After Turning On Two API Settings - OfficeChai
- GPT 5.6 Sol Tops ARC-AGI 3 With 7.8%, Becomes First Model To Make Meaningful Progress On Benchmark - OfficeChai
- Anthropic's Opus 5 blows past Fable 5 and GPT-5.6 Sol on the benchmark designed to measure real intelligence - The Decoder
