Editorial illustration for SemiAnalysis: No AI agent scores above 12%; safety filters reshuffle rankings
AI Agents Fail New Benchmark: None Score Above 12%
SemiAnalysis: No AI agent scores above 12%; safety filters reshuffle rankings
SemiAnalysis built a benchmark meant to stop AI companies from grading their own homework in a friendly classroom. GDPx throws AI agents into something closer to an actual job: eight-hour stretches operating software like Siemens NX, digging through data rooms with thousands of files, or probing live enterprise systems for weaknesses. The 206 tasks cover eleven fields, from chip design and biology to quant trading, cybersecurity, and law.
Notably absent are administration and insurance, categories SemiAnalysis says don't reflect where the real money sits. The firm argues the first trillion dollars in AI revenue will come from industries where frontier models actually move the needle, not from answering trivia.
That's also the pitch for why GDPx exists at all. SemiAnalysis frames it as a direct rebuttal to OpenAI's GDPval, which it says hands agents all the context they need upfront and skips the messy, unstructured environment of real work. GDPx strips that scaffolding away. The result, before any scores get mentioned, is a test built to be harder to game and harder to ace, which raises the obvious question of how the current crop of agents actually holds up once the training wheels come off.
The new GDPx benchmark from SemiAnalysis drops AI agents into simulated knowledge work. No model scores above 12 percent. Some critics question how meaningful the test really is.
Why this matters
GDPx is a reality check for anyone pricing agent capability off marketing decks. A 12 percent top score across 206 realistic tasks tells us these systems still struggle with the kind of multi-hour, multi-tool work that actual jobs require, not the curated demos vendors show off. The quant trading number is the sharpest data point here: 70 of 111 runs underperforming a basic reference strategy means these agents aren't just imperfect, they're often worse than doing nothing clever at all. That should worry anyone building products that assume competence by default.
The safety filter finding matters just as much for how we read benchmarks going forward. If removing cyber, biology, and alignment domains flips the leaderboard, that means published rankings are partly measuring how cautious a provider's guardrails are, not raw task ability. For developers and founders choosing between Opus 5.5, Gemini 4 Carbon, and GPT-6 Astra, that's a confound worth separating out before making procurement decisions. Researchers should treat GDPx as an early, useful stress test, and treat every single-number agent ranking with more suspicion than before.
Related Reading
Research & Benchmarks Microsoft AI Chief: "If It Isn't Safe We Shouldn't Build It
Research & Benchmarks Google's FACTS benchmark shows 70% factuality ceiling across four tests
Research & Benchmarks Nvidia's DLSS 4.5 beta adds 6x Multi Frame Generation for RTX 50 GPUs
From Archives Full-Stack Codesign Solves AI’s Memory Challenge Across Recent Models
From Archives Anthropic FlashProve system aims to verify AI model identities
Common Questions Answered
What is the GDPx benchmark and how does it differ from other AI agent tests?
GDPx is a benchmark created by SemiAnalysis designed to test AI agents on realistic, multi-hour work tasks rather than curated demos. Unlike friendly classroom-style tests, GDPx throws AI agents into actual job scenarios across eleven fields, including chip design, biology, quant trading, cybersecurity, and law, with 206 tasks that require operating real software like Siemens NX and navigating complex data environments.
Why is the 12 percent maximum score significant for AI agent capabilities?
The 12 percent top score across 206 realistic tasks demonstrates that current AI agents still struggle significantly with the kind of multi-hour, multi-tool work required in actual jobs. This low performance reveals a substantial gap between vendor marketing claims and real-world capability, showing these systems are not yet ready for autonomous knowledge work tasks.
What does the quant trading performance data reveal about AI agent limitations?
In quant trading tasks, 70 out of 111 AI agent runs underperformed a basic reference strategy, meaning the agents were often worse than doing nothing at all. This sharp data point illustrates that AI agents don't just perform imperfectly on complex financial tasks—they frequently fail to match even simple baseline approaches, highlighting critical limitations in their decision-making abilities.
Which industries and fields are covered by the GDPx benchmark?
The GDPx benchmark covers eleven fields including chip design, biology, quant trading, cybersecurity, and law, with 206 tasks designed to simulate real knowledge work across these domains. Notably absent from the benchmark are administration and insurance categories, focusing instead on technical and specialized professional fields.
Further Reading
- No AI agent scores above 12 percent on SemiAnalysis' new real-world work benchmark - The Decoder
- GDPx: Measuring Frontier Model Capabilities Across the Most Economically Valuable Domains - SemiAnalysis
- GDPx Benchmark: Best Frontier AI Scores 12% on Real Work, Zero on Finance and Legal - Tech Times
- GDPx: SemiAnalysis's New Agent Benchmark Tops Out at 12% - BitsMinds
