Editorial illustration for Benchmark shows Claude Fable 5 passes only 3% of tasks, 31 of 91 fail 50%
Benchmark shows Claude Fable 5 passes only 3% of tasks,...
Claude Fable 5 achieved perfection exactly three times. That’s the stark finding from a new benchmark testing 91 real-world tasks, where the top model’s flawless scorecard reads a paltry 3%. For 31 separate tasks, every model flunked, unable to muster even a 50% pass rate.
The cost of failure, however, varies wildly. You can spend four cents per task with DeepSeek V4 Flash. Or you can pay over thirty-one dollars for Claude Fable 5.
An 800-fold price premium doesn’t buy competence. It merely purchases a more refined, more expensive error.
The top performer, Claude Fable 5, hits the highest rubric pass rate but still nails all criteria on just 3 percent of tasks. On 31 out of 91 tasks, no model even clears 50 percent.
Forget replacing a skilled analyst. The benchmark’s 31 unimpeachable failures tell a different story. Cheap models fail loudly, forgetting crucial files or outputting gibberish.
The expensive ones, like Claude Fable 5, fail quietly. They execute the obvious steps, then falter on the final, critical act: synthesizing scattered details into a coherent whole. That synthesis is the essence of real knowledge work.
It’s also where every model—from the four-cent option to the thirty-dollar one—completely breaks down. The industry is polishing a broken process. We aren’t purchasing intelligence.
We’re leasing a sophisticated pattern-matcher that still can’t do the job.
Related Reading
Browse all Anthropic news →
Research & Benchmarks Google's FACTS benchmark shows 70% factuality ceiling across four tests
Research & Benchmarks Databricks finds multi-step agents beat single-turn RAG by 21% to 38% on STaRK
Research & Benchmarks Nvidia's DLSS 4.5 beta adds 6x Multi Frame Generation for RTX 50 GPUs
Related Topic Trump cracks down on Anthropic after Amazon tip; staff largely foreign
Open Source
Related Topic Anthropic says Claude writes >90% of its code; AI pause button urged
LLMs & Generative AI
From Archives Nobel laureate John Jumper departs DeepMind for Anthropic after AlphaFold win
From Archives OpenAI shows small 'beneficial trait' training makes AI safer, less manipulable
From Archives Anthropic adds live dashboards, Cloudflare‑compatible code to Claude
LLMs & Generative AI
From Archives Claude Fable (Mythos) 5 shows limited bug‑finding and refactoring aid
LLMs & Generative AI
Common Questions Answered
What does the benchmark reveal about Claude Fable 5's performance on real-world tasks?
Claude Fable 5 achieved perfect scores on only 3% of the 91 real-world tasks tested in the benchmark. Additionally, there were 31 separate tasks where every model tested, including Claude Fable 5, failed to achieve even a 50% pass rate, indicating significant limitations across the board.
How does the cost-per-task comparison between DeepSeek V4 Flash and Claude Fable 5 reflect their performance differences?
DeepSeek V4 Flash costs approximately four cents per task while Claude Fable 5 costs over thirty-one dollars per task. However, the benchmark shows that higher cost does not guarantee better performance, as both models struggle with the same fundamental limitations in real-world task execution.
What is the key difference between how cheap models and expensive models like Claude Fable 5 fail according to the benchmark?
Cheap models fail loudly by forgetting crucial files or outputting gibberish, while expensive models like Claude Fable 5 fail quietly by executing obvious steps correctly but faltering on the final critical act of synthesizing scattered details into a coherent whole. This synthesis capability represents the essence of real knowledge work where all models currently struggle.
Why does the benchmark suggest that AI models cannot yet replace skilled analysts?
The benchmark's 31 unimpeachable failures demonstrate that current AI models, regardless of cost, cannot reliably synthesize scattered details into coherent analysis. This synthesis capability is essential to real knowledge work, and the benchmark shows that every model—from the cheapest to the most expensive options—fails at this critical task.
Further Reading
- Announcing AA-Briefcase: a frontier knowledge work evaluation — Artificial Analysis
- Initial impressions of Claude Fable 5 — Simon Willison's Weblog
- Claude Fable 5 Model Review — CodeRabbit
- Claude Fable 5 & Claude Mythos 5 Benchmarks Explained — Vellum
- Claude Fable 5 compared to other models and benchmarks — Reddit
