Editorial illustration for Sutton argues LLM scaling can't capture world complexity after training
Sutton: LLM Scaling Hits Limits After Training
Sutton argues LLM scaling can't capture world complexity after training
Richard Sutton wrote the book on why scaling works. Now he's telling the LLM crowd they're scaling the wrong thing.
The Turing Award winner's 2019 essay "The Bitter Lesson" became something like scripture in AI research circles. Its argument: general methods that scale with compute beat clever, hand-engineered approaches every time, given enough processing power. That essay got cited constantly to justify pouring more money and chips into ever-larger language models.
So when Sutton posted again recently, people paid attention. His new argument cuts against the industry that built itself on his old one. Raw compute, he says, isn't enough on its own.
AI research still needs new methods, new thinking, and is nowhere near finished.
This isn't a sudden reversal. Sutton has already called synthetic data a "big mistake" and pushed for agents with world models that learn continuously from real experience rather than static training sets. His post landed hard on X, where researchers have been arguing over what he means for the current scaling-obsessed approach to building AI.
In a recent post, Turing Award winner Richard Sutton says raw compute alone isn't enough for intelligence. AI science needs clever new methods and is "far from finished." His post sparked wide discussion in the AI community on X, a sign that Sutton struck a nerve.
Why this matters
Sutton wrote the essay that half the field now cites to justify bigger training runs, so watching him walk back the simplest reading of his own argument is worth our attention. His point isn't anti-scaling for its own sake: it's that compute without continuous learning is a dead end, since a model frozen after training can't absorb what text alone never captured in the first place. For founders betting product roadmaps on "just wait for the next model," that's a real warning.
For researchers, it's permission to work on world models and lifelong learning without feeling like heretics against the bitter lesson's gospel of scale. We'd note the irony that Sutton is now the one calling for new methods rather than more FLOPs, which is exactly the move he once criticized researchers for resisting. His "big mistake" comment on synthetic data adds to a pattern: the people who built the scaling orthodoxy are now flagging its ceiling.
Worth tracking who in the labs, not just academia, starts building toward continuous learning instead of bigger pretraining runs.
Related Reading
LLMs & Generative AI ChatGPT's 'Nerdy' tweak rewards goblin metaphors in answers, study finds
LLMs & Generative AI Google to End Free Access to Gemini AI for Non-Subscribers
LLMs & Generative AI AI Engineers Face Rising Costs, Need New Strategies for Efficiency
From Archives Open AI models keep shrinking cyber capability gap with closed systems
From Archives Timestral, t0, Tabby, and Morpheus-2.0 Compared on GIFT-Eval
Common Questions Answered
What is Richard Sutton's main argument against current LLM scaling approaches?
Sutton argues that raw compute alone is insufficient for capturing world complexity after training, and that scaling without continuous learning is a dead end. He contends that a model frozen after training cannot absorb information that text alone never captured in the first place, making compute-only approaches fundamentally limited.
How does Sutton's recent position differ from his 2019 essay 'The Bitter Lesson'?
In 'The Bitter Lesson,' Sutton argued that general methods scaling with compute beat hand-engineered approaches, which became widely cited to justify larger language models. However, his recent post suggests he is walking back the simplest reading of that argument, clarifying that his position is not anti-scaling itself but rather against scaling without continuous learning mechanisms.
Why is Sutton's recent critique significant for AI product founders?
Sutton's argument serves as a real warning for founders betting product roadmaps on the assumption that 'just waiting for the next model' will solve their problems. His position suggests that without addressing continuous learning and real-world complexity capture, simply scaling compute will not deliver the intelligence improvements many are expecting.
What does Sutton identify as necessary beyond raw compute for achieving true intelligence?
According to Sutton, AI science needs clever new methods beyond raw compute and is 'far from finished' in developing them. He emphasizes that continuous learning capabilities are essential, as frozen models trained only on text cannot capture the full complexity of the world that intelligence requires.
Further Reading
- A Turing Award winner says the industry’s fix for running out of data is ‘a big mistake’ - The Next Web
- KI-Pioneer Sutton calls synthetic data a "big mistake" in the face of an infinitely complex world - The Decoder
- Why the textbook author thinks LLMs are a wrong turn - Founder Files
- Rich Sutton’s “The Bitter Lesson” (2019): Thesis, Reception, and Counter-Arguments Through 2026 - Ashita Orbis
- On Dwarkesh Patel's Podcast With Richard Sutton - LessWrong
