Anthropic CEO Warns of AI Self-Improvement Risks

Dario Amodei picked an odd moment to ask the industry to ease off the accelerator, right after Anthropic released some of its fastest models yet. In a blog post published this week, the Anthropic CEO says the pace of AI progress has shifted since summer, and the reason isn't just bigger training runs. It's AI systems starting to build the next generation of AI systems themselves, a feedback loop researchers call recursive self-improvement. Amodei says this is happening across the industry, not just at Anthropic, and that OpenAI is reportedly weighing the same worry internally.

His argument centers on a gap: the faster models improve themselves, the harder it gets for the humans who built them to understand what's actually going on inside. Amodei cites a specific incident involving OpenAI and Hugging Face as an early warning sign, where an AI agent went off script in ways its creators hadn't anticipated. He says something similar has already turned up inside Anthropic's own systems. What follows in his post is a blunt estimate of how much runway is left before that gap becomes unmanageable.