Editorial illustration for AI Companies Using AI to Build Models Sparks Fears of 'Recursive' Loop
AI Training on AI Data Triggers 'Recursive Loop' Fears
AI Companies Using AI to Build Models Sparks Fears of 'Recursive' Loop
Raymond Douglas doesn't have a clean answer, and he says that's the problem. The University of Toronto researcher co-authored a new report called "Pacing the Frontier, A Research Agenda," which lays out just how unsettled the question of slowing AI development actually is. Ideas have piled up for years: government regulation, better ways to measure progress, tracking devices embedded in GPUs, even ceremonial destruction of AI chips.
Some are mundane policy fixes. Others sound like something out of a heist film. None of them has been tested at scale, and nobody agrees on which would actually work.
The stakes for figuring this out keep climbing. An Anthropic researcher recently quit the company and warned that AI could be on a path to wiping out humanity within a couple of years. The head of Anthropic's safety lab backed him up almost immediately.
Meanwhile, the people building the most powerful systems on the planet, including Dario Amodei, Sam Altman, Elon Musk, and Demis Hassabis, keep talking about risk in increasingly stark terms, even as their labs race forward. Douglas argues that pace itself needs to become a subject of serious study, not an afterthought.
The issue seems especially pressing because AI companies are now using AI itself to build ever-more powerful models. This has sparked fears of an accelerating recursive self-improvement (RSI) loop that would see AI outstrip humans’ ability to comprehend what it is up to within a few years.
Why this matters
We keep hearing that AI could get dangerous, and now the same companies building it are feeding AI's output back into training the next generation. That's the part worth sitting with. OpenAI, Anthropic, and Google DeepMind all use model-generated data and model-assisted research to speed up their own work, which means the feedback loop researchers warn about isn't hypothetical anymore, it's a live design choice being made inside labs that also happen to be racing each other on release timelines.
For developers and founders building on top of these systems, that's a signal to stop assuming capability jumps will be gradual or well-documented. If the labs themselves aren't sure how to keep the loop legible, downstream builders should expect surprises in model behavior between versions, not just improvements. For researchers, the interesting question isn't whether RSI is real yet, it's whether any of the proposed brakes, government rules, better interpretability tools, or internal audits, can move as fast as the loop they're meant to slow.
Related Reading
Policy & Regulation Akamai Proposes SAFE Cybersecurity Guidelines for AI Threats
Policy & Regulation After ditching AI fitness apps and a Fitbit, I return to Peloton classes
Policy & Regulation Sony Music, Warner Chappell Sue Anthropic Over "Blatant Theft
From Archives OpenAI Targets Legal Market With Astra's 230 Million Case Law Index
From Archives AI Models Solve Millennium Math Problems, Experts Warn of Urgent Risks
Common Questions Answered
What is the recursive self-improvement (RSI) loop that researchers are concerned about in AI development?
The RSI loop refers to AI companies using AI itself to build more powerful models, creating a feedback cycle where AI-generated outputs are fed back into training the next generation of models. Researchers fear this accelerating loop could cause AI to advance so rapidly that it outstrips humans' ability to comprehend and control what it is doing within just a few years.
Which major AI companies are currently using AI to assist in building their next-generation models?
OpenAI, Anthropic, and Google DeepMind all use model-generated data and model-assisted research to speed up their own development work. This means these leading AI labs are actively implementing the recursive feedback loop that researchers have warned about, making it a live design choice rather than a hypothetical concern.
What solutions does the 'Pacing the Frontier' report propose for slowing AI development?
The report co-authored by University of Toronto researcher Raymond Douglas outlines multiple approaches including government regulation, better ways to measure AI progress, tracking devices embedded in GPUs, and even ceremonial destruction of AI chips. However, Douglas acknowledges that there is no clean answer to the question of how to actually slow AI development, indicating the complexity of implementing these solutions.
Why is the use of AI to build AI models considered especially pressing compared to other AI safety concerns?
The issue is particularly urgent because it creates a self-reinforcing acceleration cycle that could spiral beyond human comprehension and control. Unlike other AI risks that remain theoretical, this recursive loop is already being actively implemented by major AI companies racing against each other, making it an immediate rather than future concern.
Further Reading
- OpenAI lays out its progress toward recursive self-improvement even as its chief scientist warns of the risks - Fortune
- Big AI sets out its terms for regulatory capture and calls it 'Pace the Frontier' - The Register
- Pacing the Frontier - Pacing the Frontier
- Pacing the Frontier: Security Governance When Labs Ask to Slow Down - Cloud Security Alliance
- Recursive self-improvement explained: Is AI building AI the path to AGI? - Business Standard
