Editorial illustration for Sakana AI Model Improves Problem-Solving but Retains Blind Spots
Sakana AI Model Self-Improves but Still Has Blind Spots
Sakana AI Model Improves Problem-Solving but Retains Blind Spots
Sakana AI and UC Berkeley researchers have built a language model that trains itself without a bigger model checking its work. The method, called Multi-Agent Self-Supervision (MASS), splits one model into three jobs: it solves problems, it grades its own answers, and it rewrites the process for the next round of training. No outside teacher model steps in at any point.
The results look strong on paper. Win rates against the original base model climb to nearly 70 percent after just two training cycles, according to the study. That's a notable jump for a system with no external supervision, and it suggests a model can bootstrap its own improvement by acting as both student and examiner.
But the setup has an obvious weak point. If the model's own judgment is flawed in some way, that flaw doesn't get corrected, it gets reinforced. A bad call on what counts as a "good" answer in round one can get copied straight into round two's training data. The question the researchers had to confront is whether a model can actually see past its own limitations when nobody else is watching.
Sakana AI's MASS method lets one language model improve itself by solving tasks, judging results, and refining workflows without a stronger teacher model. The study finds gains across all three roles, but shared blind spots can let errors pass as progress and feed into the next training cycle.
Why this matters
MASS is a useful data point for anyone building self-improving systems: the gains are real, but they're correlated, not independent. A model that trains itself on solution traces and then gets better at evaluating and optimizing too tells us these three roles share underlying capability rather than functioning as separate checks on each other. That's worth sitting with before anyone markets this as a path to autonomous quality control.
If the evaluator improves alongside the solver because they're drawing from the same well, it can't catch mistakes the solver is structurally prone to making. Sakana's own results bear this out: mixed performance on research tasks, blind spots that persist across rounds, win rates climbing against a weak baseline without proof the ceiling moves. For founders pitching "self-correcting AI" and researchers benchmarking agentic workflows, the lesson is to separate "better at beating its past self" from "actually more reliable." Worth watching next: whether later rounds widen those blind spots or just make them harder to detect, and whether AstaBench-style evaluations get more adversarial to test that gap directly.
Related Reading
LLMs & Generative AI ChatGPT's 'Nerdy' tweak rewards goblin metaphors in answers, study finds
LLMs & Generative AI Google to End Free Access to Gemini AI for Non-Subscribers
LLMs & Generative AI AI Engineers Face Rising Costs, Need New Strategies for Efficiency
From Archives Anthropic's Andrej Karpathy Says Public Underestimates AI Capabilities
From Archives Top AI Models Match Average Human in Complex Strategy Game, Trail Experts
Common Questions Answered
What is Multi-Agent Self-Supervision (MASS) and how does it work differently from traditional model training?
MASS is a method developed by Sakana AI and UC Berkeley researchers that enables a single language model to improve itself without requiring a larger teacher model to supervise it. The approach splits the model into three distinct roles: solving problems, grading its own answers, and rewriting the process for the next training cycle, creating a self-contained improvement loop.
What performance improvements does the MASS method achieve compared to the original base model?
The MASS method demonstrates strong results, with win rates against the original base model climbing to nearly 70 percent after just two training cycles. These gains are observed across all three roles that the model performs during the self-improvement process.
What are the blind spots or limitations of the MASS self-improving approach?
While MASS shows real performance gains, the three roles in the system share underlying capabilities rather than functioning as independent checks on each other, creating correlated blind spots. This means errors and flawed reasoning can pass through the evaluation process and feed into the next training cycle, potentially reinforcing mistakes rather than catching them.
Why is it important to understand the correlation between MASS's three roles before deploying it as an autonomous quality control system?
Since the solver, evaluator, and optimizer roles all improve together from shared underlying capabilities rather than independently, the system cannot reliably serve as an autonomous quality control mechanism. Understanding this correlation is crucial before marketing MASS as a path to autonomous quality control, as it reveals the system's inherent limitations in catching its own errors.
Further Reading
- MASS: Multi-Agent Self-Supervision - arXiv
- Sakana AI's MASS Trains One Model to Run Its Own Agent Team - AlphaSignal
- Sakana's MASS Recycles One 27B Model as Solver, Judge, Optimizer - AI Weekly
- Sakana AI Publishes MASS, a 27B Recursive Self-Improvement Framework Hitting 1.2-1.6x Base Score per Output Token - AI Weekly
- Sakana AI trains a model on its agents' collective research work - RuntimeWire
