Editorial illustration for Shenfeld et al. compare SFT and SDFT on three tasks for continual learning
Shenfeld et al. compare SFT and SDFT on three tasks for...
Shenfeld et al. posted their paper on self-distillation and continual learning in late January 2026. Nine months later, several dozen follow-up papers have piled in, some extending the idea, some picking at its weak spots. The framework at the center of it all is called Self-Distillation Fine-Tuning, or SDFT, and it belongs to a broader category researchers are calling on-policy self-distillation, OPSD for short.
The pitch is straightforward: fine-tune a foundation model on new tasks without wrecking what it already knows how to do. That tradeoff, known as catastrophic forgetting, has dogged continual learning for years. Standard supervised fine-tuning tends to overwrite old skills as it bakes in new ones. SDFT claims to sidestep that by leaning on reinforcement learning techniques rather than the usual supervised approach, letting models absorb new information from a stream of data without the usual collateral damage.
This piece walks through why continual learning breaks standard methods, how SDFT and its theoretical backing differ from SFT, what the three-task experiments in Shenfeld et al.'s paper actually show, and where critics think the approach still falls short.
Shenfeld et al. provide empirical evidence from several experiments that SDFT yields excellent post-training outcomes while avoiding catastrophic forgetting almost entirely.
Why this matters
Shenfeld et al.'s three-task comparison gives us the first real data point on whether self-distillation can solve the catastrophic forgetting problem that's plagued sequential fine-tuning since the SFT era began. If SDFT genuinely holds per-task accuracy steady while SFT degrades on earlier tasks, that's a concrete engineering win for anyone building models meant to pick up new skills over time rather than being retrained from scratch. We'd flag the timeline, though: OPSD papers only started appearing in late January 2026, so we're talking about a framework with roughly nine months of scrutiny behind it.
That's thin cover for claims about "revolutionizing" post-training. Teams evaluating this for production continual-learning pipelines should look past the framing and focus on the actual per-task curves Shenfeld et al. report, not the surrounding hype.
If the gains hold up across more than three tasks and different model families, SDFT could become a standard tool for incremental skill addition. Until then, treat it as a promising but narrow result, not a settled answer to the forgetting problem.
Related Reading
Research & Benchmarks Microsoft AI Chief: "If It Isn't Safe We Shouldn't Build It
Research & Benchmarks Google's FACTS benchmark shows 70% factuality ceiling across four tests
Research & Benchmarks Nvidia's DLSS 4.5 beta adds 6x Multi Frame Generation for RTX 50 GPUs
From Archives Does Agentic RAG Retrieve Better Evidence? Test on 298 Questions
From Archives Researchers say AI agents learn skills by exploring without a goal
Further Reading
- Self-Distillation as a Performance Recovery Mechanism for LLMs: Counteracting Compression and Catastrophic Forgetting - arXiv
- A Survey of On-Policy Distillation for Large Language Models - arXiv
- Continual Learning via Self-Probe Gradients - arXiv
- Making Expert Reasoning Learnable with Self-Distillation - arXiv
- Continual Distillation of Teachers from Different Domains - Paper Notes
