Editorial illustration for SFT and RL Reweight Pretrained Distributions via Demonstration and Reward Signals
SFT and RL Reweight Pretrained Distributions via...
Forget the fighting words. Supervised Fine-Tuning and Reinforcement Learning aren't warring factions. They're the same basic math trick, just with different inputs.
Both techniques take a trained language model and adjust the odds. They reshuffle the deck of what the model might say next. SFT does it by showing examples.
RL does it by giving points. The core mechanism is identical: reweighting the original distribution. If the adjustments are subtle, the model doesn't learn anything genuinely new.
It just gets better at accessing what was already tucked inside its parameters. This changes the debate.
SFT and RL can both be seen as reweighting a pretrained reference distribution, only with different external signals. Demonstration signals define low-energy behavior for SFT, and reward signals define low-energy behavior for RL. When the update remains close to the base model, the main effect is local reweighting, not capability creation. Within this framework, the central question is no longer whether post-training is framed as SFT or RL, but whether it reweights behaviors already within reach, or instead expands the model's reachable behavioral space through search, interaction, tool use, or the incorporation of new information.
The algorithms are a distraction. The real split is between tweaking probabilities and building new roads.
Strict post-training, whether SFT or RL, mostly performs elicitation. It coaxes out latent behaviors. It makes the model's existing good answers more probable.
This is useful polishing. It is not invention. Actual capability creation looks different.
It requires mechanisms that let the model escape its initial programming. Search. Interactive loops with tools or environments.
Incorporating fresh data mid-process. These actions don't just reweight the old distribution. They define a new one.
We've been arguing about which polishing method is best. A more urgent question is whether we're content with polishing at all, or if we want models that can genuinely build something new.
Related Reading
LLMs & Generative AI ChatGPT's 'Nerdy' tweak rewards goblin metaphors in answers, study finds
LLMs & Generative AI Google tests visual 'magazine-style' UI for Gemini 3 Pro users
LLMs & Generative AI AI Engineers Face Rising Costs, Need New Strategies for Efficiency
From Archives Spatial priming beats semantic prompting in chart data extraction study
From Archives GraphDC Uses Divide‑and‑Conquer Agents to Scale Graph Reasoning
Common Questions Answered
How are Supervised Fine-Tuning and Reinforcement Learning mathematically similar?
Both SFT and RL use the same core mechanism of reweighting the original pretrained distribution of a language model. SFT adjusts probabilities by showing examples, while RL does so by assigning reward signals, but the underlying mathematical approach to reshuffling what the model might say next is identical.
What is the difference between elicitation and capability creation in post-training?
Elicitation through SFT or RL tweaks probabilities to make existing good answers more probable without creating new capabilities. True capability creation requires different mechanisms like search, interactive loops with tools or environments, and methods that allow the model to escape its initial programming constraints.
Why does the article argue that the SFT versus RL debate is a distraction?
The article contends that the real distinction in post-training is not between SFT and RL techniques, but rather between approaches that merely adjust probabilities through elicitation versus those that build fundamentally new capabilities. Both SFT and RL primarily perform elicitation, making them more similar to each other than different.
What happens when SFT or RL adjustments to a language model are subtle?
When the adjustments made through SFT or RL are subtle, the model doesn't learn anything genuinely new. The subtle reweighting of the pretrained distribution primarily serves as useful polishing rather than creating actual new capabilities or behaviors.
Further Reading
- On the Generalization of SFT: A Reinforcement Learning Perspective — arXiv
- Self-Rewarding PPO: Aligning Large Language Models with Demonstrations Only — OpenReview (COLM 2025)
- Getting More Juice Out of the SFT Data: Reward Learning from Human Demonstration Improves SFT for LLM — NeurIPS 2024
- Dense Reward Recovery: Treating SFT as Inverse Q-Learning for Post-SFT RL — Emergent Mind
