Editorial illustration for Study Questions RLVR Paper Gains: Are Models Just Learning Answer Format?
RLVR Models May Just Learn Answer Format, Not Skills
Study Questions RLVR Paper Gains: Are Models Just Learning Answer Format?
Four 7B models trained with reinforcement learning from verifiable rewards appeared to get smarter at tasks they were never trained on. Train on math problems, and MMLU accuracy climbs too. That cross-domain lift has become one of the field's favorite selling points for RLVR, cited as proof the technique teaches something closer to general reasoning than narrow pattern matching.
A new paper picks at the measurement behind that claim. The gains get tallied by answer extractors, scripts that scan a model's output for a final response, often by checking a specific spot in the text. The researchers behind this study built a separate extractor that hunts for a stated answer anywhere in the output, not just inside the formatting token the original scripts expect, then reran the comparison on four publicly released SimpleRL base-and-RL model pairs at 7B scale, plus a 32B pair trained with a different recipe.
The question is blunt: when a model's score goes up after RLVR, is it giving better answers, or just learning to park the same answer somewhere the grader happens to look first.
Reinforcement learning with verifiable rewards (RLVR) is credited with gains beyond the domain its rewards check, such as higher MMLU accuracy after training on mathematics. Such accuracies are usually read out of the model's text by an answer extractor. A model that learns to write its answer where the extractor looks can therefore score higher without answering better.
Why this matters
This paper is a reminder to check the measuring stick before trusting the result. RLVR's appeal rests on the claim that training on verifiable math rewards produces reasoning gains that transfer to unrelated domains like MMLU. If a chunk of that transfer is just the model learning to format its answer in a way the extractor script recognizes, then some of the field's reported "generalization" is closer to a parsing artifact than a cognitive one.
For researchers, that means re-running benchmark comparisons with format-blind scoring before publishing transfer claims. For developers building on RLVR-trained models, it's a flag to test actual task performance, not just leaderboard deltas that rely on the same public extraction code everyone copies. For founders selling "reasoning improvements" as a feature, this is the kind of result that invites harder questions from customers and investors about what's actually being measured.
The fix here is mundane: score outputs twice, once with the standard extractor and once with a stricter check, and see how much daylight opens up. Expect more papers doing exactly that.
Related Reading
Research & Benchmarks Microsoft AI Chief: "If It Isn't Safe We Shouldn't Build It
Research & Benchmarks Google's FACTS benchmark shows 70% factuality ceiling across four tests
Research & Benchmarks Nvidia's DLSS 4.5 beta adds 6x Multi Frame Generation for RTX 50 GPUs
From Archives Meta Muse and Apple HomeView surface US AI support for future viewpoints
From Archives Anthropic Gives METR Researchers Employee-Level Access to Systems
Common Questions Answered
What does the new paper claim about RLVR cross-domain gains in models like MMLU?
The paper questions whether the reported cross-domain performance improvements from reinforcement learning with verifiable rewards are genuine reasoning gains or artifacts of answer extraction. It suggests that models may be learning to format answers in ways that answer extractors recognize rather than actually improving their reasoning capabilities across domains.
How do answer extractors potentially inflate RLVR performance measurements?
Answer extractors are scripts that scan model outputs to identify and measure answers, but models can learn to write answers in locations or formats that these extractors specifically look for. This means a model could score higher on tasks like MMLU without actually answering questions better, creating a false impression of generalization.
Why is the distinction between parsing artifacts and cognitive gains important for RLVR research?
RLVR's primary appeal is that training on verifiable math rewards produces reasoning improvements that transfer to unrelated domains, suggesting genuine cognitive advancement. If a significant portion of reported transfer gains are actually just formatting artifacts detected by answer extractors rather than true reasoning improvements, then the field's claims about RLVR's generalization capabilities need substantial re-evaluation.
What specific example does the article give of RLVR's claimed cross-domain benefits?
The article describes how four 7B models trained with reinforcement learning from verifiable rewards showed improved MMLU accuracy after being trained on mathematics problems. This cross-domain lift from math training to MMLU performance has been cited as evidence that RLVR teaches general reasoning rather than narrow pattern matching.
Further Reading
- The Invisible Leash? Why RLVR May or May Not Escape Its Origin - arXiv
- Language Models that Think, Chat Better - arXiv
- Med-RLVR: Emerging Medical Reasoning from a 3B Base Model via Reinforcement Learning - arXiv
- Extending RLVR to Open-Ended Tasks via Verifiable Multiple-Choice Reformulation - arXiv
- [Paper Note] Self-Aligned Reward: Towards Effective and Efficient Reasoners - Paper Notes
