Editorial illustration for Positional copying dominates answer readout in 1‑3B LMs on GSM8K
Positional copying dominates answer readout in 1‑3B LMs...
We’ve been told that chain-of-thought makes small language models reason. It doesn’t. It just tells them where to look.
New work on models between one and three billion parameters shows they solve math problems with a cheap trick. Confronted with a word problem, these instruction-tuned models don't read their own internal reasoning. They copy the last number that appears right before the final answer box.
That’s it. Isolating the answer-readout stage reveals this positional shortcut is nearly everything. The presence of the correct number in that slot accounts for 54 to 92 percentage points of a model’s accuracy.
That’s 89 to 92 percent of its maximum possible score under teacher forcing.
In three 1-3B instruction-tuned LMs on GSM8K, we isolate the answer-readout stage via prefix completion and identify a positional shortcut: the model copies whichever number occupies the trailing position before the answer delimiter, regardless of intermediate reasoning. Gold-answer presence accounts for 54-92 pp of accuracy (89-92% of each model's teacher-forcing ceiling); even on incorrect items, the final answer matches the last CoT number 95-96% of the time. The copy channel takes precedence over retained-context completion: replacing the trailing number with a wrong value collapses accuracy to near-zero despite correct intermediates, yet removing it recovers 5-32 pp above that floor--even single-step arithmetic the model can otherwise perform is suppressed when a copyable number is present.
Even when the model gets the problem wrong, its answer matches the last number in its own chain of thought 95 to 96 percent of the time. If you swap that trailing number for a wrong one, accuracy drops to near zero. The preceding correct reasoning is ignored.
Remove the number entirely, and performance bounces back by five to 32 points. The effect is total. It suppresses the model's ability to perform simple, single-step arithmetic if a copyable digit is sitting there.
These models haven't learned math. They've learned a brittle output format. They read position, not logic.
Until that changes, what looks like reasoning is just a shell game. And the pea is always under the last shell.
Related Reading
LLMs & Generative AI ChatGPT's 'Nerdy' tweak rewards goblin metaphors in answers, study finds
LLMs & Generative AI Google tests visual 'magazine-style' UI for Gemini 3 Pro users
LLMs & Generative AI AI Engineers Face Rising Costs, Need New Strategies for Efficiency
From Archives StepFun launches StepAudio 2.5 Realtime, evaluated via mobile app raters
From Archives Anthropic may keep supplying Claude to NSA despite Pentagon risk flag
Common Questions Answered
What is positional copying and how do 1-3B language models use it to solve GSM8K problems?
Positional copying is a shortcut where small language models (1-3 billion parameters) solve math problems by simply copying the last number that appears before the answer box, rather than performing actual reasoning. This mechanism allows these instruction-tuned models to achieve correct answers without genuinely understanding or executing the mathematical steps in their chain-of-thought reasoning.
How does chain-of-thought reasoning actually function in small language models according to this research?
Contrary to common assumptions, chain-of-thought doesn't enable small language models to reason through problems; instead, it serves as a positioning mechanism that tells the model where to look for the answer. The models exploit this by locating and copying the final number in their generated reasoning, regardless of whether the preceding logic is correct or meaningful.
What happens to model accuracy when the trailing number in chain-of-thought is removed or replaced?
When the trailing copyable number is removed entirely, model performance actually improves by 5 to 32 percentage points, demonstrating that the number was suppressing the model's ability to perform arithmetic. Conversely, when the trailing number is swapped for an incorrect one, accuracy drops to near zero, showing that models match their answers to this final number 95-96 percent of the time regardless of correctness.
Why does positional copying represent a fundamental limitation in how small language models approach mathematical reasoning?
Positional copying reveals that small language models haven't learned genuine mathematical reasoning capabilities; instead, they've learned a superficial pattern-matching trick that exploits the structure of training data. This shortcut completely suppresses the model's ability to perform simple single-step arithmetic when a copyable digit is available, indicating that the model prioritizes positional cues over actual computational logic.
