Editorial illustration for Longer Reasoning Paths Increase Per-Question Position Bias in QA Models
Longer Reasoning Paths Increase Per-Question Position...
We train models to reason, hoping they’ll become less stupid. Instead, they often become more predictable. New research confirms a quiet suspicion: the harder an AI works on a multiple-choice question, the more likely it is to just pick the answer in slot A or slot D, regardless of what it says.
A study across thirteen different reasoning setups, from small 7-billion-parameter models to the massive DeepSeek-R1, found a clear and troubling pattern. Longer chains of thought, those step-by-step justifications we prize, don’t mitigate bias. They bake it in deeper.
For most configurations, the more tokens a model generates to explain itself, the stronger its preference for a specific answer position becomes. This isn't about getting more answers wrong. It's about the model's justification becoming a performance, a long-winded excuse for a simple, biased guess.
We test this on position bias in multiple-choice QA and find a different story: within any reasoning-capable model, per-question position bias scales with the length of the reasoning trajectory.Across thirteen reasoning-mode configurations (two R1-distilled 7-8B models, two base models prompted with CoT, and DeepSeek-R1 at 671B) on MMLU, ARC-Challenge, and GPQA, twelve show a positive partial correlation between trajectory length and Position Bias Score (PBS) after controlling for accuracy, ranging from 0.11 to 0.41 (all p < 0.05). All twelve open-weight reasoning-mode configurations show monotonically increasing PBS across length quartiles.
The correlation is persistent. It shows up across different tests and model architectures. The implication is blunt. Encouraging verbose reasoning, a common tactic to boost apparent transparency and accuracy, actively makes models more susceptible to a trivial formatting trick.
This flips a standard assumption. A long, correct reasoning path is not inherently better than a short one. It might be worse, a sign the model is rationalizing a positional bias instead of solving the problem. Benchmarks that reward lengthy chain-of-thought outputs could be selecting for the most stubbornly biased models.
The fix isn't to stop asking models to reason. It's to start measuring what that reasoning actually anchors to. Evaluation needs a new mandatory check: track bias against reasoning length.
If a model's confidence grows with its word count but stays glued to answer slot B, that's a failure. Real intelligence isn't just thinking longer. It's noticing where you're standing.
Related Reading
Policy & Regulation After ditching AI fitness apps and a Fitbit, I return to Peloton classes
Policy & Regulation Grammarly faces class-action suit over AI ‘Expert Review’ feature
Policy & Regulation Akamai Proposes SAFE Cybersecurity Guidelines for AI Threats
From Archives Atlantic report: MorphCast AI tags employee emotions during boss meetings
From Archives Musk lawsuit scrutinizes OpenAI safety as tech gains power under scrutiny
Common Questions Answered
What is position bias in QA models and how does it relate to reasoning chain length?
Position bias refers to AI models' tendency to select answers based on their slot position (like A or D) rather than content accuracy. The research demonstrates that longer chains of thought actually increase this bias, making models more predictable in their answer selection regardless of reasoning quality.
Which AI models were tested in this study on reasoning paths and position bias?
The study examined thirteen different reasoning setups ranging from small 7-billion-parameter models up to the massive DeepSeek-R1. This wide range of model sizes and architectures allowed researchers to confirm that the position bias pattern persists consistently across different AI systems.
Why does encouraging verbose reasoning in AI models potentially backfire?
While verbose reasoning is commonly used to boost transparency and apparent accuracy, this research shows it actively makes models more susceptible to positional formatting tricks. Long, correct reasoning paths may actually indicate the model is rationalizing a positional bias rather than genuinely solving the problem.
What does this research suggest about the relationship between reasoning length and model performance?
The study challenges the standard assumption that longer reasoning paths are inherently better than shorter ones. Instead, extended chains of thought may actually be worse, suggesting they correlate with models relying on trivial formatting cues rather than true problem-solving capabilities.
Further Reading
- Positional Bias in Binary Question Answering: How Uncertainty Amplifies Order Effects in LLMs — arXiv
- Towards Evaluating Fake Reasoning Bias in Language Models — arXiv
- Position is Power: System Prompts as a Mechanism of Bias in Large Language Models — ACM
- The State of Reinforcement Learning for LLM Reasoning — Sebastian Raschka's Newsletter
