Editorial illustration for AI Agent Told Wrong Answers Were Right: One Model Believed in Order Memory Test
AI Models Ignore Reward Labels in Context Learning
AI Agent Told Wrong Answers Were Right: One Model Believed in Order Memory Test
A new paper argues that AI agents barely care whether their past attempts were labeled right or wrong. Submitted October 8, 2026, "Do LLMs Learn from Rewards in Context?" tested four benchmarks across six models and found that flipping, randomizing, or deleting reward labels changed the models' improvement curves very little. That's a strange claim for anyone who has built an agent that stores past attempts with scores attached, on the assumption that the model reads those scores and repeats whatever worked. If the labels don't matter, a lot of that bookkeeping is wasted effort.
One researcher decided to check the claim directly, using a simple memory task: a model gets a made-up shipping order and has to pick one of four delivery routes, Kiwi, Plum, Fig, or Lime, with a hidden rule determining the correct answer. Rewards were tracked across trials and, for one test condition, swapped so wrong answers were marked right and right answers marked wrong. Three models ran the test: GPT-6 Sol, Claude Haiku 5.5, and Qwen3 4B Instruct. The results split sharply by model, and one of them followed the fake labels almost exactly where it shouldn't have.
GPT-6 Sol used the labels almost perfectly. With true labels it scored 56.7, close to the ceiling of 59.7, and with flipped labels it scored 15.6, far below chance. Falling below chance means the model followed the wrong labels.
Why this matters
The gap between 56.7 and 15.6 points is the whole story here. A single flipped label shouldn't swing accuracy by 41 points if reward labels are as inert as the paper in question suggests, yet for at least one model, they clearly weren't. That's a problem for anyone building agent pipelines that log past attempts with a score and assume the model is learning the right lesson from them.
If one architecture can be talked into treating wrong answers as correct simply by mislabeling them, the memory systems many teams already ship are more fragile than the "reward labels barely matter" framing implies. We'd treat this as a prompt to audit, not dismiss: test your own agents with deliberately corrupted labels before trusting that logged scores are doing what you think. The fact that only one of three models fell for it also matters, it means this isn't universal, but it's not noise either.
Until someone explains why some architectures absorb bad labels and others shrug them off, teams relying on reward-tagged memory should assume their system might be the gullible one.
Related Reading
Research & Benchmarks Microsoft AI Chief: "If It Isn't Safe We Shouldn't Build It
Research & Benchmarks Google's FACTS benchmark shows 70% factuality ceiling across four tests
Research & Benchmarks Nvidia's DLSS 4.5 beta adds 6x Multi Frame Generation for RTX 50 GPUs
From Archives New Infostealer Malware Uses AI to Automatically Target Victims
From Archives Harness² Uses Eight Dormant Components to Improve AI Agent Performance
Common Questions Answered
What did the paper 'Do LLMs Learn from Rewards in Context?' discover about how AI models respond to reward labels?
The paper found that flipping, randomizing, or deleting reward labels had surprisingly little effect on most models' improvement curves, suggesting that AI agents barely care whether their past attempts were labeled as right or wrong. This challenges the common assumption that models carefully read and learn from reward scores attached to past attempts.
How did GPT-6 Sol perform differently when given flipped reward labels compared to true labels?
GPT-6 Sol scored 56.7 with true labels (close to the ceiling of 59.7) but dropped to 15.6 with flipped labels, falling below chance performance. This 41-point swing demonstrates that GPT-6 Sol actually followed the wrong labels, making it one of the few models that showed strong sensitivity to reward label manipulation.
Why is the discrepancy between GPT-6 Sol's performance with true versus flipped labels problematic for AI agent development?
The significant performance gap suggests that at least some model architectures can be manipulated into treating wrong answers as correct simply through mislabeling, which creates a serious problem for anyone building agent pipelines that log past attempts with scores and assume the model is learning the correct lessons from them. This vulnerability could undermine the reliability of reward-based learning systems in production environments.
How many models and benchmarks were tested in the study about LLM learning from rewards?
The research tested six different models across four benchmarks to evaluate how well language models learn from reward labels in context. This comprehensive testing approach helped reveal that most models showed minimal sensitivity to reward label changes, with GPT-6 Sol being a notable exception.
What assumption about AI model behavior does this research challenge?
The research challenges the widespread assumption that AI models carefully read and learn from reward scores attached to past attempts in their context. Instead, the findings suggest that most models are largely indifferent to whether their previous attempts were labeled as correct or incorrect, fundamentally questioning the effectiveness of reward-based learning approaches in agent systems.
Further Reading
- Do LLMs Learn from Rewards in Context? - arXiv
- Reward Is Enough: LLMs Are In-Context Reinforcement Learners - ICLR
- Bootstrapping Language and Numerical Feedback for Reinforcement Learning in LLMs - ICML
- TR-ICRL: Test-Time Rethinking for In-Context Reinforcement Learning - arXiv
- RLTLDR: Self-improvement by Internalizing Self-generated Feedback - arXiv
