Editorial illustration for Ignoring Human Beliefs Leads AI to Predict Wrong Actions, Study Finds
AI Models Fail When Ignoring Human Beliefs, Study Shows
Ignoring Human Beliefs Leads AI to Predict Wrong Actions, Study Finds
Sora can render a coffee cup vanishing into a cabinet with perfect physics. What it can't tell you is what the person who owns that cup will do next, because it has no idea they didn't see it move. That gap is the subject of a new paper making the rounds this week, arguing that the world models meant to power autonomous AI agents are missing a piece nobody's bothered to build: a model of what humans actually believe is happening around them.
Systems like Sora, Genie 3, JEPA, and Marble track objects, positions, motion, occlusion. Solid physics, no psychology. The researchers behind the paper say that's a problem for anything meant to work alongside people, whether it's a service robot or a medical assistant, because human behavior runs on hidden mental states, not just visible object positions. Their proposed fix, a framework called Mental World Modeling, published on GitHub, tries to bolt belief, attention, intention, and social norms onto the physical layer that current world models already handle.
If someone's cup gets moved into a cabinet while they're not looking, the scene looks correct to a purely physical world model. But it still predicts the wrong next action. Only a model that also tracks the person's belief about where the cup is can explain what they'll actually do.
Why this matters
The paper's core claim, that a model scoring physical plausibility alone will misjudge what a person actually does next, should worry anyone building agents meant to work alongside humans rather than just around objects. Sora, Genie 3, JEPA, and Marble can render a falling glass or a moving car convincingly, but none of them, per this research, ask whether a person believes the glass is empty or thinks the car has already stopped. That gap matters most for founders selling robots, assistants, or simulators into homes, hospitals, and warehouses, where the expensive failures come from misreading intent, not physics.
The three-way split into physical, mental, and social scoring, followed by a deterministic decision, is a reasonable first architecture, but it's also an admission that current benchmarks never demanded this. We'd want to see how these mental-consistency scores hold up against adversarial or simply distracted human behavior before anyone builds safety cases on top of them. Worth watching: whether Genie 3 or JEPA's next revisions bolt on belief-modeling, or whether this stays a separate research branch.
Related Reading
Research & Benchmarks Google's FACTS benchmark shows 70% factuality ceiling across four tests
Research & Benchmarks Databricks finds multi-step agents beat single-turn RAG by 21% to 38% on STaRK
Research & Benchmarks Nvidia's DLSS 4.5 beta adds 6x Multi Frame Generation for RTX 50 GPUs
From Archives RadixAttention Speeds First Tokens by Reusing Cached KV States
From Archives Deepseek Targets Visual Agents With New Experimental Flash Model
Common Questions Answered
Why do world models like Sora and Genie 3 fail to predict correct human actions despite accurate physics simulations?
These world models lack a representation of human beliefs about the environment, focusing only on physical plausibility. For example, if a cup is moved into a cabinet while someone isn't looking, the model sees correct physics but cannot predict what the person will actually do next because it doesn't track that the person believes the cup is still in its original location.
What is the key difference between physical world models and belief-aware world models according to this research?
Physical world models track only objective reality and can render accurate physics like falling glasses or moving cars, but they ignore what humans subjectively believe is happening. Belief-aware world models would additionally track human perceptions and knowledge states, enabling them to correctly predict human actions based on what people think they know rather than what actually occurred.
Which AI systems are mentioned as examples of world models that currently ignore human beliefs?
The research specifically identifies Sora, Genie 3, JEPA, and Marble as world model systems that track objective physical observations but lack mechanisms to model what humans actually believe is happening around them. This limitation affects their ability to predict human behavior in real-world scenarios.
Why does this research matter for autonomous AI agents designed to work alongside humans?
For AI agents that need to collaborate with or work near humans, understanding human beliefs is critical to predicting correct behavior and ensuring safe interactions. An agent that only models physical reality but ignores what humans believe could make dangerous assumptions about human actions, making belief-modeling essential for human-AI cooperation rather than just object manipulation.
Further Reading
- Assessing the Capacity of LLMs to Induce Belief States via Planning and Action - ICML 2026
- AI prediction increases forgoing guaranteed rewards - arXiv
- Do Role-Playing Agents Practice What They Preach? Belief Consistency in LLM Simulation - alphaXiv
- Applying theory of mind: Can AI understand and predict human behavior? - Allen Institute for AI
- AI Outperforms Humans in Theory of Mind Tests - IEEE Spectrum
