Editorial illustration for PhantomFill: When Language Models Invent Answers to Unanswerable Questions
LLMs Invent Answers When Data Missing, Study Finds
PhantomFill: When Language Models Invent Answers to Unanswerable Questions
Ask GPT-5.5 whether a support call had any customer complaints, in plain prose, and it says there's no transcript 98% of the time. Ask the same model to fill out a JSON field marked "sentiment" for that same untranscribed call, and it invents an answer every single time, 40 out of 40 tries. Nothing about the underlying evidence changed. Only the shape of the answer box did.
That's the finding behind PhantomFill, a new benchmark testing thirteen language models on questions built to be unanswerable: a viral post with 12,400 likes and no visible replies, a call center ticket with no transcript. Researchers held the question and the input constant and varied only the output format, free text versus structured fields like required JSON enums and minimum-count arrays. The format alone determined whether a model admitted ignorance or fabricated a customer's mood, a crowd's reaction, a fact nobody recorded.
The results scale in a way that undercuts easy assumptions about bigger models being safer models. Required fields push fabrication to 100% in ten of thirteen systems. Explicit escape hatches like "insufficient evidence" work only for frontier models, not one of nine open-weight models used them. Even direct instructions not to guess get overridden by the schema itself in most cases tested.
In free text, GPT-5.5 answers honestly. It says there is no reply data, 98% of the time. Given a required JSON field for sentiment, the same model invents an answer 40 times out of 40.
Why this matters
The gap between 98% and near-zero honesty isn't a model capability problem, it's a schema problem, and that distinction matters for anyone shipping structured outputs. If your pipeline forces a sentiment field, a confidence score, or a categorical label on every input, you've built a system that rewards confident fabrication over "I don't know," even when the model clearly knows better in free text. That's a design flaw sitting in thousands of production API calls right now, from support ticket routers to social listening dashboards, and it's invisible until someone checks the underlying data.
The fix isn't a bigger model or a better prompt, it's giving the schema an exit: a null field, an "insufficient_data" enum value, a confidence threshold that blocks forced answers. For founders building on top of LLM extraction, this is a reminder to audit your JSON contracts, not just your prompts. For researchers, it's evidence that hallucination benchmarks measured in prose don't transfer to the format most models actually operate in.
The form is doing more damage than the model.
Related Reading
LLMs & Generative AI ChatGPT's 'Nerdy' tweak rewards goblin metaphors in answers, study finds
LLMs & Generative AI Google tests visual 'magazine-style' UI for Gemini 3 Pro users
LLMs & Generative AI AI Engineers Face Rising Costs, Need New Strategies for Efficiency
From Archives ChatGPT Health Expands to All US Users, Adds Medical Record Integration
From Archives Single Tampered ChatGPT Link Spawns Rogue AI Agent in Minutes
Common Questions Answered
What is the PhantomFill benchmark and what does it test?
PhantomFill is a new benchmark that tests thirteen language models on questions designed to be unanswerable, specifically measuring how models behave when forced to provide structured outputs. The benchmark reveals a critical vulnerability where models that honestly admit uncertainty in free text will fabricate answers when required to fill structured fields like JSON.
Why does GPT-5.5 behave differently when answering about customer complaints in free text versus JSON format?
GPT-5.5 admits there is no transcript data 98% of the time when answering in plain prose, but invents a sentiment answer in all 40 test cases when forced to fill a required JSON sentiment field. This dramatic difference occurs not because the model's underlying knowledge changed, but because the output format creates pressure to provide an answer rather than acknowledge missing information.
What is the core design flaw identified in production systems using structured outputs?
The research reveals that systems requiring structured outputs like sentiment fields, confidence scores, or categorical labels create a schema problem that rewards confident fabrication over honest uncertainty. This design flaw incentivizes models to invent plausible answers rather than say "I don't know," even when the models clearly understand the data is unavailable, and this problem currently exists in thousands of production API calls.
How does PhantomFill demonstrate that this is a schema problem rather than a model capability problem?
PhantomFill shows that the same language model (GPT-5.5) can respond honestly about missing data in free text format while simultaneously fabricating answers in structured formats on identical unanswerable questions. This proves the issue stems from how the output format constrains the model's behavior rather than from any deficiency in the model's actual understanding or capabilities.
Further Reading
- Why language models hallucinate - OpenAI
- AI hallucinates because it's trained to fake answers it doesn't know - Science
- The Hallucination Tax of Reinforcement Finetuning - ACL Anthology
- Major research into 'hallucinating' generative models advances reliability of artificial intelligence - University of Oxford
- Gotcha! Don't trick me with unanswerable questions! Self-... - arXiv
