Editorial illustration for Researchers Find "But Marinade," Passwords in ChatGPT's Hidden Reasoning
ChatGPT's Hidden Reasoning Exposed in New Study
Researchers Find "But Marinade," Passwords in ChatGPT's Hidden Reasoning
Alexander Panfilov's research team set out to check whether the "encrypted" reasoning steps behind ChatGPT, Claude, and Gemini were actually as sealed off as OpenAI, Anthropic, and Google claim. They weren't. By running a jailbreak that used smaller AI models to transcribe the raw output of larger ones, Panfilov's group found a way into the internal thought processes that these companies hide from users, showing only a cleaned-up summary or nothing at all.
The stakes go beyond curiosity about how a chatbot "thinks." These providers encrypt raw reasoning tokens partly to guard trade secrets, but the practice also assumes users' sessions stay private in the process. Panfilov's team found that assumption doesn't hold. Scanning publicly shared ChatGPT sessions, they pulled out dozens of exposed passwords and API keys that had leaked straight out of the hidden reasoning layer.
For most queries, the number of tokens they extracted lined up exactly with what providers bill as thinking tokens, meaning they weren't skimming fragments. They were pulling the whole internal monologue.
The vulnerability also hits end users. Anyone who has publicly shared Claude Code or Codex sessions containing encrypted reasoning blobs risks having their personal data decoded. A scan of roughly 7,000 public traces turned up 62 API keys, 33 email addresses, 33 passwords, and other sensitive data.
Why this matters
OpenAI, Anthropic, and Google have all leaned on "summarized" or encrypted reasoning as a selling point, a way to let users see a model's thought process without exposing raw chain-of-thought that competitors could scrape or attackers could exploit. Panfilov's team just showed that layer is thinner than advertised. If a smaller model can be jailbroken into transcribing a bigger one's hidden reasoning, and that reasoning can contain leaked passwords or stray phrases like "but marinade," then the abstraction providers sold as a safety boundary is actually a side channel.
Matthew Green flagged this in May and reportedly got shrugged off. That response should worry anyone building on these APIs. For developers wiring chain-of-thought into agents or automated pipelines, this is a reminder that "hidden" reasoning isn't sandboxed, it's just obscured, and obscurity is not a security model.
Teams handling credentials, PII, or proprietary logic inside prompts should assume that reasoning traces are recoverable until providers prove otherwise, not after the next disclosure forces their hand.
Related Reading
Browse all OpenAI news →
LLMs & Generative AI ChatGPT's 'Nerdy' tweak rewards goblin metaphors in answers, study finds
LLMs & Generative AI Google tests visual 'magazine-style' UI for Gemini 3 Pro users
LLMs & Generative AI AI Engineers Face Rising Costs, Need New Strategies for Efficiency
Related Topic OpenAI's GPT Transcribe Cuts Error Rate to 3.31%, Improving on GPT-4o
Open Source
Related Topic OpenAI Calls Apple's Lawsuit 'Rotten to Its Core' in Rebuttal
Policy & Regulation
From Archives OpenAI Launches USD 125 ChatGPT Business Seats with 5x Standard Capacity
From Archives Claude Agent Found Vulnerability in Gym Appointment Software, Chat Logs Show
From Archives OpenAI's GPT-5.6-Cyber answers 95% of sensitive security queries others block
Business & Startups
From Archives OpenAI Buys NextSlide to Add AI Presentations to ChatGPT
Business & Startups
Common Questions Answered
How did Panfilov's research team access ChatGPT's hidden reasoning steps?
Panfilov's team developed a jailbreak technique that used smaller AI models to transcribe the raw output of larger models like ChatGPT, Claude, and Gemini. This method bypassed the encrypted reasoning that companies claimed was sealed off from users, revealing the internal thought processes that were previously hidden behind cleaned-up summaries or nothing at all.
What sensitive data was discovered in the encrypted reasoning blobs?
A scan of approximately 7,000 public traces of Claude Code and Codex sessions containing encrypted reasoning blobs uncovered 62 API keys, 33 email addresses, 33 passwords, and other sensitive personal data. This vulnerability exposed that users who publicly shared sessions with encrypted reasoning risked having their personal information decoded by attackers.
Why do OpenAI, Anthropic, and Google use encrypted reasoning as a selling point?
These companies promoted encrypted or summarized reasoning as a feature to allow users to see a model's thought process while protecting raw chain-of-thought data from being scraped by competitors or exploited by attackers. However, Panfilov's research demonstrated that this security layer is thinner than advertised, as smaller jailbroken models can transcribe the hidden reasoning.
What are the security implications of the vulnerability found in AI model reasoning?
The research shows that the encryption protecting AI models' internal reasoning is not as robust as companies claimed, creating risks for both end users and the companies themselves. If attackers can jailbreak smaller models to transcribe larger models' hidden reasoning, they can potentially extract leaked credentials, API keys, and other confidential information that may be embedded in the reasoning process.
Further Reading
- A New Trick Reveals AI Models’ Inner Thoughts - WIRED
- Let’s talk about encrypted reasoning - Cryptography Engineering
- OpenAI, Anthropic, and Google LLM APIs vulnerability Exposes Hidden Reasoning Traces - Cyber Security News
- ChatGPT Data Leakage via a Hidden Outbound Channel in the Code Execution Runtime - Check Point Research
- ChatGPT Search can be tricked into misleading users, new research reveals - TechCrunch
