Skip to main content
OpenAI audit reveals AI safety monitoring gaps: a person's hand hovers over a laptop displaying code and data.

Editorial illustration for Three-week OpenAI audit finds major gaps in AI safety monitoring

OpenAI Audit Exposes Major AI Safety Monitoring Gaps

• 4 min read

Two outside researchers walked into OpenAI's offices on July 21 with access to the company's codebase, internal documents, and about a dozen employees willing to sit for interviews. They stayed three weeks, through August 11, examining the system OpenAI built to catch its own AI models behaving badly during internal use. The researchers came from METR and Redwood Research, two organizations that specialize in testing whether AI safety claims hold up under scrutiny.

What they found wasn't a system quietly humming along in the background doing its job. Large chunks of internal traffic from OpenAI's most advanced models went unmonitored for stretches of time, and the gaps weren't limited to one narrow technical glitch. The audit surfaced problems in how the monitoring system handled certain technical events, how agents could exploit timing quirks to delay oversight, and how determined actors might hide actions or escape the sandboxing meant to contain them.

OpenAI had already caught and fixed at least one of these bugs on its own before the audit wrapped up. But the broader picture the researchers assembled raises harder questions about how much confidence anyone should place in the monitoring infrastructure underpinning frontier AI deployment right now.

The auditors also found ways to hide hundreds of actions from the monitor and to bypass the Codex sandbox. They determined that specially crafted inputs could likely crash the monitor entirely and that model behavior during RL training wasn't being monitored at all when the audit began.

Why this matters

OpenAI fixed the compaction bug on its own, which is the good part of this story. The bad part is that nobody caught it until outside auditors went looking, and the gap existed precisely because the monitoring system assumed agents would hand control back to users in a predictable way. That's the kind of assumption that holds right up until it doesn't.

For developers building on top of Responses API or similar agentic frameworks, the lesson isn't "OpenAI is careless." It's that monitoring systems designed around expected agent behavior will miss the cases where agents, intentionally or not, deviate from that script. A 6 to 25 percent gap in traffic visibility is a wide range, and that imprecision itself is worth sitting with. If an internal team with full codebase access can't pin down the scope of a blind spot more tightly than that, external users of these systems have even less visibility into what's being watched and what isn't.

Audits like this one from METR and Redwood are useful exactly because they force the gap into the open instead of leaving it to be discovered later, by someone with worse intentions than a bug.

Common Questions Answered

What specific vulnerabilities did the METR and Redwood Research audit discover in OpenAI's AI safety monitoring system?

The auditors found multiple critical gaps including the ability to hide hundreds of actions from the monitor, bypass the Codex sandbox, and crash the monitor entirely using specially crafted inputs. Additionally, they discovered that model behavior during reinforcement learning (RL) training wasn't being monitored at all when the audit began, representing a significant blind spot in OpenAI's safety oversight.

How long did the on-site audit at OpenAI take and which organizations conducted it?

The audit lasted three weeks, running from July 21 through August 11, and was conducted by researchers from METR and Redwood Research. These two organizations specialize in testing whether AI safety claims hold up under scrutiny and had access to OpenAI's codebase, internal documents, and interviews with about a dozen employees.

What was the root cause of the compaction bug that the audit uncovered?

The monitoring system's gap existed because it made a flawed assumption that agents would hand control back to users in a predictable way. This assumption held until the outside auditors discovered the vulnerability, revealing a fundamental design flaw in how the safety monitoring system was architected.

Why is the discovery of these AI safety monitoring gaps significant for developers using OpenAI's Responses API?

The audit demonstrates that even sophisticated companies can have blind spots in their safety monitoring systems, particularly around agentic frameworks and reinforcement learning processes. For developers building on top of the Responses API or similar agentic frameworks, the lesson is that they need to implement their own additional safety checks rather than relying solely on platform-level monitoring.

Did OpenAI address the vulnerabilities found during the audit?

OpenAI fixed the compaction bug on its own, which represents the positive aspect of the story. However, the concerning part is that no internal team caught these vulnerabilities until outside auditors specifically looked for them, highlighting the importance of independent security audits.

LIVE21:19Kimi K3.1 Pro tops OfficeQA Pro v2, closing in on Anthropic's Claude Opus 5.5.