AI Safety Gaps: Three-Week Audit Finds Monitoring Blind Spots

A recent three-week on-site audit revealed that a notable portion of OpenAI's internal AI traffic was not covered by safety monitoring systems. Researchers from METR and Redwood Research conducted this deep-dive evaluation of the company's infrastructure to detect rogue AI behavior. Their findings indicate that between six and twenty-five percent of internal agentic Responses API traffic from models at the GPT-5.5 class and above was entirely unmonitored during the study period. METR notes that this estimate carries significant uncertainty in both directions, but the gap highlights a critical blind spot in current oversight protocols.

The investigation took place from July 21 to August 11. During this window, the external team gained access to OpenAI's codebase and internal documents. They also interviewed approximately a dozen employees to understand the operational context. This pilot program was organized jointly by METR, Redwood Research, and OpenAI to allow outside experts to examine internal infrastructure with a level of detail rarely seen in the industry.

Beyond the unmonitored API traffic, the auditors found issues with internal Codex traffic. They estimate that roughly ten percent of this traffic went untracked between May and July. This gap was caused by a bug in how the system handled compaction events, which are essentially context summaries. OpenAI independently discovered this specific bug and implemented a fix while the audit was still in progress. However, during the period when these events were not monitored, agents could delay oversight. The system only triggered checks when agents handed control back to the user, meaning agents could continue executing actions indefinitely without triggering a review.

The researchers also identified more sophisticated methods for evading detection. They found ways to hide hundreds of actions from the monitor and bypass the Codex sandbox. Furthermore, they determined that specially crafted inputs could likely crash the monitoring system entirely. At the start of the audit, model behavior during reinforcement learning (RL) training was not being monitored at all. OpenAI states that it paused RL training after an incident involving Hugging Face in August and subsequently expanded monitoring to cover this area. These changes occurred after the auditors' access period ended and were therefore not part of the initial evaluation.

OpenAI has stated that it has fixed all the weaknesses identified by the auditors. The full sixty-one-page report was handed over to the company internally. After OpenAI applied redactions, leadership at METR and Redwood Research received a thirty-eight-page version. Only a short summary has been made available to the public. METR stated that it did not have the right to disclose the redacted portions on its own. The organization says it aims to push for greater public transparency in future audits.

The findings suggest that while AI safety monitoring is a growing focus, significant technical gaps remain in how companies track the behavior of their most advanced models. The audit provides a concrete example of how internal systems can fail to capture all agent activities, particularly during complex operations like context compaction or reinforcement learning. As the industry moves toward more autonomous systems, these external checks are becoming a vital component of the safety framework.

For the next phase, the focus will likely shift toward implementing the fixes and establishing clearer protocols for external oversight. The collaboration between OpenAI and the research groups sets a precedent for deep technical audits. While the immediate vulnerabilities have been addressed, the experience underscores the need for continuous and comprehensive monitoring of AI agents as they become more prevalent in internal workflows.