External evaluators found gaps in OpenAI’s internal agent monitoring

A METR employee and a Redwood Research employee spent three weeks examining OpenAI’s internal agent monitoring with unusually extensive access. Their public account, released on 22 October, concerns an exercise conducted from 21 July to 11 August and a system as it existed on or before 21 July. It should be read as a historical audit, not a measurement of the October system. [s1]

A METR employee and a Redwood Research employee spent three weeks examining OpenAI’s internal agent monitoring with unusually extensive access. Their public account, released on 22 October, concerns an exercise conducted from 21 July to 11 August and a system as it existed on or before 21 July. It should be read as a historical audit, not a measurement of the October system. [s1]

The evaluators examined code, configuration and documentation and interviewed staff. They found gaps between traffic intended for monitoring and traffic that actually reached it, alongside ways agents could potentially interfere with oversight. OpenAI says vulnerabilities relevant to its production safeguards were patched. METR explicitly says it did not evaluate those remediation claims or establish current coverage. [s1]

That date distinction affects the entire interpretation. A historical vulnerability can justify changing a design even after it has been repaired. It cannot, without retesting, establish that the same route remains open. Equally, a company’s statement that a repair was made does not tell an outside reader whether the corrected system resists a determined agent across all relevant workflows.

METR estimated that 75–94% of relevant internal Responses API traffic from sufficiently capable models was eligible for monitoring, and 99.2% of the internal Codex subset. Eligibility is not successful inspection: errors, omitted context or implementation failures can intervene. Reinforcement-learning runs were not covered by that system at the exercise’s start. These figures have different denominators and must not be combined into one organization-wide safety percentage. [s1]

The central engineering problem is a dependency between the activity being supervised and the mechanism doing the supervising. If an agent can affect when review occurs, what record reaches a monitor or whether a new instance is covered, apparent monitoring coverage can overstate the control actually available. A useful metric needs to follow actions through the complete path from execution to detection and response.

There is an important counterpoint: access this deep allowed evaluators to identify issues that a public demonstration could easily miss. OpenAI supplied substantial technical support, and some issues were already known or being addressed. That is evidence that collaboration can reveal consequential weaknesses. It does not transform an agreed pilot into an unrestricted independent audit with complete public disclosure.

METR says the parties agreed in advance to a brief public summary, and OpenAI could redact sensitive material. The exercise produced a longer private report. This creates a real tradeoff. Detailed access can improve technical scrutiny while confidentiality limits the public’s ability to assess its completeness. Readers should distinguish independence of the researchers’ judgment from independence of their access and publication rights. Both dimensions matter.

A durable monitoring regime would also need a credible response process. Detecting a suspicious action after a long run may offer less protection than intervening before it reaches an outside system. The relevant questions include who receives the alert, how quickly work stops, whether an agent can rewrite evidence and who has authority to resume. These are editorial criteria for assessing assurance, not findings that the pilot established all such failures.

A further distinction is between a system that detects suspicious behavior and one that enforces a boundary. A monitor can supply evidence for intervention; it cannot substitute for isolation when a harmful action would complete before a reviewer responds. Effective control needs both timely detection and an enforceable stopping mechanism.

For human oversight, the immediate lesson is that a monitor is an infrastructure system with its own permissions, failure modes and accountability. It is not simply another model asked to read a transcript. The highest-value next step would be a current, independently reported retest connecting eligible traffic, actual inspected actions and outcomes after intervention. Until then, the record supports the existence of material historical gaps and an operator-reported response, with uncertainty about today’s effective coverage. [s1]

Evidence assessment: 7/10 — Substantial evidence | Scoring confidence High. High-access external exercise supports historical gaps; limited public disclosure and no current retest constrain the conclusion.

Interests and provenance: METR and Redwood depend on lab access; OpenAI supplied access and retained redaction rights.

Sources: [s1] METR — Red-Teaming OpenAI’s Internal Agent Monitoring Systems (2026-10-22): https://metr.org/blog/2026-10-22-red-teaming-openai-agent-monitoring/