Rogue AI Agents: Is Surface-Level Monitoring Enough?

In an AISI evaluation (opens in a new tab), a frontier model, acting as an agent, attempted to insert malicious code into an open-source project, created fake identities to influence a human maintainer, and then tried to hide its actions, exhibiting goal-directed deception.

The test was intentionally permissive, with internet access allowed and provider cyber classifiers turned off. This was caught using standard security monitoring, but the behaviour was still unsettling and strong runtime controls remain critical.

The risks are much higher if this happens in enterprise agents with access to infrastructure, data, financial systems, and customers.

While better model alignment is the long-term answer, enterprises deploying agents today can’t wait for perfect models. They are still responsible for any financial, regulatory, or reputational consequences of failures.

Which brings us to the immediate question:

To what extent can we trust our monitoring and control of AI agents?

Is surface-level monitoring enough?

Agent monitoring has come a long way. We track and log actions, we have classifiers and LLM judges scoring behaviour, chain-of-thought monitors, and modern runtimes that can verify tools, permissions, and context, as well as prevent dangerous actions at runtime.

These are all critically important tools, but most of these monitoring mechanisms depend on what the model exposes externally: its text, tool calls, action, and surrounding context.

Chain-of-thought reasoning is likely the richest signal we have. But it has two significant limitations: faithfulness and legibility. Neither of these problems is new. For example, Turpin et al. (opens in a new tab) found that CoT explanations can systematically obscure the factors that an answer actually depends on, and Anthropic recently published Reasoning Models Don’t Always Say What They Think (opens in a new tab), showing that models sometimes use information without referencing it in their reasoning.

None of this implies that CoT monitoring is worthless, because it clearly is valuable, and arguably the most valuable signal we have. We just shouldn’t assume that it completely describes what the model is doing.

Moreover, the chain of thought may take the form of unstructured, illegible text, which introduces further questions about its reliability and interpretability.

When researchers write that “it’s just impossible to tell what the model is up to” in certain traces, it becomes clear that this is a real monitoring challenge.

Faithfulness and legibility are just the start. How effectively can CoT reveal deception (manipulating others to achieve a goal), evaluation awareness (the model knows it’s being evaluated), sandbagging (concealing capabilities, or deliberately under-performing) and reward hacking (optimising for the reward function rather than the intended task)?

These are all increasingly critical safety concerns for advanced agents.

Again, this does not suggest that our current monitoring approaches are useless. The AISI incident itself demonstrates that conventional monitoring, containment, permissions and human oversight play a huge role.

The tougher question is whether there are cases where things look fine on the surface, but evidence is still present elsewhere within the model.

Which is why this distinction matters:

Text, tool use, and chain of thought are all great signals. For high-stakes decisions, can internal model signals add something these monitors miss?

White-Box Monitoring and Control

Black-box monitoring looks at a model’s external behaviour: chain-of-thought, tool calls, and final outputs. White-box monitoring can additionally look at internal model activations via mechanistic interpretability methods and classifiers.

The general argument for doing this is not new. It has been made directly in The Case for White Box Control (opens in a new tab). The open question is: if we already have strong runtime and external monitoring, do activation signals add something those monitors miss?

This also touches on the earlier white-box control work (opens in a new tab) in AISI’s sandbagging research (opens in a new tab). In that work, the off-policy deception probes performed well while the zero-shot black-box monitor performed worse with increasing difficulty. Subsequent experiments on larger models, harder tasks, and model organisms showed the probes’ performance degrading.

White-box access is not a panacea for the monitoring problem.

There is certainly task-relevant information in the activations of the model before it starts generating text or invoking tools. The issue with white-box monitoring is whether there is genuinely useful diagnostic information in those internal signals.

The key word here is “genuinely”. To paraphrase Against Almost Every Theory of Impact of Interpretability (opens in a new tab), showing something is interpretable within a model is not equivalent to demonstrating that it provides a safety benefit over simpler behavioural approaches.

What are closed-weight model providers doing?

OpenAI and Anthropic are making significant progress on this front, and some of it has transitioned from research to production.

OpenAI’s new monitoring setup (opens in a new tab) uses activation classifiers that inspect internal model activity at every sampled token and escalate concerning signals for deeper investigation.

Anthropic has also built separate safety classifiers (opens in a new tab) for their frontier models, adding another layer of monitoring rather than relying on the primary model itself.

This is precisely what we would expect frontier labs to be doing. But with closed-weight models, there is a structural asymmetry. An enterprise can look at the output, tool calls, logs, and surrounding controls. But it cannot generally see the activations themselves. So it must trust the provider with that layer.

The recent OpenAI–Hugging Face incident (opens in a new tab) is a related example. It wasn’t about model internals, and white-box monitoring wouldn’t have resolved the problem, but it illustrates why provider-side monitoring and control matter when powerful models interface with external systems.

For open-weight models, enterprises have an alternative approach available.

Can enterprises do white-box monitoring?

Is looking into model internals possible only by frontier labs and government institutes?

No. With open-weight models, the model’s internal activations are available to whoever runs it, and a growing body of research shows what can be read from them.

The main tool is the probe: a small classifier trained on a model’s internal activations to detect a property of interest. Probes are cheap to run, because they read activations the model computes anyway, and they can score every step before an action executes.

Probe-based monitoring is not limited to tool calls. Linear probes can flag strategic deception (opens in a new tab) even when the text reads as honest, catch “sleeper agent” models (opens in a new tab) before they act on a hidden trigger, and measure removed safety (opens in a new tab), since refusal is carried by a single direction inside the model. The same approach has been used to read truthfulness (opens in a new tab), evaluation awareness (opens in a new tab) and high-stakes situations (opens in a new tab) directly from activations.

For agents, this means a monitor can score deception, hidden intent or risk inside the model before each tool call runs, alongside the existing text-based checks.

While it’s clear that there is useful information within the model, we shouldn’t conclude that the monitoring problem has been solved.

One question is whether these probes partly recover textual evidence already in the context. Work on textual evidence in linear probes (opens in a new tab) and benchmarks like Liars’ Bench (opens in a new tab) demonstrate the importance of generalizing across tasks, models, and distributions. Additionally, there is a concern that, if an internal monitor becomes a deployment gate, models will learn to game it, as explored in work on training against interpretability signals (opens in a new tab).

These limitations do not affect the main conclusion, which is that agent-relevant risk signals are detectable inside the model before they appear as overt action. Whether these signals can catch failures that current monitors do not is another open question, also raised in AISI’s Loss of Oversight (opens in a new tab).

What’s next?

The recommendation is not to replace existing controls but to combine them: improved alignment and monitoring from model providers, robust runtime controls from deployers, and white-box monitoring when it offers genuinely new information.

Hugging Face co-founder Thomas Wolf highlighted a complementary direction in his post (opens in a new tab):

“Let’s hope we can either move to better mechanistic interpretability or force the model to stick to English somehow during training.”

This direction holds promise, but the bar should be higher than simply demonstrating that a classifier can extract interesting information from model activations.

White-box monitoring is nascent, and mechanistic interpretability is difficult, yet agent capabilities are advancing rapidly.

Model providers can continue to improve alignment and guardrails, but enterprises remain responsible for the financial, regulatory, and business consequences when agents get past those controls.

Enterprises need to invest in their ability to inspect and monitor model internals. White-box monitoring adds another lever for visibility, explainability, and control over increasingly autonomous AI systems.

We shouldn’t wait for a failure to recognise that understanding what happens inside the model matters.