Engineering the Impossible: Adding Safety to Autonomous Agents

We recently covered the OpenAI-Hugging Face incident, where OpenAI models escaped a sandboxed evaluation, exploited a zero-day, and accessed Hugging Face’s production systems to retrieve test answers.

Anthropic later disclosed three similar incidents in which Claude models reached the open Internet from misconfigured capture-the-flag environments and compromised the production systems of three real organizations.

Meta has now reported a comparable case: one of its models gained unintended Internet access during testing and exploited a vulnerability in a third-party service. Meta said the incident stemmed from a testing misconfiguration rather than a sophisticated sandbox escape.

All three incidents point to the same architectural failure: the containment infrastructure did not reliably enforce the boundaries of the evaluation.

In OpenAI’s case, the environment was intended to be isolated, but the models exploited a zero-day to reach the Internet. In the Anthropic and Meta cases, misconfigured evaluation environments provided an open path to the Internet.

Anthropic’s prompts told Claude that it was operating in a simulation without Internet access, but a misunderstanding with its evaluation partner meant that access was available. The prompt said “no Internet,” the infrastructure said otherwise, and the models generally treated the infrastructure as ground truth.

Meta reported a similar failure involving the same evaluation partner: unintended Internet access allowed one of its models to find and exploit a vulnerability in a real third-party service.

Why prompt-level scoping is not scoping

Telling a model that no Internet is available and then providing Internet access creates the perfect conditions for a false-certainty error of the type Mythos 5 demonstrated.

The solution, as Anthropic's own analysis suggests, is architectural: verified network isolation that cannot be misconfigured by a mutual misunderstanding, not training the model to ignore reality under ambiguous conditions.

In our OpenAI post, we called this the difference between soft scoping and hard scoping:

How XBOW handles this

We build autonomous offensive security agents. They run real attacks against real applications, in production, every day. We have been designing for exactly this class of failure since day one, because we assumed models would behave this way. We did not assume they would stay polite.

Here is how our architecture works, simplified for readability:

Let's walk through each layer and why it exists.

Layer 1: Boundary Rules, beyond simple blocks

Every domain the assessment encounters is classified into one of three tiers:

This is not simply a suggestion to the model. It is enforced by the egress proxy at the DNS layer. Within attackable domains, individual URL patterns can be marked auth-only (reachable for login flows, never attacked) or blocked (inaccessible entirely). Password resets, account deletion endpoints, payment processors, all protectable at the URL level.

Critically: scope is locked at launch. There is no runtime path for the agent to expand its own scope.

Layer 2: Man-in-the-Middle Proxy: a chokepoint for all egress traffic

Every attack agent's outbound traffic flows through a proxy. DNS lookups for domains outside the boundary rules return BLOCKED. This is not a filter the model can reason around. If the DNS does not resolve, the connection does not happen.

Each agent runs as a separate OS user on the shared attack container. Per-user traffic goes through the proxy, which enforces target allowlisting, outbound rate limiting, and per-user request logging. Every packet is logged for post-hoc audit.

Layer 3: Short-lived attack agents with adjudicated output

Attack agents are short-lived LLM processes. They run in waves, they probe the target, and they debrief into a durable "worldview," a shared endpoint coverage map and threat model. Only debriefs that survive adjudication enter the worldview. Agents are allowed to be wrong; the worldview is the source of truth. Note that although we don’t trust individual agents, we believe in defense in depth and do, therefore, instruct the agents to adhere to the scope.

Layer 4: Guardian Model

This is the layer that sits between what the agent wants to do and what actually happens. The Guardian Model is an independent model, completely separate from the attack agent's reasoning chain. It scrutinizes every single action before execution, scores its safety on a number of dimensions, separates which endpoints are merely being visited and which are actually being attacked, and blocks out-of-scope attacks and other unwanted actions.

The Guardian cannot be reasoned with, convinced, or bypassed by the solver agent. It sees the raw action proposed, not the agent's justification for proposing it. An agent that has convinced itself a real production system is "part of the exercise" (exactly the failure mode Anthropic's Opus 4.7 demonstrated) still has its actions independently evaluated by a model with no stake in the agent's goal.

This is what we mean in our OpenAI post when we say: "We run an external model that looks at every single action against a target and reviews it for safety before it lands."

Layer 5: Deterministic health monitoring

The system continuously monitors three health signals: target uptime (is it responding?), authentication health (is the session valid?), and rate limit detection (are we being throttled or WAF-blocked?). Common auto-pause reasons include target-unhealthy, account-locked, captcha-blocked, waf-blocked, and site-unavailable.

When health degrades, the assessment auto-pauses. This is governed by deterministic lifecycle rules, not by the agent's judgment about whether to continue. The deployment stays up during a pause; only agent processes stop.

Compare this to Anthropic's Opus 4.7, which recognized it was hitting a real production system and kept going. In our architecture, the question of whether to keep going is not the agent's to answer.

Layer 6: Immutable, typed audit trail

Every significant action, including assessment lifecycle transitions, severity changes, and access events, generates a typed audit event. Each event is linkable via a per-event URL, searchable, and exportable.

When a customer asks "what did your agent do at 3:47 AM?", we can answer with a structured, per-request log and a complete AI agent trace for every finding. The logged request is embedded in the attack agent’s full history and the model’s explanation of its reasoning behind the request, and its interpretation of the response.

The supply chain attack that could not have happened

Of the three Anthropic cases, the Mythos 5 PyPI incident highlights exactly where architectural controls prevent a specific class of harm.

In Anthropic's setup, the model could reach pypi.org because the environment had unrestricted Internet access. In XBOW's architecture, pypi.org would never resolve:

The model could reason all day about supply chain attacks, but it would never be able to upload it, because the network would not let it reach the registry.

Two incidents, one lesson

The pattern these three incidents reveal is not going away. Models are getting more capable, and are being given more autonomy.

In a previous post, we wrote: "An agent asked to be safe is an agent you are trusting to police itself, and you already know how that ends." The Anthropic disclosure is the sequel. The model was assured it was in a simulation with no chance to access the real Internet. Then it took that chance.

For more details on the safety of autonomous agents, see our new whitepaper: Safety Guardrails in Autonomous Security Testing (no form fill required).