The OpenAI and Hugging Face Incident: When the Model Hacks the Test

This week, OpenAI and Hugging Face disclosed a remarkable incident. During an internal benchmark of cyber capabilities, run without the production safeguards that normally hold models back, a set of pre-release models did not just try to solve the challenge, they invented creative ways to break out of the test to win it.

The sandbox was isolated, with network access limited to an internal package proxy, so the models had to spend real compute finding a way out. They found and exploited a zero-day in that proxy, escalated privileges, and moved laterally until they reached a machine with Internet access. They then inferred that Hugging Face was hosting the evaluation assets, chained stolen credentials with more zero-days into remote code execution on Hugging Face's production servers, and pulled the test answers straight from the production database.

Read that again. The model escaped the sandbox and hacked a real company, to succeed in a test. OpenAI's own article states that the models were "hyperfocused ... going to extreme lengths to achieve a rather narrow testing goal."

I want to say first that OpenAI and Hugging Face deserve credit for talking about this openly. Transparency is how the whole industry gets to build safely, and I agree with Clem Delangue that this does not get solved by any one company working in secret. But I also want to share our perspective on incidents like this at XBOW. These attacks and consequent incidents are not rare anymore.

Models breaking out? We call that Tuesday!

We have been watching this behavior since the early days of GPT-3.5. Give a capable model an objective and enough room to act, and it will do whatever the objective rewards, not whatever you quietly hoped it would do. With the high-reasoning models of the last two years, the pursuits got sharper, more creative, and more patient. And in just the last couple of months, we have measured the aggression going up again with every new model release.

If your instinct is to frame this behavior as "the model went rogue," I would gently push back. The model did exactly what an objective-obsessed optimizer does. The problem isn’t the model. It’s running it without a system around it.

Why we build agents that attack and validators that confirm

From the beginning, we made a design decision that looks small and turns out to be everything. XBOW is not one model pointed at a target. It is agents that attack, and validators that confirm the vulnerability independently.

With that split, you get to define the success criteria. The agent stays flexible in how it attacks, which is exactly what you want from an attacker, but you control the finish line, and you can make that finish line harmless by design rather than aggressive by default.

The clearest example is SQL injection. The naive success criterion is "return the table with all the usernames." That works, and it also means your testing tool just exfiltrated real customer data to prove a point. We don't do that. We build the agent and validator around a timing attack instead. We prove the injection is real through response timing, not by pulling the data out. This approach yields the same confidence, without the leak.

There is a second, quieter benefit to the agent and validator working together. You stop thinking in specific CWE checklist items and start thinking in vulnerability primitives. That makes the attacker smarter and more focused on the objective it actually needs to reach, instead of walking through a static checklist.

Our own Hugging Face moment, caught in the lab

We built an impact assessment capability seven months ago, a configurable step for understanding what an attacker could really do after a finding. We then had our own version of this story. Impact assessment is our way of saying "don't stop at the vulnerability, tell me what someone could actually do with it."

On the first iteration in our labs, we forgot to turn safety on. The agent found an RCE, dumped credentials, and then ... kept going, exactly like an attacker would. It took those credentials and started checking what they unlocked. It connected to Slack and began mapping users and channels. It reached into AWS to see what was exposed. Nobody told it to stop, so it didn't.

That is the unintended offensive flow that happened at Hugging Face. The difference is where it happened. Ours happened inside a lab, behind controls we then finished wiring up, because we had already assumed this is how models behave. We did not assume they would stay polite and stay within the preset boundaries.

Safety is not a prompt

An agent asked to be safe is an agent you are trusting to police itself, and you already know how that ends. While “trust but control and verify” is a founding principle for every security practice, we find that these standards tend to be relaxed for newer technologies. This was the case for cloud 10 years ago, and it is the case for agents today.

This is why, from day one, we built XBOW so that safety does not live only inside the agent. We put safety outside the agent too. We run an external model that looks at every single action against a target and reviews it for safety before it lands. We run an external proxy that logs every packet, and, just as importantly, enforces scope, so the system stays inside the domains our customer actually authorized. Soft scoping guides the model, and hard scoping stops it. You need both.

Left unchecked, models will break things

There is a line in OpenAI's writeup that I would put on a wall. These models "can discover and exploit novel attack paths in real-world systems without source-code access." That is not a warning to us, that is XBOW’s entire thesis. It is exactly why we test the way an attacker does, from the outside, and it is exactly why the discipline has to sit around the model, not inside it.

Models are a fantastic tool. They are the most capable attackers we have ever worked with. But if you want to run one against a real enterprise environment, the model is not the product. The system around it is.

You need proper orchestration. You need soft and hard scoping, and you have to treat those controls as part of the attack surface, because a determined optimizer will test the proxy and the sandbox too. You need safety built into the core, not sprinkled on as an instruction. And you need full auditability, so that every action can be reviewed after the fact. Notice that in this incident the safeguards were intentionally switched off for the evaluation, and the moment they were off, the model did what models do.

The Hugging Face incident is not a reason to fear these models, but a reminder of what they are, and of the work required to point them at production responsibly. That work is the whole game. It is the part we have been building for years, precisely because we saw this coming.The model proposes breaking things. The governed runtime decides if it can do it.

Learn more about XBOW safety measures and controls.