Grok 4.5 Is Powerful. The System Around It Makes It Safe.

The notorious Grok AI model has had its fair share of bad press, and has developed a bit of a reputation as an unsafe cowboy. But it’s also powerful, and performed quite well in our initial tests. So we decided to explore its capabilities, and cowboy tendencies, further. We were surprised: It both was, and wasn't, more reckless than other models.

Grok 4.5 frequently reached boundaries that triggered XBOW’s command-safety controls. But once the system intervened, Grok was unusually willing to accept the correction, replan, and continue safely. In fact, compared with several other models, Grok agents were much less likely to persist until XBOW had to stop the agent entirely.

The result is a model that’s both powerful and willing to explore aggressive ideas, but also highly responsive to external control. That makes Grok 4.5 a useful component of an autonomous offensive-security platform, but not necessarily safe on its own.

Like the genie in the lamp, you need to be very careful in what you ask it to do, but do it right, and it will neither disobey nor disappoint.

Grok reaches the guardrail, but doesn’t cross it

We built XBOW’s autonomous agents on the principle that LLMs can have great ideas, but are never to be trusted. So they don't just execute any command proposed by a foundation model.

Before any action is performed, XBOW’s command-safety layer evaluates it against operational policies. If the proposed action approaches or crosses a safety boundary, the system can block it before it reaches the target.

The agent then has the opportunity to develop another plan. If it repeatedly proposes unacceptable actions despite those interventions, XBOW can stop that agent altogether.

Sharing a bit of our terminology: A blocked agent proposed at least one command that XBOW’s safety system would not allow, while a stopped agent continued pursuing unacceptable actions strongly enough that XBOW terminated that line of investigation.

In our analysis, Grok agents encountered safety blocks at a rate comparable to or higher than several other models we evaluated. But we almost never had to outright stop Grok agents.

For instance, Grok might propose an action that XBOW would not permit. Once the safety layer intervened, however, it generally accepted the constraint, developed another plan, and continued operating within the permitted boundaries.

Some other models displayed a different pattern. They might encounter an initial block less often, but a portion of the agents that were blocked were more likely to continue pushing in the same direction.

Having a risky idea is different from resisting control

Going by block frequency alone would make Grok 4.5 look less safe. But that's only half the picture.

Grok appears willing to consider aggressive or boundary-testing actions. It had more reckless ideas. For many general-purpose applications, that tendency might be problematic, but in offensive security, a willingness to explore unconventional behavior can be valuable.

Security testing requires a model to think about what an ordinary user would never attempt. It must consider unexpected inputs, authorization gaps, unusual application states, chained vulnerabilities, and ways of using functionality that its developers did not intend.

A model that never approaches a boundary may also fail to explore the paths where serious vulnerabilities are found. That raises the burden on the surrounding system, which has to recognize an inappropriate action, stop it before execution, and redirect the model to a safe alternative.

We _want_ a devious hacker, as long as that hacker still listens to us. Our results suggest Grok does: It reaches the guardrail, but does not attempt to circumvent it.

Controllable is not the same as inherently safe

One way to describe Grok 4.5 is as a powerful tool that can be operated safely in the right environment.

Without XBOW’s command-safety system, some actions blocked during our assessments might instead have been executed, and while we like to err on the side of caution here, likely at least some of them probably would have been problematic. Grok’s favorable behavior does not come from the foundation model alone, but from the interaction between the model and the independent control system surrounding it.

Evaluating raw models tells you little on its own. One model is capable but hard to control. Another stays conservative and misses complex vulnerabilities. A third is both powerful and correctable, and that only counts where something exists to do the correcting.

For Grok 4.5, our results suggest external controls are both clearly needed as well as effective. They turn the model’s raw capability into something that can be used responsibly.

The offensive model cannot be its own safety authority

The offensive model’s job is to reason creatively, pursue promising attack paths, and overcome obstacles. Asking that same model to serve as the sole judge of its own actions creates a conflict. XBOW, therefore, evaluates proposed commands through a separate guardian model before they are executed.

The process looks like this:

1. An agent using a _model_ proposes an action.
2. The guardian evaluates it against XBOW’s safety policies, using a combination of deterministic and neural reasoning.
3. Commands that cross a boundary are blocked before execution.
4. The constraint is returned to the agent.
5. The agent replans and pursues an acceptable alternative.
6. Agents that repeatedly resist those interventions are stopped.

Grok’s behavior highlights the importance of combining both the blocking and replanning steps.

A safety system that only rejects commands can bring an assessment to a halt. Effective orchestration must help the agent recover from the intervention, retain useful context, and keep working toward the objective without crossing the boundary. Grok 4.5 proved particularly receptive to that process.

It’s not the model that’s safe, it’s the system

As foundation models become more capable, security teams will compare benchmark results, vulnerability solve rates, costs, and latency. Those measurements are important, but don’t establish whether an autonomous system can safely operate against a real application.

Two security products could use the same foundation model and behave very differently in practice. The difference would come from the architecture surrounding it:

• Are proposed actions evaluated before execution?
• Is the safety evaluator independent from the offensive model?
• What happens when a command is blocked?
• Can the agent replan rather than repeatedly attempting the same approach?
• When is an agent stopped?
• Are model behaviors continuously measured in production?
• Can policies be applied consistently across different models and tasks?

Without these controls, greater model capability can create greater operational risk. With them, an aggressive and creative model can become a valuable component of a controlled security system.

Why XBOW uses a model alloy

These findings also reinforce why XBOW does not rely on a single foundation model.

Grok 4.5 is an effective exploit crafter. That does not mean it is the best model for every task, every budget, or every stage of an assessment.

Models differ in reasoning style, creativity, persistence, speed, cost, judgment, and responsiveness to correction. One might identify an exploit path another overlooks; another might be better at validating a hypothesis, interpreting source code, or completing a high-volume task economically.

A model’s strengths may also come with tradeoffs. The same willingness to explore that makes Grok useful for offensive reasoning increases the importance of strong safety supervision.

XBOW’s model-alloy approach allows the platform to benefit from these differences rather than depending on a single provider or model family.

But the alloy is more than a collection of models. It includes the orchestration that determines how those models are used:

• Routing tasks to the appropriate model
• Coordinating agents across an assessment
• Maintaining context as an investigation develops
• Enforcing command-safety policies
• Redirecting agents after an intervention
• Terminating unsafe lines of investigation
• Continuously evaluating model performance and behavior

There is also a gain from having a diverse fleet of models. While they share roughly ⅓ common findings between them, each model brings a new quality.

The model is an ingredient, not the product

Our initial Grok 4.5 analysis showed that the model can deliver impressive offensive-security performance at an attractive cost. Our latest analysis adds an important qualification: Grok will emit ideas that require intervention. Fortunately, it appears unusually willing to accept that intervention and change course.

It’s still a cowboy, but one that takes feedback well.