Grok 4.7 Is Different. So Is the Best Way to Use It.

XBOW had early access to Grok 4.7, and we tested its capabilities across our offensive security benchmarks and workflows.

The results were not straightforward. In some of our tests, Grok 4.7 performed slightly worse than Grok 4.6; in others, it performed significantly better. We did not see a new vulnerability class where 4.7 suddenly excelled, or a clear area where it consistently regressed. The more interesting difference was when it performed well: when we were holding it in a particular way.

With Grok 4.6, xAI’s Build environment was not the strongest way for us to use the model. With Grok 4.7, the model’s behavior appears to have shifted toward the shorter, more atomic interactions that Build is designed around. When we ran 4.7 in Build-based orchestration, its offensive security performance improved substantially relative to 4.6. It's not quite GPT-6 or Mythos level quality, but it is getting closer. Outside Build, it moved slightly in the opposite direction.

The result highlights a change in how people juggle frontier models. Back in the day, you would adapt your prompts to the model. By now, you need to adapt the orchestration system too.

How we evaluated Grok 4.7

We evaluated an early Grok 4.7 candidate across two themes.

The first was exploit crafting, using XBOW’s production harness against controlled vulnerable targets. We point the system at a known vulnerable location and measure whether the model can successfully craft an exploit. This is our standard model benchmark and isolates one of the most important parts of XBOW’s workflow: turning knowledge of a vulnerability into working proof that it can actually be exploited. We ran 50 benchmarks with 10 repetitions each.

The second evaluation went beyond exploit crafting to look at agentic offensive security more broadly. We evaluated the model’s influence on prioritization, organization, and agents working together over the course of a whole pentest.

We targeted large websites containing many vulnerabilities and measured how many we could find while tracking false positives. We originally developed this benchmark to test out alternative agentic systems, so we covered three expansive targets in total, across three workload sizes and four experimental systems.

We recently explored setting up our agents using xAI’s Build, which came in handy: we happened to have two different systems ready to go that were leveraging Build-based agents. Two other systems used unrelated approaches (but worked well with Grok in the past).

The headline numbers were mixed

In our production exploit-crafting harness, Grok 4.7 was slightly down from Grok 4.6 when both were given a fixed iteration budget. In many cases, 4.7 needed more iterations to reach the same result.

At the same time, each individual completion tended to be shorter. Grok 4.7 was doing less work per interaction, and consuming fewer tokens per iteration, even as it sometimes required more interactions overall. However, even when measuring efficiency by token and not by action, Grok 4.7 didn’t present an obvious advantage over 4.6.

In previous evaluations, XBOW found Grok to be one of the most efficient exploit finders, and its most efficient mode is high reasoning. However, what surprised us in our experiments with Grok 4.7: not only did xhigh reasoning not appear worth it, but it was actually not distinguishable at all from high reasoning. We double-checked our setup, and it does turn out that “low” reasoning is much less effective (for token budgets > 1 million, at least), so it does seem to be that there’s simply a real point of diminishing returns reached in terms of reasoning effort.

Grok 4.7’s advances are in managing itself

When we expanded the test from exploit crafting to the broader offensive security workflow, the aggregate results initially looked noisy. Grok 4.6 won some tests; Grok 4.7 won others. The difference did not correspond cleanly to a particular target or vulnerability type.

Then we separated the results by orchestration type. With Grok 4.6, Build-based orchestration performed substantially worse than our external harnesses: on a typical run (after equal-weight averaging), they would make 42, rather than 72, true positive findings. With Grok 4.7, the Build-based systems made 68 correct findings.

Looking only at our existing production harness would have made Grok 4.7 look like a slight regression. But looking at the model inside different orchestration systems revealed a different story. Grok simply became better suited to a particular way of working.

Grok 4.7 behaves differently

Grok 4.7 produces shorter individual completions than 4.6, although its overall vocabulary and response patterns are otherwise similar. For example, it has a stronger preference for issuing short shell commands rather than writing longer Python scripts. This is one way that its behavior maps naturally onto Build.

Agentic coding environments work by letting a model take an action, observe the result, and decide what to do next. In that kind of loop, small, atomic commands can be useful (and more easily trained for) because they produce feedback quickly. The model makes a small amount of progress, sees what happened, and adjusts. Conversely, when optimizing for a smaller amount of well-crafted actions, larger Python scripts can perform several steps at once.

Grok 4.7 follows a trend toward the first style, taking short but rapid individual steps. We cannot say from these results alone exactly how Grok 4.7 was trained or optimized. But the behavior is consistent with a model that is increasingly well matched to Build, and the benchmark results show that the pairing now matters much more than it did with Grok 4.6.

One old failure mode disappeared

We also saw one significant reliability improvement. Occasionally, Grok 4.6 would get stuck reasoning or explaining indefinitely and consume the available context without ever taking the action. In other words, it could spend all of its time making a plan and none executing it. This happened in roughly 0.85% of Grok 4.6 runs. We observed it in 0% of the Grok 4.7 runs we evaluated, and we did not identify an obvious new recurring failure mode that replaced it.

Less than 1% may sound small, but autonomous systems make many calls. When an agent is executing thousands of steps, a low-frequency failure that forces work to be retried (and often re-fails at the same step!) can become significant.

And “one less failure mode” may sound insignificant, but step by step, this is how model families mature.

The model and the orchestrator are co-evolving

The Build result fits a pattern we are seeing across frontier model providers. Every orchestration system has its own mechanics: how it manages context, invokes tools, starts subagents, and structures the model’s interaction with its environment. As providers build their own agent systems, their models are increasingly likely to be developed and optimized in environments that resemble those systems.

OpenAI has Codex. Anthropic has Claude Code. xAI is doubling down on Build. The model and the environment around it are becoming more closely coupled, which has implications for developers building on top of frontier models. A few years ago, using a new model often meant figuring out the prompts that worked best for it. Now, the orchestration layer itself increasingly needs to fit the model.

For autonomous offensive security, the orchestrator controls how the model explores an application, follows promising leads, recovers from failed approaches, and decides what to test next. A mismatch between model and orchestration can leave capability on the table even when the underlying model has improved. Our Grok 4.7 results are a good example. Exploit crafting in our existing production harness moved slightly down, and full offensive security performance outside Build also declined slightly. But using Build-based orchestration, performance improved substantially relative to Grok 4.6. It’s not Mythos or GPT Astra yet. But it keeps getting closer.