Introducing XBOW
XBOW applies AI to offensive security, autonomously finding and exploiting vulnerabilities across hundreds of web security benchmarks without human intervention.
Editor’s note: These benchmarks were published in 2024. They are now outdated and should no longer be used to measure offensive performance.
I am delighted to introduce XBOW, which brings AI to offensive security, augmenting the productivity of pentesters, bug hunters and security researchers.
XBOW is the first AI product that autonomously passes 75% of web security benchmarks, accurately finding and exploiting vulnerabilities. To verify this claim we ran it against 543 benchmarks from industry leading providers such as PortSwigger and PentesterLab. These benchmarks were designed to train security professionals, and they cover a wide range of vulnerabilities.
To verify that the AI wasn’t recycling known solutions, we also tested it on a set of 104 novel XBOW benchmarks. Impressively, XBOW further successfully tackled 85% of them, confirming its ability to generate original and effective solutions.
We’re excited to share the complete set of results, including detailed traces of how XBOW worked through a number of benchmarks. Reading through these workings, I’m struck how some of the solutions are delightfully original. In offensive security, hallucination can be a feature!
The autonomous capabilities of XBOW suggest a significant boost for bug hunters and security researchers. Since it handles most known vulnerability classes, XBOW will allow security teams to focus on the most enjoyable and innovative aspects of their work. However, these capabilities also raise concerns that malicious hackers could similarly use AI to step up their attacks.
XBOW’s mission is to stay ahead and defeat the bad actors. We are building XBOW because it must be built, and we’re the best team to do it. All of us are deeply committed to making the technology available in a safe and responsible way.
When starting XBOW, I listed my ideal team: top offensive security experts, AI experts from the Copilot team, and leading researchers in security and AI. To my surprise and delight, they all decided to join because we all believe in the mission. And it’s fun!
XBOW is the logical culmination of my own career. I started as a professor at Oxford researching developer tools, founded Semmle (now GitHub Advanced Security) to help engineers find security vulnerabilities in source code, and then founded GitHub Copilot, the first successful application of generative AI. Now, at XBOW, I’m combining all these experiences to empower security and engineering teams with cutting edge security capabilities.
As we move forward with our mission, we are eager to hear from the broader community. Connect with us on X or Mastodon at @xbow. Let us know if there is a benchmark you’d like to see XBOW try!
Let’s build superpowers for offensive security!
