Engineering the Impossible: How XBOW De-Duplicates Findings

Key Takeaways:

• XBOW makes a judgment call that used to need a human analyst: are these two findings the same bug?
• Testing a large application autonomously often unearths the same weaknesses on different endpoints and with different parameters.
• Reporting every finding buries the security team, but de-duplicating too much risks hiding vulnerabilities that need fixing.
• The concept of "duplicate" depends on the finding and is very nuanced.
• XBOW splits each finding into four views (description, reproduction, location, impact) and embeds them separately. The weights shift by vulnerability class.
• The results: Reported finding volume is down 30%, and 90% of duplicate findings have been eliminated.

One of the less obvious challenges that comes with finding vulnerabilities at scale is deciding when two findings are actually the same vulnerability.

XBOW may encounter the same underlying weakness repeatedly as it tests a large application across different endpoints, parameters, users, and workflows. Reporting every occurrence can overwhelm security teams with redundant information. But collapsing findings too aggressively creates the opposite problem.

At XBOW, reducing that noise meant teaching the system something security analysts do intuitively: deciding whether two findings are duplicates depending on context. This is how we approached that problem.

The de-duplication problem is context-dependent

Consider a verbose stack trace or debug error disclosure that affects an entire application. The same underlying condition may appear across dozens of URLs or endpoints, since every route hits the same framework misconfiguration. That does not make each occurrence a separate vulnerability. In many cases, a customer only needs one canonical finding that captures the application-wide issue.

Cross-site scripting is different. A reflected XSS in the search parameter on /products may be entirely distinct from a reflected XSS in the redirect_url parameter on the same endpoint. Each may require separate investigation and remediation.

A rule that correctly groups application-wide error disclosures might incorrectly collapse distinct XSS findings. On the other hand, a rule designed to preserve each XSS injection point might produce dozens of redundant reports for one systemic disclosure issue.

Finding A GET /employee/reviews/{id} "Server returns full stack trace..." 
Finding B GET /api/employees/review/{id} "Verbose error leaks internal paths..." 

Finding C GET /employee/reviews?paramA=<XSS_PAYLOAD> 
Finding D GET /employee/reviews?paramB=<XSS_PAYLOAD>

Why embedding the entire finding is not enough

We began with an embedding-based approach.

An embedding model transforms data, such as text or images, into a numerical representation called a vector. Related concepts are placed closer together in a mathematical space, allowing a system to measure how similar they are, typically via cosine similarity.

This made embeddings a natural starting point for de-duplication. Findings that appeared close together in the vector space could be grouped. But embedding an entire finding as a single block of text created too much noise.

A vulnerability finding contains several kinds of information, including:

• A description of the vulnerability
• Steps for reproducing it (the exploit chain, HTTP requests, payloads)
• The affected location (host, path, parameter, method)
• The potential impact

Findings also contain a lot of repeated language. Phrases such as “an attacker could” appear across many unrelated attack types since explanatory text may be reused in descriptions, reproduction steps, and impact statements. When all of that content was embedded together, the repeated language pushed unrelated findings closer together in the vector space.

URLs and paths created another problem. The same location string could appear in the description, the reproduction steps, and the location field of a finding, unintentionally giving it more weight in the final similarity score.

We could clean up the text and embed only the most finding-specific language, but a single embedding still provided limited flexibility. It did not allow the system to treat location as critical for one vulnerability class and largely irrelevant for another.

Breaking each finding into independent views

Instead of treating a finding as one piece of text, we projected it into four independent views:

• Description: the nature of the vulnerability
• Reproduction: how it can be triggered
• Location: where it occurs
• Impact: what an attacker achieves

Each view is embedded separately. The resulting per-view similarity scores are then recombined into a single similarity score using a weighted sum, where the weights vary by CWE.

For an application-wide error disclosure, location receives relatively little weight. The same underlying issue appearing across several URLs could still be treated as one canonical finding. For cross-site scripting, location and parameter details might carry much more weight. Two findings with nearly identical descriptions and impacts could remain separate when they affect different inputs.

The views gave the system multiple perspectives on the same finding. By changing the relative importance of each perspective, XBOW could define a different de-duplication topology for each class of vulnerability rather than forcing every finding through one global model.

Preventing cross-view pollution

Separating findings into views was only part of the solution. We also needed to ensure that each view contained only the information it was intended to represent.

Early versions still included URLs and paths in the description, even though those details were also present in the location view. Repeated templates and boilerplate language appeared across descriptions, reproduction steps, and impact statements. We called this problem cross-view pollution.

If location information appeared in both the location and description views, lowering the weight of the location view would not fully remove location from the calculation. The same information would still influence the result through the description embedding. Repeated information could therefore be counted multiple times, distorting the intended weighting.

To prevent this, we cleaned and normalized the text before embedding it. URLs, paths, repeated templates, and overlapping elements were removed or standardized so that each view remained as independent as possible.

Normalization was especially important for dynamic locations. For example, /users/1/profile and /users/2/profile may represent the same logical location even though their literal paths differ. If those identifiers were compared directly, the system could incorrectly treat the paths as distinct.

We normalize numeric IDs, UUIDs, tokens, and other high-cardinality path segments to canonical placeholders (e.g. /users/{id}/profile) before embedding. The same normalization is applied inside reproduction steps, so a captured request like:

GET /api/v2/orders/7f3c9b21-4e1d-4f88-a2c5-8d9b0a1e2f34/items HTTP/1.1
Host: target.example.com
Authorization: Bearer eyJhbGciOi...

becomes something structurally comparable across findings that hit the same route with different IDs and tokens.

This separation and normalization made the final similarity score more meaningful, and the behavior of the system easier to tune.

Turning analyst judgment into ground truth

When development began, we didn’t have a dataset showing which findings experienced security analysts considered duplicates. Without that dataset, there was no reliable way to determine whether a new configuration was improving de-duplication or making it worse.

So we built a custom browser extension for XBOW’s triage analysts. The extension allowed analysts to work directly with live findings and organize them into clusters using drag and drop. Each cluster represented a canonical finding and the findings the analyst considered duplicates of it. The extension then exported the cluster map using finding IDs. (No customer data was included in the export, only IDs and cluster assignments.)

Over several weeks, this process captured the judgment of XBOW’s triage team and converted it into a structured ground-truth dataset spanning the CWE classes we see most often in production. The exports were processed offline and combined with the embedded vectors for each finding view.

This gave the team a repeatable way to measure whether the system’s clusters matched the decisions experienced analysts would make, using standard clustering metrics (adjusted Rand index, homogeneity, completeness) against the analyst clusters.

Auto-tuning for each vulnerability class

The next challenge was choosing the right weights and similarity thresholds.

With four views, there were many possible configurations. We could assign more or less weight to description, reproduction, location, and impact, while also adjusting the threshold at which two findings would be considered duplicates.

Selecting those settings manually would have required repeatedly running the de-duplication system, comparing the results with the analyst-created dataset, and adjusting the configuration by hand. A setting that worked well for one vulnerability class could also perform poorly for another. Instead, we built an auto-tuner.

For each CWE, the auto-tuner searched the configuration space, testing different combinations of view weights and similarity thresholds. It ran the de-duplication system against the dataset and measured how closely the resulting clusters matched the analysts’ decisions. The highest-scoring configuration is selected for that CWE and checked into the platform’s per-CWE config.

This per-CWE approach removed the bias created by applying one global configuration to every vulnerability class. It also reflected the central insight behind the project: different types of findings require different definitions of “duplicate.”

The production impact

The de-duplication system is now deployed across all XBOW organizations. It runs on every new finding produced by the platform before it reaches the customer's queue. Results so far:

• Total reported finding volume reduced by about 30%.
• Approximately 90% of duplicate findings eliminated.
• On the tuning set, the resulting clusters achieved approximately 90% agreement with analyst-created clusters, based on our clustering evaluation metrics.

The system will continue to evolve. Applications change, testing techniques improve, and new findings create new edge cases. We keep collecting analyst decisions through the same extension workflow and re-run the tuner as the ground-truth dataset grows, so the weights, thresholds, and normalization rules stay honest.

De-duplication is a security judgment problem

De-duplication cannot be solved by textual similarity alone. Two findings can use nearly identical language and still represent separate vulnerabilities. Others can appear across different URLs, users, and workflows while tracing back to the same root issue.

By combining independent embedding views, per-CWE auto-tuning, and human-curated ground truth, XBOW reduces noise without removing the distinctions security teams need to act.

Get more details on the XBOW autonomous offensive security platform in our short 5-minute demo video.