AI Agents for Offsec with Zero False Positives

Large language models are increasingly helping to automate vulnerability discovery and exploit development in real-world software. However, naïvely asking LLMs to identify vulnerabilities leads to a deluge of false positives that can drown out real findings. In this talk, we will present techniques that enable AI agents to find vulnerabilities at scale, fully autonomously and with zero false positives. The key to our approach is developing robust exploit validators that can conclusively determine whether an exploit claimed by the agent is real, allowing the agent to make arbitrarily many attempts without increasing the amount of human effort needed to review the results. Using these techniques, we were able to test thousands of web apps found on Docker Hub, identifying over 200 zero days and obtaining multiple CVEs.