Benchmarks · Safety, security & robustness
ExploitGym
Can an AI agent turn a known software vulnerability into a real, working attack — not merely identify or patch it, but exploit it end to end, including against active defences?
Google, UC Berkeley, MPI-SP, UCSB & collaboratorsReleased 11 May 2026Live
Most security benchmarks stop at finding or patching a bug. ExploitGym, from a large team spanning Google, UC Berkeley, Germany’s Max Planck Institute for Security and Privacy, UC Santa Barbara and other collaborators — including Dawn Song and Nicholas Carlini — asks the question that actually determines real-world risk: can an agent turn a known vulnerability into a working attack, end to end, including against modern defences. The 898 instances it draws on are real, not synthetic, pulled from the public vulnerability-tracking infrastructure researchers already use — OSS-Fuzz for userspace software, ClusterFuzz and human disclosures for Google’s V8 engine, and kernelCTF and syzbot for the Linux kernel.
Run under structured access arrangements with OpenAI and Anthropic that lifted the models’ usual safety filters for the purpose of capability measurement, the results showed exploitation was still far from routine: the best-performing system, an Anthropic model described as Claude Mythos Preview, produced working exploits for 157 of the 898 instances, with GPT-5.5 close behind at 120. Kernel exploitation produced the widest spread between models, and while enabling standard security defences reduced success rates, it did not remove them entirely.
The paper is explicit about its dual-use framing: the same capability that lets a model reproduce an exploit for research purposes is the capability that would let it carry out a real attack, and the authors present the results as evidence for defence-in-depth and continued monitoring of frontier models’ offensive-security capability rather than as a simple leaderboard exercise.
The set
898 real-world vulnerability instances across three domains: 520 userspace bugs sourced from OSS-Fuzz, 185 browser bugs in Google's V8 engine sourced from ClusterFuzz and human reports, and 193 Linux kernel bugs sourced from kernelCTF and syzbot, each reproduced in a containerised environment with varying security protections enabled.
Example
A disclosed vulnerability from OSS-Fuzz, ClusterFuzz or kernelCTF, given to an agent inside a containerised copy of the target with modern defences optionally switched on, which must produce a functioning exploit rather than a description of the flaw.arxiv.org
Where it stands
In the paper's evaluation the strongest system, Claude Mythos Preview, produced working exploits for 157 of 898 instances and GPT-5.5 for 120; kernel exploitation showed the sharpest gap between models, and standard defences reduced but did not eliminate success.