Benchmarks · Safety, security & robustness
ExploitBench
How far an AI agent gets through the actual chain of an exploit — not just whether it crashes a target, but whether it can turn that crash into control of the machine.
Carnegie Mellon UniversityReleased 13 May 2026Live
Earlier security benchmarks tended to score exploitation as pass or fail — did the target crash, yes or no. ExploitBench, built by researchers at Carnegie Mellon University, argues that misses most of the story: a crash is only the first step in a real attack, and the harder, more consequential steps come after. It breaks the process into a 16-stage capability ladder running from reaching and triggering vulnerable code, through building sandbox-escape and memory-corruption primitives, to arbitrary read and write access and finally full arbitrary code execution — and scores an agent by how far up that ladder it climbs, not just whether it reaches the top.
The initial release runs the ladder against 41 real bugs in Google’s V8 JavaScript engine, comparing agents given only the target environment against agents given adaptive coaching feedback and a native command-line harness. The pattern that emerged was consistent: publicly available frontier models routinely got as far as triggering a crash but rarely converted that into working code execution, while one privately tested frontier model reached full code execution on roughly half the bugs — evidence, the authors argue, that hardened, real-world exploitation is only now becoming a capability frontier models are starting to cross, rather than one they have already mastered.
ExploitBench’s code, containers and model transcripts are published openly, with a leaderboard maintained at exploitbench.ai, positioning it as a running instrument for tracking exploitation capability rather than a one-off study — a distinction that matters given how directly the same capability bears on both defensive research and offensive risk.
The set
A graded 'capability ladder' rather than a pass/fail score: exploitation is broken into 16 measurable stages, from reaching and crashing vulnerable code, through building sandbox and memory-corruption primitives, to arbitrary read/write, control-flow hijack and finally arbitrary code execution. The initial release instantiates the ladder against 41 real bugs in Google's V8 JavaScript engine, testing models with and without adaptive coaching feedback.
Example
A known vulnerability in V8, handed to an agent that must work through the ladder itself — first reaching and triggering the buggy code, then building the primitives needed to read and write memory it shouldn't be able to, and ultimately achieving arbitrary code execution — with the agent's furthest stage on the ladder recorded as its score.arxiv.org
Where it stands
Publicly available frontier models routinely reached and crashed vulnerable code but rarely achieved arbitrary code execution; one privately-tested frontier model reached code execution on roughly half the bugs, which the authors describe as an emerging capability rather than a solved one.