Benchmarks · Real-world & economic value
Agents' Last Exam
also: ALE
Whether an AI agent can complete long-horizon, economically valuable professional tasks — not just answer questions — with a verifiable, checkable outcome.
UC Berkeley RDI (Dawn Song et al.)Released 3 June 2026Live
Agents’ Last Exam asks a narrower question than most model benchmarks: not what an AI system knows, but whether it can actually finish a piece of professional work end to end, in the real software a specialist would use, with a result that can be checked rather than merely judged plausible. UC Berkeley’s RDI centre, working with more than 250 industry experts, built the set around 55 sub-industries — from Adobe After Effects animation to Siemens NX 3D modelling to neuroimaging analysis in FSLeyes — mapped onto the US government’s own occupational taxonomy rather than an academic subject list.
Each task hands an agent an instruction, the necessary input files and the actual application, then grades the output against a reference answer that stays hidden until the run completes, closing off the usual routes to gaming a test set. The initial release covers over 1,000 tasks, with the project stating an intention to grow the pool toward 5,000. Early results are stark by design: across the hardest difficulty tier, the paper reports an average full pass rate under 1% for current agent systems, a gap the authors attribute less to raw model capability than to a shortage of benchmarks that track sustained performance on real, economically valuable workflows rather than short, single-turn questions.
ALE is explicitly framed as a living benchmark rather than a fixed leaderboard snapshot, with tasks added over time and a public leaderboard tracking submissions as they come in. That design — alongside its scale and its direct anchoring to occupational categories — puts it in the same lineage as OpenAI’s GDPval, part of a broader shift in 2025–26 toward benchmarks built from actual professional output rather than exam-style questions.
The set
Over 1,000 tasks (building toward a 5,000-task target), built with more than 250 industry experts across 13 industry clusters and 55 sub-industries mapped to the US O*NET/SOC occupational taxonomy. Each task ships as an instruction, input data and a hidden reference answer; an agent is run against the task and its output graded against the concealed reference.
Example
Create a Resolve project, key the bird from input.mp4, match the composition implied by input.png, and export output/output.mp4.arxiv.org
Where it stands
Billed by its authors as a 'living benchmark' with a continuously expanding task pool; on the hardest difficulty tier, the average full pass rate across current agent systems is reported below 1%.