Humanity's Last Exam

Humanity’s Last Exam was built to solve a specific problem: MMLU and its peers had stopped being useful once frontier models cleared roughly 90% accuracy, leaving no way to see continued progress. CAIS and Scale AI assembled 2,500 questions submitted by nearly 1,000 academics across roughly 500 institutions, each one filtered to be unanswerable by internet search and gradable by a fixed answer key. At launch in January 2025, the design succeeded almost too well — every model tested, including OpenAI’s o1 and GPT-4o, scored under 10%, and CAIS director Dan Hendrycks said the team genuinely did not know how fast scores would climb.

They climbed fast. Reasoning-focused models pushed past 40% within the year: xAI reported 44.4% for a multi-agent configuration of Grok 4, though the figure had not yet cleared independent verification on the public leaderboard, and Gemini 3 Pro reached 37.5% without tools, rising to 41.0% in its higher-effort Deep Think mode. By February 2026, an upgraded Deep Think reported 48.4% without external tools.

Scores with tool access — letting a model search or use a code interpreter mid-answer — have run higher still, which makes cross-model comparison harder: a number is only meaningful alongside the conditions it was measured under. HLE has nonetheless become, alongside GPQA and ARC-AGI, one of the benchmarks labs reach for specifically because it has not yet saturated, and its scores remain a standard citation in frontier model announcements.