ARC-AGI

François Chollet introduced the Abstraction and Reasoning Corpus in a 2019 paper arguing that most AI benchmarks measured skill at a task rather than intelligence itself — the ability to handle situations a system had not been prepared for. Each ARC-AGI puzzle gives a handful of input-output grid pairs and asks the solver to infer the hidden rule and apply it to a new grid; humans find the tasks straightforward, but systems that rely on pattern-matching over training data historically did not. For five years the benchmark moved slowly: Chollet and Mike Knoop launched the $1 million-plus ARC Prize competition in June 2024 partly because the public state of the art had crept from 20% to only 34% since 2019.

Reasoning-focused models broke that pattern almost immediately. OpenAI’s o3 scored 75.7–91.5% on the original benchmark within months, close to the foundation’s rough human baseline, though at a cost per task that ran from roughly five times a human solver’s to nearly a thousand times it depending on how much compute was spent. The foundation read the jump as evidence that test-time compute could buy benchmark performance directly, and replaced the saturated original with ARC-AGI-2, redesigned so pure language models scored zero and even strong reasoning systems scored only in the single digits.

ARC-AGI-2 fell just as fast — Gemini models cleared 80% within a year — so in March 2026 the foundation launched ARC-AGI-3, replacing static puzzles with interactive game environments that give no stated rules at all, testing whether a system can learn by exploring rather than recognise a pattern in one shot. Humans scored 100% at launch against 0.51% for frontier AI, though an OpenAI investigation months later found part of that gap came from an evaluation harness that discarded a model’s reasoning between moves rather than from the model itself — a reminder, recurring throughout the ARC-AGI series, that what a score measures depends as much on how it was taken as on what generated it.