MATH

Where GSM8K tests grade-school arithmetic, MATH tests the mathematics that ambitious high-schoolers compete over. Dan Hendrycks and colleagues at UC Berkeley drew 12,500 problems from AMC and AIME-style competitions, each requiring a full derivation rather than a single lookup, and covering everything from algebra and geometry to number theory and probability. At release, even the largest available language models solved barely one problem in twenty — the paper’s authors concluded that scaling model size alone was not going to close the gap, and that genuine algorithmic progress in reasoning would be needed.

That prediction held for a while and then didn’t. Google’s Minerva, further trained on scientific text, reached roughly 50% in 2022 — close to a median human competitor. OpenAI’s process-supervised reward models, trained to check each reasoning step rather than only the final answer, pushed a representative subset to 78% the following year, and open 7B models such as DeepSeekMath closed much of the remaining gap to proprietary systems without any external tools.

By the time OpenAI’s o1 reasoning model arrived at the end of 2024, MATH scores were reported near 95%, and the benchmark had stopped distinguishing frontier systems from each other. Researchers built Olympiad-level successors — Omni-MATH among them — explicitly because MATH, like GSM8K before it, had run out of headroom to measure the models it was designed to challenge.