miniF2F

miniF2F asks a narrower question than most maths benchmarks: not whether a model can produce the right final answer, but whether an automated prover can construct a complete formal proof that a strict proof-checking system accepts — and whether that proof can be attempted equally in more than one such system. OpenAI researchers Kunhao Zheng, Jesse Michael Han and Stanislas Polu published it in 2021 as 488 Olympiad-level statements, drawn from AMC, AIME and IMO problems, hand-translated in parallel into four formal proof assistants — Lean, Metamath, and partially Isabelle and HOL Light — so that progress in different research communities, each built around its own system, could finally be compared on the same problems. The benchmark’s own baseline, GPT-f, solved under 30% of the Lean test split even at pass@8.

Formal theorem proving stayed a slow-moving research corner for several years before reinforcement-learning-trained provers arrived in force in 2025. DeepSeek’s DeepSeek-Prover-V2-671B reached an 88.9% pass ratio on the Lean test split in April, and by August, Goedel-Prover-V2 pushed past 90% in self-correction mode using a 32-billion-parameter model roughly a twentieth the size of DeepSeek’s — while ByteDance’s Seed-Prover, published the same summer, described its own result as essentially saturating the benchmark.

With leading systems clearing the overwhelming majority of its 244 test problems, miniF2F’s role shifted from a live scoreboard to a baseline sanity check: OpenAI archived the original repository in May 2026, and labs reporting formal-proving results increasingly cited harder successors, including PutnamBench’s undergraduate-level problems and research-level sets built to outlast the same saturation that overtook miniF2F.