Codeforces / CodeContests

Codeforces is not an AI benchmark by design — it is a competitive-programming platform, running timed contests judged by strict hidden test cases, that has existed since long before large language models. It became one anyway, because its problems are genuinely novel each contest, its scoring is unforgiving, and its Elo-style rating system already gives a precise, human-calibrated scale to compare against. DeepMind’s AlphaCode made that comparison explicit in 2022, entering ten real Codeforces contests and ranking in roughly the top 54% of several-thousand-strong fields — DeepMind’s own description of the first AI system to reach a competitive standard in programming contests. Alongside the result, DeepMind released CodeContests, a curated dataset of some 13,000 problems drawn from Codeforces and similar judges, which became a standard training and evaluation resource in its own right.

What followed was one of the sharper capability curves in this record. OpenAI’s o1-preview reached roughly the 89th percentile in 2024, and o3 reported a 99.2nd-percentile rating that December — a jump from median-human to elite-human performance in under three years. As frontier reasoning models pushed further into territory with few human competitors to compare against, reporting shifted from contest percentile toward raw Elo figures quoted directly on the same numeric scale used for humans: Google reported Gemini 3 Deep Think reaching an Elo of 3455 in early 2026, within range of the strongest human grandmasters on the platform.

Because live Codeforces problems are set after any given model’s training cutoff, the platform has held up better against contamination concerns than most static coding benchmarks — though methodologies still vary: some labs enter live contests directly, others estimate a rating by scoring retrospectively against archived problems, and the two are not strictly comparable. It remains one of the few benchmarks in coding where a model’s score can be read on literally the same scale as a human’s.