LiveCodeBench

By early 2024, two of coding’s most-cited benchmarks had a shared problem: HumanEval and MBPP had been sitting on the open web for years, and nobody could rule out that their exact problems and solutions had been absorbed into the training data of the models being scored against them. LiveCodeBench, published by researchers at UC Berkeley, MIT and Cornell, answered with a benchmark that could not go stale in the same way: it continuously pulls fresh problems from LeetCode, AtCoder and Codeforces contests and tags each with its publication date, so a model’s score can be split between problems that predate its training cutoff and ones that came after.

The date-segmented design did what it was built to do — it produced direct evidence that some models scored measurably better on older, plausibly-memorised problems than on newer ones, turning contamination from a suspicion into something a benchmark could actually show. Beyond plain code generation, LiveCodeBench also grades self-repair, test-output prediction and code execution, giving a fuller picture of coding competence than a single pass/fail generation task. It was adopted quickly and now appears in most frontier model release reports: DeepSeek, for instance, reported its V3 and R1 updates through 2025 climbing from the 30s and 60s into the 70s on the benchmark.

Because the problem pool keeps refreshing, LiveCodeBench has so far avoided the saturation that overtook its predecessors, and its live-update approach became a template other domains borrowed when they ran into the same contamination problem. It is not the final word on coding ability — the underlying problems are still competitive-programming puzzles rather than the messy, multi-file work of real software engineering, which is closer to what benchmarks like SWE-bench try to test — but as a coding leaderboard number resistant to memorisation, it has become close to a default.