LiveCodeBench paper published

Researchers published LiveCodeBench, a coding benchmark built to resist a problem that had grown obvious by early 2024: static test sets like HumanEval and MBPP had circulated on the open web for years and were increasingly likely to sit inside the training data of the models being scored against them. LiveCodeBench instead pulled problems continuously from competitive-programming contests on LeetCode, AtCoder and Codeforces, tagging each by publication date so a model’s performance could be checked separately on problems that predated and postdated its training cutoff.

The initial release evaluated 52 models — 18 base models and 34 instruction-tuned variants — across four hundred problems published between May 2023 and May 2024, testing not just code generation but self-repair, code execution and test-output prediction. The date-segmented design produced direct evidence of contamination: several models scored measurably better on older problems than on ones published after their training cutoff, a gap that a single static benchmark score could not have revealed.

The paper argued that headline leaderboard numbers on existing coding benchmarks likely overstated real capability for at least some models, and that time-segmented, continuously refreshed evaluation was necessary to keep pace with training-data absorption of any fixed test set. LiveCodeBench was adopted widely over the following two years as a standard coding benchmark, and its live-update method influenced later evaluations built to resist the same contamination problem in other domains.