HumanEval

HumanEval asks a narrow, precise question: given nothing but a function signature and an English description of what it should do, can a model write Python code that actually works? OpenAI introduced it alongside Codex in 2021 as a deliberate departure from earlier code-generation tests, which typically scored output by how closely it resembled a reference solution. HumanEval instead ran the generated code against hidden unit tests — a program either passed or it didn’t, regardless of how it got there.

The original 164-problem set was small by later standards but demanding in 2021: Codex solved 28.8% of problems on a single attempt, and the paper’s finding that sampling 100 attempts and keeping any passing one lifted that to 70.2% shaped how code models were evaluated for years afterward. As general-purpose language models absorbed more code in pretraining, HumanEval scores climbed fast — Claude 2 reached 71.2% in mid-2023, and Claude 3 Opus reported 84.9% less than a year later.

That trajectory is also why HumanEval’s usefulness declined. Its 164 problems and reference solutions have been public and widely mirrored since 2021, making contamination — models having effectively seen the answers during training — a standing concern once scores pushed into the 90s. The benchmark still appears in release reports as a familiar reference point, but the field’s attention has shifted to larger, harder or more contamination-resistant tests of coding ability, from MBPP and BigCodeBench to LiveCodeBench and SWE-bench.