MBPP

MBPP — Mostly Basic Python Problems — sets a lower bar than its more famous cousin HumanEval, and that was the point. Google researchers introduced it in 2021 as a crowd-sourced set of roughly 974 short Python tasks pitched at what an entry-level programmer could solve: check whether a number is a perfect square, find the largest value in a list, that kind of thing. Each problem carries a one- or two-sentence description, a reference solution, and three test cases, and — like HumanEval — a submission is graded by whether it passes those tests, not by how closely it resembles the reference code.

The benchmark’s simplicity made it an easy second data point to report alongside HumanEval, and it appears that way in most coding-model releases through the early 2020s: Meta’s Code Llama 34B reported 56.2% in 2023, and labs including DeepSeek and Hugging Face’s StarCoder team routinely cited MBPP scores next to HumanEval ones as a general fluency check.

That pairing became MBPP’s undoing as much as HumanEval’s. Both benchmarks are small, static, and have been public on the web for years, which by 2023–24 made training-data contamination a real concern once scores climbed into the 80s and 90s. Researchers built LiveCodeBench partly in direct response, pulling fresh competitive-programming problems on a rolling basis so a model’s cutoff date could be checked against the problem’s publication date. MBPP still gets reported as a familiar baseline, but it no longer carries much signal about frontier coding ability on its own.