HELM

HELM began as an argument about what a benchmark should even measure. When Stanford’s Center for Research on Foundation Models published it in November 2022, most language-model evaluation reduced a model to a single accuracy figure; HELM instead scored 30 models across 16 scenarios on seven metrics at once — accuracy alongside calibration, robustness to prompt changes, fairness, bias, toxicity and computational efficiency — on the view that a model’s weaknesses on the secondary measures were as informative as its strength on the primary one.

That multi-metric standard proved harder to sustain as a single fixed leaderboard than a narrower benchmark would have, and HELM evolved into a family of separately run leaderboards on a shared open-source framework rather than one table: HELM Classic preserves the original paper’s scenarios, HELM Lite offers a lighter in-context-learning suite, and newer additions such as HELM Capabilities, MedHELM and a long-context leaderboard target specific capabilities or domains, alongside vision-language and audio-language variants and several non-English-language benchmarks. Every leaderboard publishes full prompt-level outputs and is reproducible through the framework’s code, a transparency commitment few commercial or single-metric benchmarks match.

That breadth is also HELM’s limitation as a public reference: without one headline number, it has never carried the same shorthand power in press coverage or release announcements as MMLU or Chatbot Arena, even as it remains a standard citation in academic and policy discussions of evaluation methodology, including the Stanford AI Index. Its leaderboards continue to be updated at different speeds — the flagship Capabilities leaderboard tracks frontier releases, while some domain-specific variants move more slowly.