Open LLM Leaderboard

For much of 2023 and 2024, a new open-weight model’s standing was measured against one place: Hugging Face’s Open LLM Leaderboard, a public ranking that ran submitted models through a fixed set of automated academic benchmarks via the EleutherAI evaluation harness and published the results without the developer able to cherry-pick which numbers to show. It existed by Falcon-40B’s release in May 2023, where the leaderboard’s top spot served as the third-party evidence behind the model’s “best open-source model” claim, and Hugging Face’s own team used it as a research platform, including a 2023 study documenting how GPT-4 behaved as an automated judge — finding it favoured longer answers and outputs resembling its own training data, a caution that shaped how model-graded evaluation was treated across the field afterward.

The leaderboard’s fixed benchmark suite was also its expiry date. As models improved, scores compressed toward the ceiling on the original tests, and in 2024 Hugging Face replaced them with a harder second-generation suite — MMLU-Pro, GPQA, MuSR, IFEval and BBH among them — to restore separation between models. Even that proved temporary: as reasoning-tuned, assistant-style models became the norm, Hugging Face concluded that static multiple-choice and knowledge tests no longer captured the behaviour that mattered to users.

Hugging Face retired the leaderboard in March 2025, after it had ranked more than 13,000 models across its two versions, rather than build a third static suite it expected to saturate again. It did not name a single successor, pointing instead to a fragmented landscape of more than 200 community-run, domain-specific leaderboards on its platform — an acknowledgment that no single fixed benchmark was likely to serve the open-model ecosystem the way it once had.