MMLU-Pro

MMLU had a guessing problem: with four answer choices, a model that knew nothing could still score 25%, and by 2024 leading systems were closing in on the accuracy ceiling anyway. TIGER-AI-Lab’s response, published in June 2024, kept MMLU’s format — multiple-choice questions across academic and professional subjects — but expanded each question to ten options and skewed the question mix toward problems that need several reasoning steps rather than a single recalled fact.

The redesign changed which techniques helped. On the original MMLU, chain-of-thought prompting made little difference over a model simply answering directly; on MMLU-Pro it produced a real gain, evidence that the harder questions were rewarding actual reasoning rather than pattern-matched recall. Scores also became more stable: sensitivity to how a question was phrased fell from a 4–5 point swing on MMLU to about 2 points here. At launch, the best-performing model, GPT-4o, scored around 72.6%, some 14 points below its MMLU number, and Claude 3.5 Sonnet moved ahead within weeks at just over 76%.

MMLU-Pro has proved more durable than its predecessor but is not immune to the same pressure: by 2026 the top model on TIGER-Lab’s own leaderboard, Gemini 3.1 Pro, was scoring above 91%, and the gap between frontier systems had narrowed again. It remains one of the standard secondary citations alongside MMLU in model release reports, particularly for labs wanting to show a harder knowledge test than the original still supports.