IFEval

Judging whether a model followed an instruction usually means judging quality — did it write well, was the tone right — which is exactly the kind of subjective call that resists automated scoring. Google Research published IFEval in November 2023 to sidestep that problem: around 500 prompts, each built from one or more of 25 “verifiable instruction” types — a word-count range, a required keyword, an output format, a capitalisation rule — chosen specifically because a script can check compliance without ambiguity.

The launch paper’s own comparison showed a wide spread even among capable models. GPT-4 followed 76.9% of prompts exactly on the strict prompt-level metric (83.6% at the more forgiving instruction level, where partial credit counts individual constraints within a prompt separately), while PaLM 2-S managed only 43.1%. The gap mattered because it measured something knowledge tests like MMLU do not: whether a model that clearly knows the right content will actually do what it is told with it.

IFEval became one of the standard components of Hugging Face’s Open LLM Leaderboard and a routine citation in instruction-tuned model releases through the mid-2020s. Because its checks are mechanical rather than model-graded, it has been comparatively resistant to the grading drift that affects benchmarks scored by another LLM. Frontier models released since 2023 are widely reported to clear the original GPT-4 baseline comfortably, though a specific up-to-date leaderboard figure was not independently confirmed here.