HellaSwag

HellaSwag’s premise is almost mischievous: take a sentence describing an everyday situation, generate several machine-written continuations, and keep only the ones that fool the strongest models of the day while still looking obviously wrong to a human. Researchers at the University of Washington and AI2 built the roughly 60,000-example set from ActivityNet and WikiHow sentences using a technique they called “Adversarial Filtering” — an iterative process that discarded any wrong answer a contemporary classifier could already rule out, so what remained was calibrated to defeat exactly the models it would be tested on.

That calibration worked. At launch in 2019, state-of-the-art models — including BERT, which had reached near-human performance on earlier, less adversarial sentence-completion benchmarks — scored under 48% against a human accuracy above 95%, a far wider gap than most contemporary tests showed. The benchmark’s title asked, half-rhetorically, whether a machine could really finish your sentence; for several years after its release, the honest answer was no.

That changed with the arrival of much larger models. By 2023, OpenAI reported GPT-4 scoring 95.3% in a base, few-shot configuration — essentially matching the human ceiling of 95.6% on the same test set — and the official leaderboard closed to new submissions the following year. HellaSwag’s arc mirrored several other adversarially-filtered benchmarks from the same period: built to be hard for the models it was released alongside, and solved within a few generations by models it wasn’t designed to anticipate.