BIG-Bench

BIG-Bench was assembled the way few benchmarks are: rather than one lab designing every task, more than 450 authors from 132 institutions each proposed their own, producing 204 tasks spanning linguistics, mathematics, common-sense reasoning, biology, social bias and more. The resulting 2022 paper’s central finding was about scale — performance and calibration both improved as models grew, but stayed well behind human raters even at hundreds of billions of parameters, and most tasks improved gradually in ways consistent with better memorisation rather than a qualitative shift.

A minority of tasks didn’t behave that way. On those, models scored near chance until a size threshold, then jumped sharply — the pattern that became central to the “emergent abilities” argument published days later, which used BIG-Bench tasks as some of its clearest examples of discontinuous scaling. That same year, a separate team extracted the 23 tasks where language models had never beaten the average human rater into BIG-Bench Hard, and found that chain-of-thought prompting — reasoning step by step before answering — let PaLM clear the human baseline on 10 of the 23 and Codex on 17, far outperforming the standard prompting used in the original evaluation.

BIG-Bench’s scale made it valuable as an infrastructure project as much as a single result: it demonstrated that broad, crowd-sourced evaluation could surface capabilities narrower benchmarks missed. But that same breadth left it without the focus that let sharper successors like MMLU and later GPQA dominate model release announcements, and by the time frontier labs were routinely citing harder science and reasoning tests, BIG-Bench and BBH had settled into the reference literature rather than the leaderboard.