Benchmarks · Knowledge & factuality
TriviaQA
Whether a model can answer a real trivia question, either by reading supporting evidence documents (reading comprehension) or from parametric knowledge alone (closed-book question answering).
University of Washington (Joshi, Choi, Weld & Zettlemoyer)Released 9 May 2017
TriviaQA predates this record’s own timeline — the University of Washington team of Mandar Joshi, Eunsol Choi, Daniel Weld and Luke Zettlemoyer published it in 2017 — but it stayed in active use well into the language-model era as one of the standard tests of closed-book question answering: can a model answer a real trivia question from what it already knows, with no document to consult? The dataset paired roughly 95,000 questions written by trivia enthusiasts with more than 650,000 question-answer-evidence triples drawn from the web and Wikipedia, and because the questions and evidence were collected separately, matching one to the other often demanded genuine compositional reasoning rather than simple keyword overlap.
That structure gave it a second life once large language models arrived. OpenAI’s GPT-3 paper reported 71.2% accuracy in a few-shot, closed-book setting — no retrieval, no fine-tuning, just examples in the prompt — ahead of the best fine-tuned retrieval system of the time, and the result became one of the pieces of evidence for the paper’s broader claim that scale alone could substitute for task-specific engineering.
TriviaQA continued to appear in evaluation suites for LLaMA, PaLM and other models through the early 2020s, but as newer benchmarks such as MMLU, SimpleQA and TriviaQA’s own successors targeted broader or harder knowledge tests, it gradually receded from release-day headline reporting into the background of standard evaluation harnesses, where it is still commonly run rather than commonly quoted.
The set
About 95,000 question–answer pairs written by trivia enthusiasts, paired with over 650,000 question-answer-evidence triples drawn from Wikipedia and web search results (roughly six supporting documents per question). Because questions and evidence were collected independently, answers often require compositional reasoning and matching across significant wording differences between question and source text — the paper's authors designed it to be harder than prior reading-comprehension sets such as SQuAD.
Example
'The Dodecanese Campaign of WWII that was an attempt by the Allied forces to capture islands in the Aegean Sea was the inspiration for which acclaimed 1961 commando film?' (Answer: The Guns of Navarone) — the paper's own example of a question whose wording diverges from its supporting evidence text.arxiv.org
Where it stands
Predates this record's January 2020 start; included here because it remained a standard closed-book QA citation through the early 2020s, before broader benchmarks like MMLU and SimpleQA took over as headline knowledge measures.