Benchmarks · Knowledge & factuality
SimpleQA
also: Simple QA
Whether a model gives a correct, confidently stated answer to a short factual question that has exactly one verified answer, or appropriately admits it doesn't know.
OpenAIReleased 30 October 2024Live
Most factuality benchmarks mix reasoning, comprehension and recall together, which makes it hard to tell whether a wrong answer came from bad knowledge or bad reasoning. OpenAI built SimpleQA to isolate the first problem: 4,326 short questions, each adversarially chosen to have one verified, unambiguous answer, so that a model’s response can only be graded correct, incorrect, or — crucially — not attempted. That third category was the point. Rather than only rewarding accuracy, the benchmark also credits a model for declining to answer when it is unsure, treating a confident wrong answer as worse than an honest “I don’t know.”
At launch, no model came close to acing it: OpenAI’s own o1-preview led with 42.7% correct, ahead of GPT-4o’s 38.2%, while contemporary Claude models scored under 30%. The gap closed quickly. By the time GPT-4.5 shipped in February 2025, OpenAI’s results table put it at 62.5% — the company framed the same improvement as a drop in confabulation rate, from roughly 60% for GPT-4o to 37% for GPT-4.5.
SimpleQA became a standard reference for measuring hallucination and calibration rather than raw capability, cited alongside accuracy benchmarks in most frontier release reports through 2025. OpenAI’s simple-evals repository, the reference implementation, stopped adding new model results in July 2025, though the dataset and grading code remain live — so later frontier scores exist in individual model reports rather than in one continuously updated table.
The set
4,326 short, adversarially collected fact-seeking questions spanning history, science, technology, art and more, each checked by two independent reviewers for a single unambiguous answer. Responses are graded correct, incorrect, or not attempted, since the authors wanted the benchmark to reward calibration rather than confident guessing.
Example
'Who received the IEEE Frank Rosenblatt Award in 2010?' (Answer: Michio Sugeno) — one of the paper's own worked examples of a single-answer, adversarially checked factual question.arxiv.org
Where it stands
OpenAI's simple-evals reference repository stopped tracking new model releases in July 2025, but kept SimpleQA as one of three evaluations it still hosts.
How the top score changed hands
- October 2024OpenAI o1-preview42.7%Best-scoring model in the launch paper's own results table; GPT-4o scored 38.2% and contemporary Claude models under 30%.
- February 2025GPT-4.5 (preview)62.5%Reported alongside OpenAI's own hallucination-rate framing of the same evaluation.
Current best: GPT-4.5 (preview) — 62.5% Highest score recorded in OpenAI's own simple-evals results table; the repo stopped adding new models a few months later, so later frontier models are not confirmed here.
In the timeline · 2 entries
OpenAI releases GPT-4.5
Priced at $75/$150 per million tokens, about thirty times GPT-4o's rate, and retired from the API within five months in favour of the cheaper GPT-4.1.
Models & capabilities
OpenAI publishes SimpleQA, a benchmark for factuality
A short-form factuality benchmark designed to be more challenging and less saturated than prior QA benchmarks.
Benchmarks & progress