SimpleQA Verified

SimpleQA Verified exists because OpenAI’s original SimpleQA — for all its early adoption — turned out to have the kind of flaws that are easy to miss until a lot of people are relying on the numbers. Google DeepMind researchers published a re-checked version in September 2025: 1,000 questions cut down from the original 4,326 through de-duplication, topic and answer-type rebalancing, and reconciliation of reference answers that had conflicted with each other, plus a tightened grading prompt. The task itself is unchanged — a short factual question with one verifiable answer, scored correct, incorrect, or not attempted.

In the launch paper’s own comparison, Gemini 2.5 Pro led with an F1 score of 55.6, ahead of GPT-5 at 52.3 and o3 at 51.9; Claude Opus 4 scored well behind at 28.3. Those figures sit below the headline scores some of the same models had posted on the original SimpleQA, consistent with the paper’s stated aim of removing noisy or overly generous labels from the earlier set.

The benchmark is new enough that it has not yet built up its own history of leaderboard turnover, and outside scrutiny has already begun: Epoch AI, which reruns SimpleQA Verified independently for its own capability tracking, has flagged possible data contamination affecting at least one open-weight model’s result. Whether SimpleQA Verified displaces the original as the default short-form factuality reference, or simply becomes one more entry in a lineage of QA benchmarks that keep needing a stricter second pass, was not yet settled as of writing.