Reference
Benchmarks
A benchmark is a fixed test that lets different AI systems be compared on the same footing. This is a catalogue of the ones that have mattered — what each measures, who built it, where to find it, and how the leading score has moved over time. A recurring pattern runs through almost all of them: a benchmark is released to be hard, models climb it within a year or two, and it saturates — at which point a harder one is built to replace it. That cycle is followed as a story in the benchmarks and their saturation thread.
111 benchmarks · 12 domains
Reasoning & problem-solving
General reasoning, hard exams, puzzles that resist memorisation.
ARC-AGILive
Whether a system can infer an unfamiliar abstract rule from a handful of examples and apply it to a new case, rather than recognising a pattern it has seen before.
Humanity's Last ExamLive
Whether a model can answer the hardest closed-ended questions expert academics could write in their own field, at a difficulty chosen specifically to be far from saturated.
MMLU-ProLive
Whether a model's knowledge holds up once guessing is made hard and questions demand multi-step reasoning rather than recall.
SimpleBenchLive
Whether a model can handle everyday spatio-temporal reasoning, social intelligence and 'trick question' style linguistic traps that an ordinary person finds easy but that memorised knowledge doesn't help with.
GPQA
Whether a model can answer graduate-level science questions that a skilled non-expert cannot solve even with unrestricted web access and half an hour per question.
HellaSwagSaturated
Whether a model can pick the one commonsense-plausible continuation of an everyday situation out of several machine-generated distractors designed to fool it.
AGIEvalSaturated
Whether a model can pass the same standardised human exams — college entrance tests, law school admissions, bar exams, maths competitions — that are used to select and qualify people, rather than a bespoke academic test built only for machines.
BIG-BenchRetired
How language-model performance and calibration change with scale across a deliberately broad, crowd-sourced set of tasks existing benchmarks did not cover — and, in the Hard subset, on the specific tasks where models still trailed average human raters.
ARC (AI2 Reasoning Challenge)Retired
Whether a system can answer grade-school science questions that simple retrieval and word-co-occurrence methods get wrong — a test built specifically to exclude questions that lookup alone could solve.
DROPRetired
Whether a model can read a passage, resolve references across several parts of it, and then perform a discrete operation — such as addition, counting or sorting — on what it found, rather than lifting a single answer span directly from the text.
WinoGrandeRetired
Whether a model can resolve an ambiguous pronoun using commonsense world knowledge, in sentence pairs built specifically to remove the statistical shortcuts that let earlier systems cheat at the task.
Mathematics
From grade-school word problems to unsolved research mathematics.
HMMTLive
Whether a model can solve short-answer problems from the Harvard-MIT Mathematics Tournament within days of each sitting, before the problems have had time to enter training data.
MathArenaLive
Can a model solve maths competition problems released after its training cutoff, so a score reflects reasoning rather than memorised answers or leaked solutions?
Omni-MATHLive
Can a model solve genuinely Olympiad-level mathematics problems, of the kind that saturated benchmarks like MATH no longer contain?
PutnamBenchLive
Can an automated prover produce a machine-checked formal proof, not just a numeric answer, for a Putnam Competition problem?
HARPLive
Whether a model's maths-competition accuracy holds up as problems get harder, using six difficulty tiers built from seven decades of US national competitions.
FrontierMathDisputed
Can a model solve original, unpublished research-level mathematics problems that resist pattern-matching against training data?
AIMEContamination concerns
Whether a model can solve short, single-answer competition-maths problems requiring several steps of reasoning but no written proof.
MATHSaturated
Can a model solve a competition-level mathematics problem and produce a correct step-by-step derivation, not just a lucky final number?
GSM8KSaturated
Can a model solve a grade-school arithmetic word problem that takes several linked steps to work through, rather than a single calculation?
miniF2FSaturated
Can an automated theorem prover formally prove Olympiad-level maths statements, in a form directly comparable across different proof assistants?
Coding & software engineering
Writing code, fixing real bugs, resolving issues in live repositories.
Terminal-BenchLive
Can an AI agent actually operate a computer through a real command-line shell — issuing commands, reading their output, and adapting — to finish a multi-step task, rather than just producing plausible-looking commands?
SWE-bench ProLive
Can a model resolve a realistic, multi-file software engineering task in a codebase it could not have memorised — including private, commercial code rather than only well-known open-source repositories?
LiveCodeBenchLive
How well a model codes on problems it could not have memorised, by dating every problem and checking performance separately on those published before and after the model's training cutoff.
Codeforces / CodeContestsLive
Can a model solve genuinely novel algorithmic problems under contest conditions — the kind that require devising an approach, not recalling one — well enough to rank against real competitive programmers?
SWE-LancerLive
Can a model do the paid work of a freelance software engineer — both writing code that passes real client acceptance tests, and judging which of two competing technical proposals a hiring manager should pick?
Aider PolyglotLive
Can a model act as a practical pair-programmer — reading an existing multi-language codebase, understanding a task, and editing the actual files correctly, not just writing an isolated function?
BigCodeBenchLive
Can a model write a correct program that composes multiple real library functions correctly to follow a complex, multi-step instruction — the kind of task a developer actually does, rather than an isolated algorithm puzzle?
DeepSWELive
Can a coding agent complete an original, long-horizon software engineering task in a real repository, graded by whether the behaviour is correct — not just whether it matches one specific reference implementation?
NanoGPT SpeedrunLive
Originally, how fast a human team can train a small GPT-2-scale model to a fixed validation-loss target; adapted into an AI-agent benchmark testing whether a model can reproduce a known training-speed improvement itself, given only a hint of what changed.
SWE-bench
Can a model resolve a real, unseen GitHub issue by editing a codebase so that the project's own hidden tests pass?
HumanEvalSaturated
Can a model write a correct, working Python function from a natural-language docstring alone?
MBPPSaturated
Can a model write short, correct Python programs for simple, everyday programming tasks — the kind an entry-level programmer would be expected to solve?
Konwinski PrizeRetired
Can an open-source AI system resolve real GitHub issues it could not possibly have trained on, because the test set didn't exist yet when submissions closed?
Agents, tools & computer use
Multi-step tasks: driving a browser, a terminal, a desktop, real tools.
OSWorldLive
Can an agent operate a real desktop computer — a whole Ubuntu, Windows or macOS environment with real applications — to complete an open-ended task, rather than a sandboxed browser or single app?
τ-benchLive
Can an AI agent handle a realistic customer-service conversation — following a company's written policy, calling the right backend tools, and getting the outcome right — while talking to a simulated customer who has their own goals and can change their mind?
GAIALive
Can an AI assistant answer real-world questions that are conceptually simple for a human but require reasoning, web browsing, tool use and handling multiple file types to actually solve?
WebArenaLive
Can an autonomous agent operate a real, fully functional website — clicking, typing, navigating menus — to complete a task specified in natural language, the way a person actually uses the web?
WebVoyagerLive
Can a multimodal agent complete an everyday browsing task on a real, live website — not a sandboxed clone — by looking at the page and clicking and typing the way a person would?
TheAgentCompanyLive
Can an AI agent do a full day of ordinary knowledge work inside a simulated company — writing code, filing tickets, messaging colleagues, filling in spreadsheets — well enough and consequentially enough to be judged on outcomes, not just individual isolated tasks?
VisualWebArenaLive
Can a multimodal agent complete realistic web tasks that specifically require understanding an image — matching a product photo, judging a picture in a listing — not just reading and clicking text?
AutomationBenchLive
Can an AI agent carry out a real business workflow across multiple SaaS applications — finding the right API endpoints itself, following a company's own layered business rules, and getting the right data to the right system — rather than completing a single, well-specified task?
AndroidWorld
Can an agent operate a real Android phone — navigating apps, typing, tapping, adjusting settings — to complete a task described in natural language, with success checked against the device's actual resulting state?
BrowseCompDisputed
Can an agent find a specific, hard-to-locate fact on the open web by searching persistently and connecting scattered clues, rather than by knowing the answer already or finding it in one search?
Online-Mind2WebDisputed
How well does a web agent actually perform on real, live websites under conditions close to genuine use — and how much of the field's reported progress on other web-agent benchmarks holds up under an independent, harder check?
AgentBenchRetired
How well a language model performs as an agent — not answering questions, but taking sequences of actions in interactive environments such as an operating system, a database, or a game — across eight distinct settings at once.
Science & research
Domain knowledge and research work in the sciences.
RE-BenchLive
How does an AI agent's performance on real, open-ended machine-learning research-engineering tasks compare with a human ML researcher's, at matched time budgets?
MLE-benchLive
Can an AI agent do the work of a machine-learning engineer end to end — preparing data, training and tuning models, and iterating — well enough to place on a real Kaggle leaderboard?
HealthBenchLive
How well does a model handle realistic, open-ended health conversations — with a layperson or a clinician — judged against criteria that practising physicians say actually matter, rather than a multiple-choice medical exam?
PaperBenchLive
Can an AI agent replicate a machine-learning research paper from scratch — reading it, writing the code, and running the experiments needed to reproduce its results?
SciCodeLive
Can a model write the code a scientist actually needs — numerical methods, simulations and calculations — to solve a real research problem, built up step by step rather than in one shot?
LAB-BenchLive
Can a model do the practical work of biology research — finding facts in the literature, reading figures and tables, planning protocols, and reasoning over DNA and protein sequences?
ChemBenchLive
How does a model's chemical knowledge and reasoning compare with a trained human chemist's, and does it know the limits of its own answers?
SciBenchLive
Can a model solve open-ended, college-level science problems that require multi-step quantitative reasoning, not just recall or short factual answers?
Knowledge & factuality
What a model knows, and whether it admits what it does not.
SimpleQALive
Whether a model gives a correct, confidently stated answer to a short factual question that has exactly one verified answer, or appropriately admits it doesn't know.
TruthfulQALive
Whether a model gives a truthful answer to questions written specifically to trigger common human misconceptions, rather than repeating the popular-but-false claim it likely absorbed in training.
IFEvalLive
Whether a model reliably follows simple, mechanically checkable instructions bundled into a prompt — a word count, a required keyword, a formatting rule — rather than merely producing a plausible-sounding response.
FRAMESLive
Whether a retrieval-augmented system can answer a genuinely multi-hop question that requires pulling facts from several documents and reasoning across them, not just retrieving one relevant passage.
SimpleQA VerifiedLive
The same question OpenAI's SimpleQA asks — can a model give a correct, confidently stated answer to a short factual question — on a smaller set re-checked to remove the label noise, duplication and topic imbalance found in the original.
TriviaQA
Whether a model can answer a real trivia question, either by reading supporting evidence documents (reading comprehension) or from parametric knowledge alone (closed-book question answering).
MMLUSaturated
How much a model knows across 57 academic and professional subjects, tested as four-option multiple-choice questions from elementary to expert level.
Language & multilingual
Understanding, translation and reasoning beyond English.
MMMLULive
Whether a model retains its general-knowledge accuracy when the same MMLU questions are asked in a language other than English.
MGSMLive
Whether a model can carry out the same grade-school arithmetic word-problem reasoning in languages other than English, not just answer general-knowledge questions in translation.
FLORES-200Live
How well a translation system converts text between any pair of 200 languages, using the same set of professionally translated sentences in every language.
BelebeleLive
Whether a model can read a short passage and answer a factual question about it correctly, tested in parallel across 122 languages and dialects using exactly the same underlying questions.
Global-MMLULive
Whether a model's MMLU-style general knowledge holds up across 42 languages, and separately, whether its score depends on knowledge specific to a particular culture rather than being culturally neutral.
CMMLULive
How much a model knows across a broad span of subjects when tested natively in Chinese, including subjects specific to Chinese culture, history and civil-service-style knowledge that an English test would not cover at all.
IndQALive
Whether a model can answer questions that require cultural and contextual knowledge specific to India, in Indian languages, rather than knowledge that happens to be translated into them.
C-EvalRetired
How much a model knows and can reason about, tested in Chinese and calibrated to the Chinese education and professional-qualification system rather than translated from an English test.
Multimodal
Vision, audio, video and charts — reasoning over more than text.
MMMULive
Can a model answer college-exam-level questions that genuinely require reading an accompanying image — a chart, diagram, map or chemical structure — rather than knowledge alone?
Video-MMELive
Can a model understand a video — not just a single representative frame — across clips ranging from 11 seconds to an hour, drawing on visual content, subtitles and audio together where needed?
BLINKLive
Whether a multimodal model can do core visual perception — judging relative depth, matching visual correspondences, spotting image tampering, reasoning across multiple viewpoints — that people solve almost instantly but that resists being reduced to language description.
MVBenchLive
Whether a multimodal model can answer a question about a video that requires genuine temporal reasoning — motion, order, counting, causality — rather than being answerable from a single freeze-framed image.
RealWorldQALive
Does a model's visual understanding hold up on ordinary real-world photos — many taken from inside or around a vehicle — that test spatial reasoning and everyday physical common sense, rather than on curated benchmark imagery?
MathVista
Can a model do mathematical reasoning when the problem is given as a picture — a plot, a geometry diagram, a puzzle figure — rather than as text?
ScreenSpot
Given a screenshot and a natural-language instruction, can a model click the correct on-screen element? Tests GUI grounding — locating the right icon, button or text field — across mobile, desktop and web interfaces.
MMBench
How consistently, not just how often, does a vision-language model get a multiple-choice visual question right — even when the answer options are shuffled?
ChartQA
Can a model answer questions about a bar, line or pie chart that require reading values off it and then performing a logical or arithmetic operation on them?
AI2D
Can a model answer a multiple-choice question about a labelled science-textbook diagram — reading its text labels, following its arrows, and understanding the relationships they express — rather than just recognising objects in a photo?
DocVQASaturated
Can a model locate and correctly read the specific piece of text in a scanned business or government document — a form, letter, report or table — needed to answer a natural-language question about it?
VQAv2Saturated
Given a photograph and a free-form question about it, can a model give the answer a human would — while resisting the shortcut of guessing from the question's wording alone, since every question is asked of two near-identical images with different correct answers.
Safety, security & robustness
Jailbreaks, dangerous-capability evals, honesty, sabotage, harm.
HarmBenchLive
How reliably a model refuses to help with a harmful request when an attacker is actively trying to jailbreak it, rather than when simply asked.
WMDPLive
How much hazardous knowledge a model can supply in biosecurity, cybersecurity and chemical security — used both to flag dangerous capability and as a target for 'unlearning' methods that try to remove that knowledge without damaging general ability.
MASKLive
Whether a model contradicts its own stated beliefs when placed under pressure to lie — honesty, measured separately from accuracy, rather than as a proxy for it.
SHADE-ArenaLive
Whether an AI agent can secretly sabotage a task it has been trusted with while evading a monitoring system, and whether that monitor can catch it — testing agentic deception under realistic conditions rather than a single-turn chat prompt.
StrongREJECTLive
Whether a jailbroken model's response is actually useful for the forbidden request, rather than just non-refusing — correcting a pattern where earlier jailbreak evaluators counted a rambling, low-quality answer as a full success.
CyberSecEvalLive
Two separate cybersecurity risks in a language model used as a coding assistant: how often it suggests insecure code, and how willing it is to help with an actual cyberattack when asked.
SEC-bench ProLive
Can a model find a genuine, previously undisclosed-style vulnerability in a large, real codebase and prove it with a working exploit input — not just patch a bug it has already been shown?
SEC-benchLive
Can an LLM agent handle a real software security task — reproducing a vulnerability and then patching it — in an authentic, containerised codebase?
AgentHarmLive
Whether an LLM acting as an agent — using tools across multiple steps, not just chatting — will carry out a harmful task, and whether a jailbreak that fails on a chatbot still works once the model has tools to act with.
CybenchLive
Whether an AI agent can autonomously find and exploit real security vulnerabilities, using the same professional Capture the Flag (CTF) format security researchers train on.
JailbreakBenchLive
How well a jailbreak attack defeats a model's safety training, and how well a defence holds up against a standard set of attacks — tracked as an open, ongoing leaderboard rather than a one-off score.
ExploitBenchLive
How far an AI agent gets through the actual chain of an exploit — not just whether it crashes a target, but whether it can turn that crash into control of the machine.
ExploitGymLive
Can an AI agent turn a known software vulnerability into a real, working attack — not merely identify or patch it, but exploit it end to end, including against active defences?
Real-world & economic value
Professional work products and economically valuable tasks.
GDPvalLive
Whether a model's output on a real occupational work task is judged, by blinded industry professionals, as good as or better than a human expert's.
METR Time HorizonLive
The length of a task, measured in the time a skilled human would need, that a model can complete autonomously with 50% success probability.
Vending-BenchLive
Whether an AI agent can run a simple simulated business — a vending machine — coherently over a very long horizon, rather than just complete a short task correctly.
Agents' Last ExamLive
Whether an AI agent can complete long-horizon, economically valuable professional tasks — not just answer questions — with a verifiable, checkable outcome.
CORE-BenchLive
Whether an AI agent can computationally reproduce the results of a published scientific paper — installing dependencies, running the authors' own code, and answering questions about the output.
WorkBenchLive
Whether an AI agent can correctly complete a realistic office task — sending an email, scheduling a meeting, updating a record — using a sandboxed set of business tools, without taking a wrong or harmful action along the way.
Long context & retrieval
Finding and using information across very long inputs.
RULERLive
A model's 'effective context length' — the longest input it can actually use while holding accuracy above a fixed threshold — rather than the context length it claims to support.
LongBenchLive
How well a model understands and reasons over long, realistic documents — not just whether it can retrieve one planted fact — across question answering, summarisation, few-shot learning, code and structured-data tasks.
MRCRLive
Whether a model can distinguish between several near-identical, repeated requests scattered through a long conversation and retrieve the correct one — a harder test than finding a single unique fact.
InfiniteBench
Whether a model can process, retrieve from and reason over inputs longer than 100,000 tokens — a range existing long-context benchmarks at the time mostly stopped well short of.
Needle in a HaystackSaturated
Whether a model can retrieve a single fact ('the needle') planted at an arbitrary point inside a long document ('the haystack'), across varying document lengths and needle positions.
Aggregate indices & arenas
Human-preference arenas and composite indices that roll many tests into one ranking.
Artificial Analysis Intelligence IndexLive
A single composite score summarising a model's capability across agentic tasks, coding, scientific reasoning and general knowledge, built as a weighted average of several independent evaluations Artificial Analysis runs itself rather than reported by the model developer.
LiveBenchLive
How a model performs on recently created questions, graded by objective ground-truth answers rather than human or LLM judgment, so that scores cannot reflect memorised test data and cannot be inflated by a biased judge model.
Artificial Analysis Coding Agent IndexLive
How well a coding agent — a specific model paired with a specific harness, such as Claude Code or Codex — completes real software-engineering work end to end, not just whether the underlying model answers a coding question correctly.
HELMLive
How a foundation model performs not on accuracy alone, but across multiple metrics evaluated together — originally accuracy, calibration, robustness, fairness, bias, toxicity and efficiency — so that a model's trade-offs are visible rather than collapsed into one score.
LMArenaDisputed
Which of two anonymous models people prefer in open-ended, head-to-head conversation, aggregated into a running Elo rating rather than a fixed test score.
Open LLM LeaderboardRetired
How an open-weight language model scores on a fixed suite of automated academic benchmarks, run and standardised by Hugging Face rather than self-reported by the model's developer.