Benchmarks · Science & research
PaperBench
Can an AI agent replicate a machine-learning research paper from scratch — reading it, writing the code, and running the experiments needed to reproduce its results?
OpenAIReleased 2 April 2025Live
PaperBench tests a specific, high-stakes capability: not whether a model can answer questions about machine-learning research, but whether an agent can turn a published paper into working, reproducing code entirely on its own. OpenAI selected 20 Spotlight and Oral papers from ICML 2024 and, with help from the original authors, broke each one down into a hierarchical grading rubric — 8,316 gradable sub-tasks in total — covering whether the agent correctly understood the paper’s contribution, built a codebase implementing its method, and ran experiments that actually reproduced the reported results.
The task is deliberately close to what a research engineer does when replicating someone else’s work, rather than a simplified proxy for it. At launch, the best-performing combination — Claude 3.5 Sonnet (New) paired with an open-source agent scaffold — averaged around 21% on the replication score, and OpenAI reported that top machine-learning PhD graders working the same rubric outperformed every model tested, leaving a wide gap between current agents and a competent human replicating a paper end to end.
PaperBench belongs to a cluster of benchmarks, alongside MLE-bench and RE-Bench, built to measure whether AI systems can do the practical, multi-step work of research itself rather than talk about it — a capability OpenAI and others treat as directly relevant to how much AI could eventually accelerate its own field. The rubric-based grading and full task set were released openly so other groups could evaluate their own agents against the same standard.
The set
20 ICML 2024 Spotlight and Oral papers, each broken down (with the original authors' help) into a hierarchical rubric of 8,316 individually gradable sub-tasks covering understanding the paper's contributions, building a codebase, and executing experiments that reproduce its findings.
Example
Two literal leaf-node grading criteria from the paper's Section 2.4: Result Match — "The recorded F1-scores show that removing the frequency prior term from the representation based forecasting method reduces the average F1-score for all model, dataset and fine-tuning setups." Execution — "The code to evaluate the prior-free representation based forecasting method on all model, dataset and fine-tuning configurations present in Table 1 has been executed and the F1-scores have been recorded."arxiv.org
Where it stands
At release, the best-performing setup — Claude 3.5 Sonnet (New) with an open-source agent scaffold — reached an average replication score of about 21%, well below top ML PhD graders working the same tasks.