AI coding agents used to mass-check whether 2,200 ICML papers reproduce

Hugging Face reported results from a 19-day open reproducibility challenge in which 1,221 volunteers used AI coding agents — including Claude Code, Codex, Cursor, Pi and OpenResearch’s orx — to test claims made in papers accepted to ICML 2026 — the International Conference on Machine Learning, one of the field’s largest and most selective academic conferences, where being accepted is a mark of prestige. Participants examined 2,226 of the conference’s roughly 6,350 accepted papers, about a third of the total, and logged individual judgements on 35,908 specific claims between them.

The exercise targets a long-standing weakness in scientific publishing. A paper accepted to a conference like ICML has been vetted by peer reviewers, but those reviewers read the write-up and rarely rerun the authors’ code to confirm the reported results actually hold. “Reproducing” a study means doing exactly that — rebuilding its experiment independently and checking that the same numbers come out — and it is how bad or overstated results are meant to be caught. Doing it by hand across thousands of papers is infeasible, which is what made AI agents that can read a paper and then write and run code to test it an attractive shortcut.

The agents were tasked with reading a paper’s stated claims, writing code to reproduce the reported experiment, running it, and reporting whether the result held up. Across the papers checked, 51% had at least one claim independently verified through a real rerun of the experiment, while 23% had at least one claim the volunteers judged falsified or contested. Problems participants surfaced included mathematical proofs with undetected errors, theoretical claims with working counterexamples, mismatches between a paper’s stated method and its actual implementation, and evaluation metrics inflated through what the organisers described as padding.

Hugging Face framed the exercise as a demonstration of what large-scale, agent-assisted verification can add to peer review, but its write-up also emphasised the limits of automation: participants found that agents were reliable at executing experiments but that human oversight remained necessary to steer agents toward the right checks, question underlying assumptions and make judgement calls — such as evaluating whether a result matched a paper’s claim in spirit, not just in output — that current systems could not consistently make on their own. The challenge ran from 15 July to 2 August 2026, with results published on 13 August.