ARC Prize launches ARC Prize Verified program

The ARC Prize Foundation launched ARC Prize Verified, a certification process for scores on its ARC-AGI benchmarks, in response to growing inconsistency between self-reported and third-party results. The foundation said figures for the same model often varied by dataset curation, prompting method and other undisclosed choices, making leaderboard comparisons unreliable.

Under the new program, only scores run through ARC’s own process on its confidential hidden test set qualify for a “verified” badge; the test set itself stays private, but the methodology and verification steps are published and reviewed by an independent academic panel — Todd Gureckis (NYU), Guy Van den Broeck (UCLA), Melanie Mitchell (Santa Fe Institute) and Vishal Misra (Columbia) — which audits testing practices and co-authors best-practice guidance.

The move formalised a distinction the field had been making informally for some time, between numbers labs published themselves and numbers an outside party could reproduce. It did not stop labs from self-reporting scores elsewhere, but gave ARC Prize’s own leaderboard a tier that other benchmark maintainers, facing the same complaint about score inflation, could point to as a model.