ARC Prize 2024 winners and technical report published

The ARC Prize Foundation closed ARC Prize 2024 — the annual competition built around François Chollet’s ARC-AGI benchmark of visual reasoning puzzles designed to resist memorisation — without any team reaching the 85% score needed to claim the $500,000 grand prize.

The top score on the private evaluation set was 55.5%, up from 33% the previous year, which the organisers called the largest single-year gain since the competition began in 2020. That top entry, from the team MindsAI, was not eligible for prize money because it did not open-source its approach, a condition of entry; the highest-scoring open submission, at 53.5%, came from a team called the ARChitects and won the $25,000 first prize. Five teams won placement prizes and seven papers split a further $90,000 in awards, with the top paper award going to a submission on combining program induction with transduction for abstract reasoning.

The accompanying technical report grouped the competing approaches into three families: program synthesis guided by code-generating language models, transductive models refined with test-time training, and hybrids of the two. The organisers framed the result as double-edged — proof that the field could still make rapid progress on a benchmark specifically designed to resist brute-force scaling, but also confirmation that ARC-AGI had survived a year of sustained, well-funded attack without being solved. OpenAI’s o3, tested against the same benchmark two weeks later, would substantially outscore every 2024 entrant.