OpenAI fixes ARC-AGI-3 harness bug, tripling Sol's score

OpenAI reported that GPT-5.6 Sol’s score on the public ARC-AGI-3 task set nearly tripled, from 13.3% to 38.3%, once it swapped the benchmark’s default evaluation harness for settings the company already used internally for ChatGPT and Codex. ARC-AGI-3 tests whether a model can learn the rules of unfamiliar 2D puzzle games through trial and error across many moves, and the official harness had been discarding the model’s private reasoning after each action and pruning older moves once the running history grew too long — so the model retained a record of what it had done but not why, and had to effectively re-derive each game’s rules from scratch on every turn.

Enabling two settings in OpenAI’s Responses API — retained reasoning and compaction — fixed this, and the corrected run also used roughly six times fewer output tokens per game, since the model no longer had to reconstruct prior reasoning at each step. OpenAI framed the finding as a case for scrutinising evaluation harnesses rather than only model capability: the company noted GPT-5.6 Sol had already been used to solve open problems in mathematics, which sat awkwardly against a low score on a benchmark of children’s puzzle games, and traced the gap to the test infrastructure rather than the model.

The episode became a reference point in a broader argument about what benchmark scores measure at all — whether a low result reflects a model’s limits or an evaluation harness that failed to let the model use its own architecture properly, a question with consequences for every reported score run through a third-party harness rather than a lab’s own infrastructure.