Apple researchers question whether reasoning models reason

Apple’s machine learning research group published “The Illusion of Thinking,” a paper by Parshin Shojaee, Iman Mirzadeh and colleagues arguing that large reasoning models — chatbots trained to produce extended chains of thought, such as OpenAI’s o-series and Claude with extended thinking — do not reason in the way their step-by-step output suggests. Rather than testing on standard maths or coding benchmarks, whose training-set contamination the authors said made results hard to trust, they used controllable puzzle environments, including Tower of Hanoi and river-crossing variants, where difficulty could be tuned precisely by adding pieces or constraints while keeping the underlying logic the same.

The paper described three regimes as puzzle complexity increased: at low complexity, standard non-reasoning models matched or beat reasoning models; at medium complexity, the extra “thinking” tokens produced by reasoning models earned a real advantage; and past a certain complexity threshold, accuracy for both kinds of model collapsed to zero. The authors reported a further counter-intuitive finding — that models’ reasoning effort, measured in tokens generated before answering, rose with complexity only up to a point and then declined even though the models had token budget left, which they read as evidence of a scaling limit in the underlying reasoning process rather than a resource constraint.

The paper circulated widely and was read by some commentators as evidence that frontier “reasoning” was pattern-matching dressed up in chain-of-thought language rather than genuine problem-solving. It drew a rebuttal within days: a paper by Alex Lawsen of Open Philanthropy, published shortly afterwards, argued that the apparent collapse was largely an artefact of the experimental setup — output-token limits mistaken for reasoning failure, and river-crossing instances that were mathematically unsolvable as posed. The exchange became one of the more closely watched methodological disputes of the year over what puzzle benchmarks actually measure.