Scaling test-time compute paper argues extra inference compute can beat bigger models

A paper by Charlie Snell, Jaehoon Lee, Kelvin Xu and Aviral Kumar, researchers at UC Berkeley and Google DeepMind, argued that spending additional compute at inference time — rather than at training time — could be a more efficient way to improve a language model’s answers than simply training a bigger model.

The paper compared two families of technique: letting a verifier model search over and re-rank multiple candidate answers, and letting the model revise its own answer iteratively. Its central finding was that neither approach worked uniformly; effectiveness depended heavily on how hard a given prompt was relative to the base model’s own capability, so a fixed test-time strategy wasted compute on both easy and very hard problems. The authors proposed adaptively allocating compute per prompt based on an estimate of its difficulty, and reported that this “compute-optimal” allocation could match the performance gain of pretraining a roughly 14-times-larger model, on prompts where extra test-time compute was well suited to the task, and could improve on a naive best-of-N sampling baseline by more than four times.

The paper was released without any commercial framing — no product used the technique — but it landed weeks before OpenAI shipped its first reasoning model, o1-preview. Because o1 externalised model reasoning into a longer chain of intermediate steps that consumed more compute at answer time, the paper was widely read afterward as an early theoretical case for exactly the trade-off OpenAI had made commercially: that scaling how long a model thinks about a problem could substitute, within limits, for scaling how big the model itself is. It became a frequently cited reference point in the subsequent argument over whether pretraining scaling was running into diminishing returns and where the next capability gains would come from.