OpenAI publishes scaling laws for neural language models

Jared Kaplan, Dario Amodei and colleagues at OpenAI published an empirical study of how language model performance depends on scale. Testing models spanning more than seven orders of magnitude in size, the paper reported that cross-entropy loss fell as a smooth power law with model size, dataset size and the amount of compute used for training, with remarkably little dependence on architectural details such as network width or depth.

The practical claim was more specific than “bigger is better.” The paper argued that for a fixed compute budget, the most efficient strategy was to train very large models on a comparatively modest quantity of data and stop well before the loss converged, rather than training smaller models to completion. That conclusion — train big, train briefly — reframed how a compute budget should be spent, and it shaped the design of GPT-3, which OpenAI released later that year at 175 billion parameters, considerably larger relative to its training data than the scaling laws’ own later reappraisal would recommend.

The paper’s influence lay less in any single number than in giving the industry a predictive tool: loss curves that behaved lawfully across scales let labs forecast the return on a larger training run before committing the compute, turning scale into a budgeting decision rather than a research gamble. That predictability became the empirical backbone of what the field came to call the scaling hypothesis.

The recommendation did not hold up unmodified. Two years later, DeepMind’s Chinchilla paper argued that Kaplan et al.’s models had in fact been undertrained relative to their size, and that a compute-optimal model needed far more training tokens per parameter than the original scaling laws implied — a correction that reset how every major lab subsequently allocated compute between model size and data.