DeepMind's Chinchilla paper rewrites the scaling laws

DeepMind published “Training Compute-Optimal Large Language Models,” reporting that the field had been allocating its training compute badly. The prevailing guidance, from OpenAI’s 2020 scaling-laws paper, had been read as implying that most additional compute should go into more parameters. Training over 400 models across a range of sizes and data volumes, the DeepMind authors found instead that parameters and training tokens should be scaled in roughly equal proportion.

The demonstration was direct. Chinchilla, at 70 billion parameters trained on 1.4 trillion tokens, outperformed Gopher at 280 billion parameters trained on 300 billion tokens — a model four times its size, trained with the same compute budget. It also beat GPT-3 and Megatron-Turing NLG across a wide set of benchmarks, and reached 67.5% on MMLU.

The practical consequence was immediate and lasting. A smaller model trained on more data is cheaper to serve, which changed the economics of deployment as much as of training. Nearly every significant model released afterwards was trained on far more tokens per parameter than the pre-Chinchilla norm, and the ratio kept climbing past what the paper called optimal, because inference cost — not training cost — increasingly dominated.

It also reframed what the binding constraint was. If capability required tokens as much as parameters, then the supply of high-quality text became a strategic resource. The paper is the origin of the “running out of data” argument that shaped later debates about synthetic data, licensing deals with publishers, and the copyright litigation that followed.