OpenAI publishes the GPT-3 paper

OpenAI posted “Language Models are Few-Shot Learners” to arXiv, describing GPT-3: an autoregressive transformer with 175 billion parameters, roughly ten times larger than the largest dense language model previously described in public.

The paper’s central claim was not about size for its own sake. It was that a sufficiently large model could perform tasks it had never been trained on simply by being shown a few examples in its input — what the authors called in-context, or few-shot, learning. On a range of benchmarks the model was competitive with systems that had been explicitly fine-tuned for each task. No gradient updates were involved; the capability appeared to be a side effect of scale.

The paper also reported the limits. GPT-3 performed poorly on tasks requiring multi-step reasoning or comparison of two sentences, showed the biases present in its training corpus, and the authors noted that its samples could still lose coherence over long passages. A section on broader impacts discussed misuse for disinformation and the energy cost of training — an unusual inclusion for a capabilities paper at the time, and one that later became standard.

Access was the other departure. Rather than releasing weights, OpenAI made GPT-3 available through a commercial API, initially by invitation. The decision was contested, and it set the pattern that defined the following years: frontier capability distributed as a metered service rather than as downloadable artefacts.

The paper is the most-cited single reason the field turned toward scale. Its scaling curves became the argument that capability was, to a useful approximation, purchasable — a claim that shaped research agendas, capital allocation and eventually government policy for the rest of the decade.