OpenAI ships InstructGPT and makes RLHF the default

OpenAI announced InstructGPT, a set of GPT-3 variants fine-tuned to follow instructions using reinforcement learning from human feedback (RLHF). Human labellers first wrote demonstrations of desired behaviour, then ranked model outputs against one another; those rankings trained a reward model, which was used to fine-tune the language model further with reinforcement learning. OpenAI made the resulting models the default on its API, replacing raw GPT-3.

The headline result was a size comparison rather than a benchmark score: outputs from a 1.3-billion-parameter InstructGPT model were preferred by human evaluators to outputs from the 175-billion-parameter GPT-3, despite having roughly 100 times fewer parameters. OpenAI also reported that the fine-tuned models fabricated facts less often and produced less toxic output than the base model, with only small performance costs on standard NLP benchmarks — a pattern described in more technical detail in the paper published the following March.

The result reframed what “improving” a language model meant. Raw parameter count and pretraining loss had been the field’s proxy for capability; InstructGPT showed that a comparatively small amount of human preference data, applied through reinforcement learning, could matter more to how useful and controllable a model felt than another order of magnitude of scale. RLHF was not a new research idea, but this was the point at which it became a shipped default rather than an experimental technique.

The lineage is direct: the same alignment approach, refined and applied to a GPT-3.5 model, underpinned ChatGPT’s launch at the end of the year, which took the conversational, instruction-following behaviour InstructGPT had established and put it behind a free public chat interface.