Speculative decoding paper shows drafting tokens ahead can speed up LLM inference

Google researchers Yaniv Leviathan, Matan Kalman and Yossi Matias published a paper describing speculative decoding, a way to speed up generating text from a large language model without changing what it outputs. Standard autoregressive generation produces one token at a time, each requiring a full forward pass through the large model — a costly, sequential process that is the main source of latency in serving large models.

Their method instead used a small, fast “draft” model to guess several tokens ahead, which the large model then checked in a single parallel pass. Where the draft model’s guesses matched what the large model would have generated on its own, those tokens were accepted for free; where they diverged, the large model’s own token was substituted instead. Because the large model’s output distribution was still the one determining what was ultimately accepted, the method produced exactly the same outputs, token for token, as standard decoding — it was a wall-clock speedup, not an approximation, which the paper demonstrated on the T5-XXL model with roughly a 2-3x reduction in latency compared to a standard implementation. The technique required no retraining and no architecture changes to the large model itself.

Speculative decoding is a small, technical paper rather than a capability milestone, but it addressed a bottleneck that mattered increasingly as models scaled and serving costs became a real constraint on deployment. The core idea became a standard technique across most major inference-serving stacks — including vLLM and TensorRT-LLM — within about two years, one of several efficiency techniques from this period that shaped what running large language models cheaply and quickly at scale actually looked like in practice.