Google trains a 1.6-trillion-parameter Switch Transformer

Google researchers led by William Fedus, Barret Zoph and Noam Shazeer published the Switch Transformer, a language model architecture that reached a trillion parameters while keeping the computational cost of processing each input roughly constant — the first work to demonstrate mixture-of-experts sparsity at that scale.

Standard “dense” transformers, the architecture behind GPT-3 and most models before it, activate every parameter for every input: doubling parameter count roughly doubles compute cost per token. Mixture-of-experts models had existed before Switch Transformer but were considered difficult to train, typically routing each input through several experts at once, which added communication overhead and training instability. Switch Transformer simplified the approach by routing each token to exactly one expert, cutting that overhead, and the paper reported this let training proceed in lower-precision bfloat16 format for the first time in a model of this kind, which had previously been too unstable for sparse routing.

Benchmarked against a dense T5 model trained with the same computational budget, the authors reported up to a sevenfold increase in pre-training speed, a fourfold speedup specifically against T5-XXL, and improvements across all 101 languages tested in a multilingual setting. The trillion-parameter figure headlined the paper, but the substantive claim was that sparsity let parameter count and compute cost decouple — a model could be made far larger without a proportional rise in the compute needed to train or run it.

Switch Transformer’s numbers were not directly comparable to dense models of similar total parameter count, since only a fraction of its parameters activated on any given input, and it did not itself become a widely deployed product. Its architecture nonetheless previewed a mixture-of-experts approach that later frontier labs adopted at production scale, as sparse routing became one of the standard tools for growing model capacity without growing inference cost in lockstep.