Microsoft and NVIDIA train Megatron-Turing NLG at 530 billion parameters

Microsoft and NVIDIA announced Megatron-Turing NLG, a 530-billion-parameter dense transformer language model, describing it as the largest of its kind trained to date — three times the parameter count of Turing-NLG 17B, the pair’s previous largest model, and larger than GPT-3’s 175 billion. It followed the by-then familiar formula of training a single large dense model on a broad web-scale text corpus and evaluating it on zero-, one- and few-shot NLP benchmarks, where the companies reported state-of-the-art results.

The announcement’s substance was as much about infrastructure as about the model. Training combined NVIDIA’s Megatron-LM tensor-slicing with Microsoft’s DeepSpeed pipeline parallelism in a three-way parallel scheme — data, pipeline and tensor parallelism together — spread across DGX A100 servers on NVIDIA’s Selene and Microsoft’s Azure NDv4 supercomputing clusters, connected by InfiniBand. Each model replica spanned hundreds of GPUs. The companies published a fuller technical paper the following January detailing the training system and reporting per-GPU throughput figures, alongside benchmark scores on tasks including reading comprehension and commonsense reasoning.

Megatron-Turing NLG was never released, as a product or as open weights; like GPT-3, it was described through blog posts and a technical paper rather than made broadly accessible. Its significance lay less in any specific capability than in what the collaboration demonstrated: that two of the infrastructure companies best positioned to do so — a chipmaker and a cloud provider — could jointly stand up training runs at a scale that outpaced what any single lab had shown, reinforcing the industry’s growing bet that more parameters and more compute, engineered carefully across thousands of GPUs, continued to buy measurable gains on standard benchmarks.