DeepSeek publishes DeepSeekMath, introducing GRPO

DeepSeek published a paper describing DeepSeekMath, a 7-billion-parameter language model built on DeepSeek-Coder-Base and further pretrained on 120 billion tokens of mathematics-related data drawn largely from Common Crawl, supplemented with natural-language and code data. On the MATH benchmark, a set of competition-level mathematics problems, the model scored 51.7% without external tools or majority voting, and 60.9% when using self-consistency over 64 sampled solutions — results the paper described as approaching those of much larger models including GPT-4 and Gemini-Ultra.

The paper’s more consequential contribution was methodological. It introduced Group Relative Policy Optimization (GRPO), a reinforcement-learning technique built as a variant of the widely used Proximal Policy Optimization (PPO) algorithm. Where PPO estimates how good an action was by training a separate value-estimating model, GRPO instead samples a group of candidate outputs for the same prompt and scores each relative to the average of the group, removing the need for the extra value model entirely. DeepSeek reported this both improved mathematical reasoning performance and substantially cut the memory required to run the reinforcement-learning stage of training, compared with standard PPO.

At publication, GRPO registered as an efficiency improvement to a specialised mathematics model rather than a general breakthrough. Its significance became clearer roughly a year later, when DeepSeek used GRPO as the core reinforcement-learning method behind DeepSeek-R1, the reasoning model whose release in January 2025 prompted a sharp reassessment of the compute required to train competitive reasoning systems. GRPO was subsequently adopted widely across the field as a lower-cost alternative to PPO for reinforcement learning from verifiable rewards, well beyond DeepSeek’s own models.