AI2 releases OLMo 2

The Allen Institute for AI released OLMo 2, a family of base language models at 7 billion and 13 billion parameters trained on up to 5 trillion tokens, and published the weights, training data, code, intermediate checkpoints and instruction-tuned versions together — a fuller disclosure than the “open-weight” releases common from most labs, which typically publish weights without the data or training pipeline behind them.

AI2 reported that OLMo 2 7B outperformed Meta’s Llama 3.1 8B, and that OLMo 2 13B outperformed Alibaba’s Qwen 2.5 7B, in both cases using less total training compute than the comparison models, and described the family as the best-performing fully open models available at the time. Training proceeded in two stages: an initial phase over roughly 3.9 trillion tokens drawn from sources including DCLM, Dolma, Starcoder and Proof Pile II, followed by a roughly 843-billion-token refinement stage weighted toward higher-quality web and domain-specific data. The architecture incorporated RMSNorm and rotary positional embeddings, changes AI2 said improved training stability over long runs. Post-training applied the Tülu 3 recipe AI2 had released five days earlier, combining supervised fine-tuning, direct preference optimisation and reinforcement learning with verifiable rewards.

Released as a paired system with Tülu 3 — a fully open base model and a fully open post-training pipeline — OLMo 2 was aimed less at matching frontier proprietary models than at giving outside researchers a complete, reproducible reference model at a moment when even most “open” releases from larger labs withheld data and training methodology.