AI2 releases Dolma open training corpus

The Allen Institute for AI (AI2) released Dolma, a 3-trillion-token dataset drawn from web pages, academic papers, code repositories, books and encyclopedic text, built specifically to support open research on large-scale language models. The dataset applied filtering for personally identifiable information and toxicity, and AI2 published its processing methodology openly so the corpus’s construction, not just its contents, could be scrutinised.

Rather than a fully unrestricted release, Dolma was published under AI2’s “ImpACT” licence, a medium-risk tier that required downstream users to register their contact details and intended use, disclose any derivative datasets, redistribute those derivatives under the same terms, and avoid a short list of prohibited uses including military surveillance and disinformation generation — a middle path between a fully open dataset and a closed one, and a departure from the largely unrestricted licences used by most earlier open corpora.

Dolma’s stated purpose was to close a specific gap: most open-weight language models at the time, including Meta’s Llama family, published model weights without publishing the training data or data-processing code behind them, making it impossible for outside researchers to fully reproduce or audit training. AI2 used Dolma to train its own OLMo family of language models, whose intermediate checkpoints and training logs were also released, positioning the pairing as one of the most fully open pretraining pipelines available at the time.