Together AI launches RedPajama to reproduce LLaMA's training data

Together, working with Ontocord.ai, ETH DS3Lab, Stanford’s Center for Research on Foundation Models, and the Hazy Research lab, released the RedPajama dataset: a roughly 1.2-trillion-token training corpus assembled to match, as closely as its authors could reconstruct from Meta’s paper, the mixture and proportions of data Meta had used to train LLaMA two months earlier. It drew on the same seven broad sources — Common Crawl web text, C4, GitHub code, books, arXiv papers, Wikipedia and StackExchange — combined in similar ratios.

The point was licensing, not novelty. Meta had released LLaMA’s weights only for approved researchers and under a licence that barred commercial use, which meant no company could legally build a product on it and no one outside the approved list could easily reproduce or audit how it had been trained. RedPajama’s dataset carried no such restriction: anyone could download it, inspect its contents, and train a model on it for any purpose, commercial or otherwise. Together said the goal was “a fully-open, reproducible” alternative to LLaMA covering the full training pipeline — data, then models, then instruction-tuning — rather than just weights someone else had already produced.

RedPajama arrived amid a broader scramble to open up what Meta had kept closed. LLaMA’s weights had themselves leaked onto the open web within days of release, and a wave of derivative fine-tunes — Alpaca, Vicuna and others — had already shown demand for an open LLaMA-class model unconstrained by licence. RedPajama supplied the piece those projects lacked: an open training corpus other groups could use to train comparable models from scratch rather than fine-tuning leaked weights. Together and its partners went on to train and release RedPajama-INCITE models on the dataset later in 2023, contributing to the argument that year, prominent in the open-versus-closed debate, that leading capability did not require keeping a model’s construction secret.