EleutherAI releases the Common Pile v0.1

EleutherAI released the Common Pile v0.1, an 8TB dataset built entirely from public-domain and openly licensed text, drawn from around 30 sources including roughly 300,000 books digitised by the Library of Congress and Internet Archive, an openly licensed subset of the Stack v2 code corpus, and academic papers. The project used the Open License Definition, requiring that included material could be used, studied, modified and redistributed by anyone for any purpose — a stricter bar than “openly available” and one designed to withstand the copyright disputes that had accumulated against web-scraped training sets.

To demonstrate the dataset’s usefulness rather than just its provenance, EleutherAI trained two 7-billion-parameter models on it, Comma v0.1-1T and v0.1-2T, trained on one and two trillion tokens respectively. In ablation studies, the Comma models outperformed other licensed-data efforts such as KL3M, OLC and Common Corpus, and roughly matched older unlicensed datasets like the original Pile and OSCAR, though they trailed FineWeb, which EleutherAI attributed to FineWeb’s much larger source pool permitting stricter filtering.

The release addressed a specific gap in the open-source AI ecosystem: while open-weight models had proliferated, almost none disclosed training data that was verifiably free of copyright risk, leaving open-weight releases exposed to the same infringement claims as closed ones. The Common Pile offered evidence that a fully licensed dataset could produce a model within reach of unlicensed alternatives, even if not yet matching the best of them.