BigScience releases BLOOM open multilingual model

The BigScience research collective, coordinated through Hugging Face, released BLOOM, a 176-billion-parameter language model that generated text in 46 natural languages and 13 programming languages. It was trained over 117 days, from March to July 2022, on the publicly funded Jean Zay supercomputer in France, using compute granted by the French research agencies CNRS and GENCI. BigScience described the effort as the largest collaboration of AI researchers ever assembled for a single project: more than 1,000 people from over 70 countries and 250 institutions contributed to the model’s design, training data curation and evaluation.

BLOOM’s stated purpose was to demonstrate that a model of GPT-3’s scale could be built with full transparency rather than behind a closed research organisation, and its release went further than typical open-weight launches: BigScience published intermediate training checkpoints and optimiser states as well as the final weights, letting outside researchers study how the model’s capabilities developed during training rather than only inspect the finished product. The model was released under a “Responsible AI Licence,” a novel structure that permitted broad use and modification but prohibited a specified list of harmful applications, an attempt to keep weights open while placing some legal constraint on misuse.

BLOOM did not match GPT-3 or the contemporary PaLM on English-language benchmarks, and its multilingual training data was uneven in quality and volume across languages. Its significance was structural rather than a matter of leading capability: it showed that GPT-3-scale training was reproducible outside a commercial lab, on public compute, with public accountability for the process — a template that later open efforts, including Meta’s LLaMA models, would follow in spirit if not in openness of process.