EleutherAI forms to build open language models

EleutherAI began as a Discord server where researchers and hobbyists discussed GPT-3 after OpenAI declined to release its weights, restricting access to a paid API. Founders including Connor Leahy, Sid Black and Leo Gao turned the conversation into a working group with an explicit goal: train and release, at no cost, a language model comparable in scale to GPT-3, and do the surrounding research in the open.

The group had no institutional backing, no dedicated compute budget and no formal membership — anyone could join the Discord and contribute. What it had was volunteer labour from people already fluent in the transformer literature, and a position that access to large models should not be gated by a single company’s commercial terms. That premise put EleutherAI in direct contrast to the path OpenAI had taken, which hardened further two months later when Microsoft took an exclusive licence to the underlying GPT-3 model.

Over the following two years the collective shipped what it had promised: the GPT-Neo and GPT-J model families, and, by the end of 2020, an 825GB training corpus called The Pile built because no open equivalent to OpenAI’s training data existed. EleutherAI later formalised as a non-profit and its research emphasis broadened toward interpretability and alignment as open models became easier to obtain elsewhere. Its founding is usually cited as the point at which “open weights” became an organised counter-current to the frontier labs, rather than an occasional side release — one of the earliest working answers to the question the field otherwise had no ready answer for: who gets to run a large language model.