EleutherAI releases GPT-Neo

EleutherAI, an informal collective of independent and volunteer researchers, released GPT-Neo, its first attempt at an open replication of the GPT-3 architecture. The release shipped two trained checkpoints, at 1.3 billion and 2.7 billion parameters — far short of GPT-3’s largest 175-billion-parameter version — trained on the Pile, the 825GB text corpus the group had assembled itself the previous year.

Code and trained weights were published under an MIT licence, with the checkpoints hosted for free download, at a time when GPT-3 was accessible only through OpenAI’s paid, waitlisted API and neither its weights nor its training code were public. Full replication of GPT-3’s scale was still out of reach for a volunteer group without comparable compute; GPT-Neo’s largest model was less than 2% of GPT-3’s parameter count.

The release mattered less for matching GPT-3’s capability than for establishing a template EleutherAI repeated and extended over the following two years — GPT-J, GPT-NeoX and later Pythia — of building the largest openly licensed language models available at each point, while commercial labs kept their own weights closed. It became a reference point in the argument, which sharpened as models grew larger, over whether the field’s most capable systems should be available for anyone to inspect, fine-tune and run.