Meta releases OPT-175B with its training logbook

Meta released Open Pretrained Transformer (OPT-175B), a 175-billion-parameter language model built to match GPT-3’s scale, along with a family of smaller baseline models from 125 million up to 66 billion parameters trained on the same data. Meta made the pretrained weights, training code and deployment code — runnable on as few as 16 NVIDIA V100 GPUs — available to researchers, and said the run had used roughly a seventh of the carbon footprint typically associated with training a model of that size, alongside higher reported GPU utilisation than comparable published training runs.

Access was restricted to a noncommercial research licence, available on request to academic researchers, and to people affiliated with government, civil-society and industry research organisations — short of the fully open release Stability AI and others would adopt later that year, but a marked departure from OpenAI’s and Google’s practice of publishing capability claims about GPT-3-scale models without releasing anything runnable.

The most distinctive part of the release was not the weights but the accompanying logbook: a detailed, day-by-day record of the training run, including the compute consumed and the human intervention required each time the underlying infrastructure or the training process itself became unstable at that scale — restarts, loss spikes, hardware failures and the fixes applied. Large-model training runs routinely involve this kind of instability, but labs rarely disclosed it; Meta’s logbook gave outside researchers a rare, granular account of the operational reality of training a model at the GPT-3 frontier.

OPT-175B became a reference point in the open-versus-closed debate that intensified through 2022 and 2023: a large lab demonstrating that near-frontier capability could be shared, imperfectly, without the secrecy that had come to define releases from OpenAI and, later, Google DeepMind.