OpenAI announces DALL·E and CLIP

OpenAI published two multimodal systems on the same day. DALL·E was a 12-billion-parameter version of GPT-3 trained to generate images from text descriptions, demonstrated with compositional prompts such as an armchair in the shape of an avocado. CLIP was trained on 400 million image–text pairs collected from the internet to score how well a caption matched an image.

DALL·E drew the coverage; CLIP had the larger effect. Because CLIP could rank arbitrary images against arbitrary text without task-specific training, it provided a general-purpose measure of “does this picture match this description” that other systems could optimise against. Within months, independent researchers were pairing CLIP with existing image generators — VQGAN, then diffusion models — to build text-to-image pipelines outside any lab. That improvised ecosystem, largely conducted in public on Twitter and in shared Colab notebooks, is the direct ancestor of the tools that arrived in 2022.

Neither model’s weights were released initially, though CLIP’s were published shortly afterwards along with the paper in February 2021, and that release is what made the downstream experimentation possible. DALL·E itself was never opened; its successor arrived as a hosted product.

The announcements also marked the point at which the scaling argument from GPT-3 was shown to generalise past text. The same recipe — a large transformer, a large web-scraped dataset, a simple training objective — produced capability in a second modality. That generality became a central plank of the case that the approach had further to run.