DeepMind's Flamingo tackles multimodal few-shot learning

DeepMind introduced Flamingo, an 80-billion-parameter visual language model built on top of its Chinchilla language model, that could take interleaved sequences of images, video and text as a prompt and generate text in response. Its distinguishing claim was strong performance from a handful of examples: given as few as four demonstrations of a new task at inference time, Flamingo set a new state of the art on 16 multimodal benchmarks, without any task-specific fine-tuning.

That mattered because the previous approach to a new vision-language task — fine-tuning a model on thousands of labelled examples for that specific task — required both a large annotated dataset and a separate training run per task. Flamingo instead handled visual question answering, image and video captioning, and open-ended dialogue about images with the same fixed set of weights, adapted only by which examples were placed in its prompt. DeepMind reported it beat models that had been fine-tuned and optimised specifically for each individual task despite using orders of magnitude less task-specific data, and pointed to potential applications such as describing scenes for visually impaired users and flagging harmful imagery.

Flamingo did not ship as a public product — it remained a research model, described in an accompanying paper rather than released as an API or open weights. Its significance was in demonstrating that the in-context, few-shot pattern that had made GPT-3 useful without fine-tuning could be extended past pure text into a model that reasoned jointly over images and language, a design lineage that fed directly into the multimodal frontier models — GPT-4V, Gemini — that followed over the next two years.