Needle in a Haystack

Needle in a Haystack began as one researcher’s informal pressure-test rather than an academic benchmark. Greg Kamradt built it in November 2023 to check whether OpenAI’s newly released 128,000-token GPT-4 Turbo could actually use all of the context window it claimed, by planting a single out-of-place sentence somewhere inside a long document and asking the model to find it. Sweeping the test across a grid of document lengths and insertion depths produced a heatmap that made a model’s blind spots visible at a glance — and the visual made the test spread fast, becoming close to a default first check in any long-context model announcement.

The test also produced one of the more widely circulated anecdotes in the field: when Anthropic tested an early version of Claude 3 Opus with it, the model not only found the inserted sentence but remarked that it appeared to have been placed there to test its attention — a moment Anthropic’s own researchers described as a curiosity rather than a claim about the model’s broader awareness. Within roughly a year, Google’s Gemini 1.5 Pro launch helped make million-token context windows a standard line in system cards, and needle-in-a-haystack sweeps became a routine part of how labs demonstrated them.

Basic single-needle retrieval is now close to solved for frontier models at most context lengths, so labs have moved to harder variants: multiple needles inserted at once, at the longest context lengths a model supports. Anthropic’s Claude Opus 4.6 release, for instance, reported 76% accuracy on an eight-needle test at 1 million tokens against 18.5% for its predecessor Sonnet 4.5 — evidence that, even as the original test saturates, its harder descendants still separate current models by a wide margin.