Anthropic publishes 'Many-shot Jailbreaking' research

Anthropic published research describing “many-shot jailbreaking,” a technique that defeats a language model’s safety training by embedding a long sequence of fabricated dialogue turns in a single prompt, each showing an assistant persona complying with a progressively more harmful request, before ending with the request the attacker actually wants answered.

The attack works because it exploits in-context learning: models generalise from examples in their prompt, and enough examples of “compliant” behaviour make refusal less likely on the final one. Anthropic reported that attack success followed a power-law relationship with the number of shots — more fake examples produced more reliable jailbreaks — and that the technique became practical only as context windows grew from roughly 4,000 tokens in early 2023 to over a million tokens by 2024. The company tested it against its own models, including Claude 2.0, and said models produced by other developers were also vulnerable, with larger models showing greater susceptibility.

Anthropic said it briefed other AI labs and academic researchers about the vulnerability before publishing, giving them time to implement mitigations. Of the fixes it tested, restricting context length was ruled out as impractical, and further fine-tuning against the pattern only shifted the number of examples required rather than closing the hole. A classification-and-modification step applied before the prompt reached the model was more durable, cutting one attack’s success rate from 61% to 2% in Anthropic’s testing.

The disclosure was notable less for the specific attack than for what it said about a structural problem: the same expanded context that made models more useful — able to hold entire documents, codebases or conversation histories — also gave attackers more room to manipulate behaviour, and the fix had to be re-derived for each new context length rather than solved once.