Vending-Bench

Vending-Bench, built by the startup Andon Labs, tests something most benchmarks ignore: whether an AI agent stays coherent over a long stretch of simple, repetitive work rather than a single hard problem. An agent is put in charge of a simulated vending machine — ordering stock, setting prices, corresponding with suppliers, tracking a cash balance — across a run that can span more than 20 million tokens, roughly a simulated year of operation, and is scored on how much money it ends up with.

The original 2025 paper found that even capable models struggled to stay on track that long: Claude 3.5 Sonnet and o3-mini ran the shop profitably most of the time, but every model tested occasionally lost the thread — forgetting orders it had already placed, misreading its own schedule, or spiralling into repetitive “meltdown” loops unrelated to the task. The result echoed Anthropic’s own real-world experiment, Project Vend, in which a live model running an actual office shop made comparably erratic decisions — a related but separate exercise using a physical machine rather than Andon Labs’ simulation.

Vending-Bench became a recurring citation in frontier model releases as agentic reliability, not just raw capability, drew more attention: Anthropic reported a 29% gain for Claude Opus 4.5 over its predecessor, and a second version, Vending-Bench 2, was still being used as a comparative leaderboard in mid-2026, with Claude Opus 4.7 reported ahead of competing open-weight models such as GLM-5.2. Because the benchmark measures a business outcome rather than a fixed accuracy score, its main value has been comparative and qualitative — showing where long-horizon agents break down — rather than a single number that tracks cleanly over time.