Terminal-Bench launched

Stanford researchers and the Laude Institute released Terminal-Bench, an open-source benchmark testing whether AI agents can complete realistic tasks inside a real command-line environment, rather than in a chat window. In beta at launch, the benchmark covered roughly 100 tasks, ranging from compiling code repositories and training small machine-learning models to configuring servers and debugging broken systems.

Each task runs inside an isolated, Docker-containerised sandbox with a natural-language instruction. The agent under test drives an actual shell — issuing commands and reading their output — rather than producing a single answer, and success is judged by an automated verification script that checks the final state of the environment rather than grading the agent’s transcript. Alongside the benchmark, the team released Terminus, a minimal reference agent scaffold meant to give a neutral baseline for evaluating how different underlying language models performed at terminal-driven work, independent of any particular commercial coding-agent product.

Early results reported by outside evaluators showed a wide spread: general-purpose commercial coding agents outperformed models paired with the plain Terminus scaffold, and even the strongest systems completed well under half of the tasks, underscoring a persistent gap between agents’ ability to generate plausible-looking commands and their ability to complete multi-step, stateful work in a real environment. The benchmark drew fast community engagement, with outside contributors adding new tasks and alternative agent submissions to its public leaderboard in the months that followed, and a more difficult second version followed later in 2025.