AgentBench

AgentBench, published by a Tsinghua-led team in August 2023, was one of the first attempts to measure “LLM-as-agent” ability as a single, structured thing rather than a scattering of one-off demos. It bundles eight distinct interactive environments — operating a real Linux shell, querying a database, navigating a knowledge graph, playing a digital card game, solving lateral-thinking puzzles, and three tasks adapted from existing agent datasets including Mind2Web — and scores a model across all of them at once, since a genuinely capable agent should not fall over the moment the setting changes.

The original results captured a moment when “agentic” behaviour was still mostly a commercial-model story: GPT-4 led with an overall average of 4.01 on the paper’s scoring scale, ahead of Claude 2’s 2.49, while open-source models of the time averaged well under one — a roughly fourfold gap between API-based and open-weight systems that the authors flagged as the paper’s headline finding, alongside poor long-horizon reasoning and instruction-following as the dominant failure modes across every environment tested.

That original benchmark is no longer where frontier labs compete; its fixed environments and 2023-era task design were overtaken by harder, more realistic successors such as WebArena and GAIA. The same Tsinghua group relaunched the project in October 2025 as AgentBench FC, rebuilt around function-calling and reinforcement learning through their AgentRL framework, aiming to give the underlying idea — one composite score across many kinds of agent task — a second run against current models.