WebArena

WebArena, released by a Carnegie Mellon University team in mid-2023, tests agents against real software rather than a simplified stand-in. The researchers built and self-hosted four fully working websites — an online shop, a Reddit-style forum, a GitLab-style code-collaboration tool, and a content-management admin panel — plus reference access to a map service and a Wikipedia mirror, and set 812 tasks that require actually clicking, typing and navigating to reach a genuine end-state, checked automatically rather than graded on a written answer.

That realism made early results humbling: a zero-shot GPT-4 agent completed only 14.41% of tasks against a 78.24% human baseline. Progress since has come less from bigger models than from better scaffolding — memory of past interactions, structured planning, retrying failed steps — with systems such as Agent Workflow Memory and IBM’s CUGA climbing into the 35–62% range through 2024 and 2025, and OpenAI’s Operator reaching 58% at its January 2025 launch.

WebArena’s own leaderboard is open submission, tracked on a public spreadsheet linked from the project’s site and code, which keeps it current but means the newest, highest scores come from agents with varying degrees of independent documentation behind them. It remains one of the field’s reference environments for browser-using agents precisely because the sites are real enough that an agent cannot succeed by guessing — and the CMU group behind it has since extended the same approach to harder successors, including a multimodal version and a benchmark of longer, more realistic office-style work.