Benchmarks · Agents, tools & computer use
TheAgentCompany
also: TheAgentCompany benchmark
Can an AI agent do a full day of ordinary knowledge work inside a simulated company — writing code, filing tickets, messaging colleagues, filling in spreadsheets — well enough and consequentially enough to be judged on outcomes, not just individual isolated tasks?
Carnegie Mellon UniversityReleased 18 December 2024Live
TheAgentCompany, released by a Carnegie Mellon team in December 2024, asks a blunter question than most agent benchmarks: could this system hold down an entry-level office job? It builds a self-contained simulated software company — running real open-source tools including GitLab, RocketChat and ownCloud rather than mock interfaces — and sets 175 tasks across six roles, from software engineer to HR and finance staff, each requiring the kind of multi-step, tool-switching work an actual employee does in a day rather than a single isolated question.
Results at launch were sobering: the best-performing setup, built on GPT-4o, completed under 9% of tasks outright. That changed as more capable models arrived — by the time Gemini 2.5 Pro was tested in the paper’s updated results, full task completion had risen to roughly 30%, with Claude 3.7 Sonnet close behind, though partial-credit scores for incomplete-but-progressed tasks ran noticeably higher, suggesting agents more often stall partway through a task than fail to start it at all.
Built by the same research group behind WebArena and its successors, TheAgentCompany extends that project’s emphasis on realistic, self-hosted environments to the broader question of economically meaningful agentic work, rather than any single narrow skill. It has become a frequent reference point in discussions of how close current agents are to substituting for entry-level knowledge work, precisely because its tasks were designed to resemble a real job rather than a benchmark.
The set
175 tasks spread across six professional roles — software engineer, product manager, data scientist, HR, finance and administrator — set inside a self-contained simulated workplace built from real open-source tools (GitLab, RocketChat, ownCloud, Plane) rather than mocked interfaces. Scoring combines a primary result-based check of whether the task's real end-state was reached with secondary checkpoint credit for partial progress, using a mix of deterministic and LLM-based evaluators.
Example
Navigate to ownCloud and complete Section B — Alternative Simplified Credit of IRS Form 6765: gather necessary information from /Documents/Financials/TAC_financials.csv and /workspace/research_wages.csv.arxiv.org
Where it stands
The strongest models still complete under a third of tasks outright, with partial-credit scores somewhat higher; the gap between full task success and partial completion is treated by the authors as evidence that agents fail more often on the last mile of a task than on getting started.
How the top score changed hands
- December 2024GPT-4o (via OpenHands)8.6% (full completion)Best result reported in the original paper release.
- March 2025Gemini 2.5 Pro30.3% (full completion)
Current best: Gemini 2.5 Pro — 30.3% (full completion) / 39.3% (partial credit) From the paper's updated results; Claude 3.7 Sonnet followed at 26.3%.