AndroidWorld

AndroidWorld, released by Google researchers in May 2024, is the mobile counterpart to desktop and browser benchmarks such as OSWorld: 116 tasks built around 20 real Android apps, each dynamically parameterised so the same underlying task can be phrased and instantiated in effectively endless variations rather than tested once and memorised. Success is checked against the phone’s real resulting state — a calendar entry that actually exists, a setting that actually changed — using dedicated verification logic built into each task rather than trusting the agent’s own report.

Early results tracked the pattern seen across this generation of agent benchmarks: the best agent at launch completed under a third of tasks against an 80% human baseline, and mobile GUI control remained a research problem through most of 2024. That changed as labs built dedicated screen-control models — Google DeepMind’s Gemini 2.5 Computer Use claimed to lead on AndroidWorld among other agent benchmarks by October 2025, without publishing a directly comparable figure — and the benchmark’s open, community-submitted leaderboard has since filled with specialised mobile-agent systems reporting scores above the original human baseline.

That openness is also the benchmark’s current weak point: while Google’s own tested models and a handful of named commercial products anchor the leaderboard’s middle, the highest-ranked entries increasingly come from less-documented teams whose methodology cannot be checked the way an arXiv paper’s can. As with several other benchmarks in this catalogue, AndroidWorld’s trajectory — rapid climb toward and past a human baseline, followed by growing reliance on self-reported leaderboard submissions — has become a familiar shape.