Benchmarks · Agents, tools & computer use

WebVoyager

also: WebVoyager benchmark

Can a multimodal agent complete an everyday browsing task on a real, live website — not a sandboxed clone — by looking at the page and clicking and typing the way a person would?

Tencent AI Lab & Westlake UniversityReleased 25 January 2024Live

WebVoyager, published by researchers at Tencent AI Lab and Westlake University in January 2024, tests browser agents against 15 real, live websites rather than a sandboxed imitation — 643 everyday tasks such as finding a product, checking a price, or booking something, each completed by actually looking at and operating the genuine site. Because the sites are real and not scripted, the authors built an automated judge using GPT-4V to compare an agent’s final screenshot and action trace against the task’s goal, reporting 85% agreement with human graders.

The original multimodal agent solved about 59% of tasks, comfortably ahead of a text-only variant working from HTML alone, which set an early marker for how much visual grounding mattered for browsing. That marker moved quickly once dedicated browser-agent products arrived: Google DeepMind’s Project Mariner reported 83.5% in December 2024, and OpenAI’s Operator reported 87% a month later, each citing WebVoyager as a headline comparison point against the other.

Because the benchmark runs against live websites that change over time, there is no single continuously maintained public leaderboard the way there is for WebArena’s self-hosted environments — most labs report a figure from their own testing run rather than a result checked against a shared, frozen version of the sites. WebVoyager nonetheless remains one of the most frequently cited yardsticks in browser-agent announcements, alongside newer, harder successors such as Online-Mind2Web.

The set

643 task queries across 15 popular real websites (40-plus tasks per site), plus a further 90 browsing tasks adapted from GAIA. The agent works from screenshots of the actual live site rather than a simulated copy. Grading is automated, using GPT-4V as a judge comparing the agent's final screenshot and trajectory against the task goal, which the authors reported agreed with human judgment 85.3% of the time.

Example

Search Apple for the accessory Smart Folio for iPad and check the closest pickup availability next to zip code 90038.arxiv.org

Where it stands

Cited as a standard reference point in most 2024–2026 browser-agent product launches, with reported scores rising from the high-50s at release to the high-80s within about a year; because it runs against real, changing websites rather than a fixed snapshot, most labs report their own figures rather than a single continuously updated public leaderboard.

How the top score changed hands

  1. January 2024WebVoyager (original, GPT-4V-based)59.1%From the original paper; the text-only variant scored lower still.
  2. December 2024Google DeepMind Project Mariner83.5%Reported by Google as state-of-the-art at the time.
  3. January 2025OpenAI Operator (CUA)87%

Current best: OpenAI Operator (CUA) — 87% OpenAI's own reported figure, ahead of Google DeepMind's Project Mariner at 83.5%.

In the timeline · 3 entries

  1. Google DeepMind ships a computer-use model via the Gemini API

    Built on Gemini 2.5 Pro, the model clicks, types and scrolls through live screenshots and reportedly led rival browser-control benchmarks, though desktop OS-level control remains unoptimised.

    Models & capabilities

  2. OpenAI launches Operator

    Built on a new Computer-Using Agent model layered on GPT-4o, it scored 38.1% on OSWorld against a 72.4% human baseline, and launched to $200-a-month Pro subscribers only.

    Models & capabilities

  3. Google unveils Project Mariner, an agent that operates a Chrome browser

    The prototype scored 83.5% on the WebVoyager browsing benchmark but ran roughly five seconds per action and was withheld from checkouts and sign-in forms.

    Models & capabilities

More agents, tools & computer use benchmarks