Claude gets computer use

Anthropic released an upgraded Claude 3.5 Sonnet and a public beta called computer use, which let the model operate a computer the way a person does: it viewed screenshots and issued commands to move a cursor, click, and type. Developers could direct it through ordinary desktop software rather than through an API built for the purpose, and Anthropic said it was the first frontier lab to ship the capability broadly rather than as a research demo.

The company reported that the upgraded Sonnet, evaluated with computer use, scored 14.9% on OSWorld, a benchmark of screenshot-only desktop tasks, against 7.8% for the next-best system reported. On SWE-bench Verified, a software-engineering benchmark, the model improved from 33.4% to 49.0%. Anthropic also announced Claude 3.5 Haiku, a smaller model priced at $0.80 per million input tokens.

Anthropic was explicit that the feature remained experimental. It described computer use as “cumbersome and error-prone at this stage,” citing specific weaknesses with scrolling, dragging and zooming, and recommended developers start with low-risk, reversible tasks and supervise the model rather than leave it unattended. The company said it had built classifiers aimed at detecting misuse such as spam and fraud attempts carried out through the interface.

The release mattered less for its accuracy, which lagged well behind human performance on the same tasks, than for what it signalled about direction: rather than adding narrow, purpose-built integrations for each application, a model could in principle be pointed at any software with a screen and keyboard. It set up a wave of “computer use” and browser-control products from competing labs over the following year, and previewed the interoperability problem — how a model should discover and call the tools available to it — that Anthropic addressed a month later with the Model Context Protocol.