DeepMind's SIMA agent follows instructions across 3D game worlds

Google DeepMind published SIMA (Scalable Instructable Multiworld Agent), a system trained to follow natural-language instructions across a range of 3D video game worlds using only the same interface a human player has: screen pixels in, keyboard-and-mouse commands out. The agent had no access to a game’s underlying code, state or API — it perceived and acted purely through the audiovisual interface, the same constraint DeepMind’s earlier game-playing systems like AlphaStar had been built to avoid.

SIMA was trained across nine commercial games, including titles from studios such as Hello Games and Coffee Stain, plus four DeepMind research environments, and was evaluated on roughly 600 distinct basic skills spanning navigation, object interaction and menu use. DeepMind reported that training across many games at once produced better generalisation than training a specialist agent per game: an agent trained on eight of the nine games performed close to as well on the ninth, unseen game as a specialist trained on it directly. Instructions were essential to this — without language guidance, agents behaved without clear purpose, according to the company.

DeepMind described the work as early-stage, noting that SIMA’s skills were limited to short, discrete tasks rather than the kind of multi-step planning needed for complex goals. The project sat inside DeepMind’s broader push toward general-purpose, instructable agents rather than game-specific specialists, and a Gemini-powered successor, SIMA 2, followed in November 2025.