DeepMind's RT-2 lets robots follow natural-language instructions

Google DeepMind announced RT-2 (Robotic Transformer 2), a vision-language-action model built by adapting two existing vision-language models, PaLI-X and PaLM-E, to control a robot arm. The core technique represented robot actions — position changes, rotations, gripper open/close commands — as strings of text tokens, the same representation the underlying language model already used for words, so that a single model trained jointly on web-scale image-and-text data and on robot demonstration data could output either language or physical actions from the same architecture. RT-2 could also produce intermediate chain-of-thought text, describing a plan in natural language before executing the corresponding actions, to handle instructions requiring several steps.

DeepMind reported that folding in web-scale vision-language pretraining substantially improved RT-2’s ability to generalise beyond the specific objects, environments and instructions it had seen during robot-specific training — the central weakness of its 2022 predecessor, RT-1, which had scored 97% on more than 700 trained tasks it had seen 130,000 demonstrations of, but generalised far less well beyond them. On situations absent from the robot training data, DeepMind said RT-2’s success rate rose from 32% for RT-1 to 62% for RT-2, and that emergent capabilities such as recognising symbols, performing basic reasoning, and identifying people improved more than threefold. On the Language Table benchmark, RT-2 reached a 90% success rate in simulation against 74% for RT-1.

RT-2 extended the same underlying premise as DeepMind’s RoboCat, announced the previous month: that robotics could borrow the pretraining-and-generalisation recipe that had worked for language and vision models, rather than relying solely on data collected directly from robot demonstrations, which remained far scarcer and more expensive to gather than internet text and images.