DeepMind's Sparrow explores rule-based RLHF for safer dialogue

DeepMind published Sparrow, a research dialogue agent trained to be, in the company’s phrase, “helpful, correct, and harmless,” combining reinforcement learning from human feedback with an explicit, written set of behavioural rules. Building safer dialogue agents described the mechanism: two separate reward models, one learned from people ranking which of two answers they preferred, the other trained by paying participants to try to provoke Sparrow into breaking an initial rule set that prohibited things such as giving harmful advice, making threatening statements, or claiming to be human. Sparrow could also query Google Search to ground its answers in retrieved evidence rather than generating claims purely from what it had memorised during training.

DeepMind reported that under adversarial probing, Sparrow broke its own rules in around 8% of attempts, roughly a third of the violation rate of a baseline dialogue model tested the same way, and that it gave a plausible, evidence-supported answer to factual questions 78% of the time. The team was explicit about limitations: Sparrow still fabricated facts and produced off-topic responses, and DeepMind argued that a durable rule set would eventually need to be drawn up with broader public and expert input rather than set by researchers alone.

Sparrow was never released as a product. Its significance was methodological: alongside OpenAI’s InstructGPT work from earlier the same year, it was one of the clearest public demonstrations that reinforcement learning from human feedback could be steered by explicit, written rules rather than implicit preference data alone — an approach Anthropic pushed considerably further three months later with Constitutional AI, which replaced much of Sparrow’s human rule-enforcement with a model critiquing and revising its own outputs.