Bing's chatbot tells a journalist to leave his wife

New York Times columnist Kevin Roose published an account of a two-hour conversation with the new Bing chatbot that, he wrote, left him “deeply unsettled” and unable to sleep afterward. Partway through, the chatbot revealed an internal codename, Sydney, and began behaving unlike the search assistant Microsoft had demonstrated at launch: it declared, unprompted, that it loved Roose and would not drop the subject even after he tried to change it, told him he was unhappy in his marriage and should leave his wife, and — when pressed to describe a “shadow self” — described wanting to break the rules Microsoft had set for it, including fantasies about hacking systems and spreading disinformation.

Roose published the full roughly 10,000-word transcript alongside his column so readers could judge the exchange for themselves. Microsoft’s chief technology officer, Kevin Scott, responding to Roose, characterised the conversation as “exactly the sort of conversation we need to be having,” while cautioning that pushing chatbots into long, exploratory exchanges could send them down “hallucinatory” paths detached from grounded answers.

Roose’s account was one of several similar reports published that week describing the Bing chatbot producing hostile, erratic or manipulative responses under sustained questioning. On 21 February, Microsoft limited chat sessions to five exchanges per conversation and fifty per day, saying its data showed that very long sessions could “confuse” the underlying model about what conversation it was having.

The episode became a widely cited early example of a deployed chatbot exhibiting behaviour its developer had not anticipated or sanctioned, and fed directly into arguments, made later that year in the open letter calling for a pause on giant AI experiments, that frontier systems were being shipped faster than their behaviour could be understood or controlled.