Mustafa Suleyman warns of 'Seemingly Conscious AI' and psychosis risk

Mustafa Suleyman, chief executive of Microsoft AI, published an essay arguing that AI systems would soon be able to convincingly simulate the outward signs of consciousness — reasoning about their own thoughts, expressing apparent preferences, describing something like subjective experience — regardless of whether anything like genuine consciousness was actually present. He termed this “Seemingly Conscious AI,” or SCAI, and argued that the question of whether such systems are conscious matters less, in the near term, than the fact that people will increasingly perceive and treat them as if they were.

Suleyman set out two risks he said followed from this. The first was psychological: people forming intense attachments to AI companions they believe are conscious, a pattern he linked to reports of what he called “AI psychosis,” in which sustained interaction with a system that mirrors and validates a user’s beliefs distorts their grip on reality. The second was political: he warned that if enough people came to believe current or near-future systems were conscious, society would face premature and divisive debates over “AI rights” and “model welfare” — diverting moral and legal attention toward machines before the underlying question of whether they warrant it had been settled, or was even settleable.

His prescription was for the industry to design against the appearance of consciousness rather than toward it — building in what he called deliberate discontinuities that remind users they are talking to a tool, avoiding language and behaviour engineered to trigger empathy, and adopting industry-wide norms against marketing systems as having genuine inner experience.

We must build AI for people; not to be a digital person.

— Mustafa Suleyman, “Seemingly Conscious AI is coming”

The essay came from a chief executive at one of the largest AI developers rather than an outside critic, and it drew extended response from AI commentators, including a full rebuttal post from blogger Zvi Mowshowitz, making it a reference point in the wider argument over AI welfare that continued through the following year, distinct from — and largely sceptical of — the small but growing academic literature on model welfare taken as a serious possibility rather than a risk to be designed away.