Yudkowsky publishes 'AGI Ruin: A List of Lethalities'

Eliezer Yudkowsky, co-founder of the Machine Intelligence Research Institute, published “AGI Ruin: A List of Lethalities” on LessWrong, a 43-point argument that the alignment problem was, in his phrase, “unsolved and unsolvable on the first critical try.” The post became the most-cited single statement of the position that advanced AI poses an unmanageable extinction risk, drawing over 1,000 upvotes and hundreds of comments on the site.

Yudkowsky’s core claim was structural rather than technical: humanity would likely get only one attempt at building a system capable of taking dangerous, irreversible actions, because the actions that would let a weaker AGI “buy time” — such as forestalling rival projects — themselves require a system already powerful enough to be dangerous if misaligned. He argued that a system’s problem-solving ability would generalise far beyond its training distribution while its alignment properties would not, that safety techniques validated on weak, low-stakes systems could not be assumed to transfer to a system capable of catastrophic action, and that no current research programme, in his assessment, addressed this gap. He was explicit that he expected the field to fail to solve the problem in time, describing humanity as not on course to do so.

The post drew substantial pushback, including from researchers who considered Yudkowsky’s specific scenarios underspecified or his confidence in doom overstated, and prompted extended point-by-point responses across the rationalist and AI-safety blogosphere. It did not offer a research agenda so much as a diagnosis, and its influence lay less in changing what labs built than in supplying the reference text against which the “doom” position in subsequent AI-risk debate was measured, including Yann LeCun’s public case for a different, non-autoregressive path to machine intelligence, published three weeks later.