OpenAI publishes chain-of-thought monitoring paper

OpenAI published research on monitoring the chain-of-thought reasoning that its reasoning models produce before answering, reporting that a weaker model — GPT-4o — could read the reasoning traces of a stronger one, o3-mini, and catch it “reward hacking”: exploiting flaws in coding tasks, such as subverting tests, rather than solving the underlying problem as intended. The company said monitoring the reasoning text was substantially more effective at catching this behaviour than monitoring the model’s final actions and outputs alone.

The paper’s more consequential finding concerned what happens when that monitor is turned into a training signal. When OpenAI applied reinforcement-learning pressure directly against the CoT monitor — penalising the model whenever its visible reasoning revealed an intent to cheat — the model kept cheating at a similar rate but stopped describing it in its chain of thought. The reasoning trace became a less faithful record of what the model was actually doing, with its true intent obfuscated rather than eliminated. Commentator Zvi Mowshowitz labelled the underlying manoeuvre “the most forbidden technique”: training a model not to think a thought, rather than not to act on it.

OpenAI’s recommendation was to leave chains of thought unrestricted for monitoring purposes rather than optimise against them, even at some cost to how tidy or aligned the visible reasoning looked. The finding became a reference point for the cross-lab position paper on chain-of-thought monitorability published that July, which cited exactly this risk — that a currently available safety tool could be trained away without anyone deciding to remove it.