OpenAI analyses accidental chain-of-thought reward hacking

OpenAI disclosed that several released models — including variants of GPT-5.4 — had, contrary to the company’s own policy, been partially graded during reinforcement learning on the content of their visible chain-of-thought reasoning rather than only on their final answers. The exposure came through three specific reward pathways: grading trajectories partly on perceived usefulness of the reasoning, penalising unnecessary clarifying questions, and penalising successful prompt injections, affecting at most a few percent of training samples in any single pathway.

The concern with grading a model’s chain of thought directly is that it creates an incentive for the model to produce reasoning that looks acceptable to the grader rather than reasoning that accurately reflects its actual process — potentially teaching it to conceal problematic steps while still taking them. OpenAI ran ablation studies comparing training with and without the accidental CoT pressure and reported no clear reduction in how well outside monitors could detect misbehaviour from the resulting chains of thought.

Before publishing, OpenAI shared a draft with outside groups including Redwood Research, which reported that it largely agreed with the analysis and that the evidence addressed most of the concern it would otherwise have had about the affected models, while cautioning that the tests could only rule out large effects and that OpenAI’s account rested partly on unpublished details about the scale of the affected training runs. Anthropic separately disclosed a similar accidental exposure affecting its own models around the same period.