Google publishes RLAIF paper comparing AI-feedback to human-feedback alignment

Google researchers Lee et al. published a paper on Reinforcement Learning from AI Feedback (RLAIF), reporting that reward models trained on preference labels generated by an off-the-shelf large language model — rather than by paid human raters — could match the performance of conventional Reinforcement Learning from Human Feedback (RLHF) on summarisation, helpful dialogue and harmless dialogue tasks.

The paper also introduced a variant it called direct-RLAIF, which skipped training a separate reward model altogether and instead obtained reward signals directly from the labelling LLM during reinforcement learning; the authors reported this outperformed the standard RLAIF approach. Notably, RLAIF performed comparably to RLHF even when the AI labeller and the model being trained were the same size or, in some settings, identical checkpoints — evidence that a model’s judgements about output quality could be useful for improving itself, and not merely be a cheap substitute for a more capable rater.

RLHF’s dependence on large volumes of human preference labelling had been one of the more expensive and slower parts of aligning a model to be helpful and safe; substituting AI-generated labels offered a route to scaling that process without a proportional increase in human annotation cost. The paper served as an empirical companion to Anthropic’s 2022 Constitutional AI work, which had also proposed using AI-generated feedback rather than exclusively human raters, and it fed into a broader shift across the industry toward using models to help supervise and align other models, a trend increasingly relevant as human-generated feedback data became more expensive to source at frontier scale.