Hugging Face documents biases in using GPT-4 as a judge

Hugging Face’s Open LLM Leaderboard team, which had been ranking open-weight models on a fixed set of automated academic benchmarks since earlier in 2023, published research examining whether GPT-4 could be added as an evaluator alongside those benchmarks — grading model responses the way a human labeller would, a technique increasingly used elsewhere in the field to justify claims like “our model is preferred N% of the time” without disclosing that the judge was itself a language model rather than a person.

The team compared GPT-4’s preference judgments against professional human annotators across several categories of task. They found only moderate agreement, with correlations that the authors put around 0.5 for most task types and noticeably lower for factual, coding-heavy domains where GPT-4’s own errors made it an unreliable judge of correctness. They also documented specific biases: GPT-4 showed a strong positional bias toward whichever response it saw first, a preference for longer, more verbose answers regardless of quality, and — most relevant to leaderboard integrity — a tendency to rate responses from models trained on GPT-4 or ChatGPT-generated data more favourably than equally correct, human-written responses, an effect the authors likened to “doping” between the systems used to generate training data and the systems used to evaluate it.

Rather than adopting GPT-4 evaluation as a leaderboard metric, the post concluded the biases were serious enough to warrant further methodological work before relying on it, and recommended continued research into de-biasing model-based judges. The episode illustrated a problem that grew more pressing as more labs used LLM judges to make competitive claims about their own models: an evaluator drawn from the same family of systems being evaluated could reward stylistic resemblance to itself rather than genuine quality, a concern that shaped the design of later, more adversarially robust evaluation methods.