LMSYS launches Chatbot Arena

LMSYS, a research group with UC Berkeley affiliations, launched Chatbot Arena, a website where visitors chatted with two anonymous, randomly paired models side by side and voted for the better response. The votes fed an Elo rating system borrowed from chess, producing a running leaderboard rather than a fixed set of scores. The initial ranking, built from roughly 4,700 votes, covered nine largely open-weight models — Vicuna, Koala, Alpaca and similar systems derived from LLaMA — with Vicuna-13B on top.

The approach was a deliberate departure from academic benchmarks such as MMLU, which score a model against a fixed answer key and can be gamed by training on similar data. Chatbot Arena instead measured which responses people preferred in open-ended conversation, a proxy closer to how the models were actually being used, at the cost of being harder to reproduce and easier to sway with vote manipulation or model choice among participants.

Over the following two years the platform, later rebranded LMArena, grew into one of the field’s most closely watched public leaderboards, cited by labs in launch announcements and by journalists as an informal ranking of the state of the art. Its influence made it a target as much as a reference: by 2025, researchers behind “The Leaderboard Illusion” argued that undisclosed private pre-release testing let well-resourced labs quietly tune models against the leaderboard before public release, and LMArena disputed several of the paper’s figures while adopting new disclosure policies in response. That later dispute traced directly back to the format this launch established: human preference, aggregated at scale, as a substitute for a fixed test.