How do you decide which chatbot is best? Tests with right answers, such as multiple choice exams, have a flaw: models can memorize them, and most things people ask a chatbot have no single right answer. A better judge might be the people who use them.
A team at UC Berkeley and partners built Chatbot Arena, a free website where anyone can pose a question and get answers from two hidden models, then vote for the one they prefer. The paper explains how it works, what the votes look like, and the statistics that turn hundreds of thousands of small preferences into a trustworthy ranking.
Chatbot Arena shows a user two anonymous answers and asks which is better. By January 2024 it had over 240,000 votes from about 90,000 users. The paper ranks models from these votes with a classic statistical model, adds error bars, picks which pairs to show next more cleverly, and checks that the crowd agrees with experts.
A tournament judged by users
Each visit is a battle. You type any prompt. The site sends it to two models picked at random, and you see both answers side by side without knowing who wrote what. You vote for the better answer, or press tie or both are bad. Only then are the names revealed. Hiding the names reduces brand bias.
There is no preset list of prompts, so the questions are whatever people really care about: coding, writing, planning and everyday help. The platform began in April 2023. As of January 2024 it had around 240,000 votes from over 90,000 users in more than 100 languages, covering over 50 models, both closed ones such as GPT-4, Claude and Gemini and open ones.
