Skip to content
Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference 4 min

Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios N. Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael I. Jordan, Joseph E. Gonzalez, Ion Stoica · arXiv (UC Berkeley) 2024

Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference

Where leaderboards come from: tens of thousands of people voting between two hidden chatbots. Simulate an arena and see how many votes a ranking needs.

Read the original paper

AI & ML · Yellow Belt · Pro · 4 min read

Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference

How do you decide which chatbot is best? Tests with right answers, such as multiple choice exams, have a flaw: models can memorize them, and most things people ask a chatbot have no single right answer. A better judge might be the people who use them.

A team at UC Berkeley and partners built Chatbot Arena, a free website where anyone can pose a question and get answers from two hidden models, then vote for the one they prefer. The paper explains how it works, what the votes look like, and the statistics that turn hundreds of thousands of small preferences into a trustworthy ranking.

Chatbot Arena shows a user two anonymous answers and asks which is better. By January 2024 it had over 240,000 votes from about 90,000 users. The paper ranks models from these votes with a classic statistical model, adds error bars, picks which pairs to show next more cleverly, and checks that the crowd agrees with experts.

A tournament judged by users

Each visit is a battle. You type any prompt. The site sends it to two models picked at random, and you see both answers side by side without knowing who wrote what. You vote for the better answer, or press tie or both are bad. Only then are the names revealed. Hiding the names reduces brand bias.

How a vote is collected. Because the models are anonymous, a vote is about the answer, not the brand. If you cannot decide, you can keep chatting, or press tie or both are bad.

There is no preset list of prompts, so the questions are whatever people really care about: coding, writing, planning and everyday help. The platform began in April 2023. As of January 2024 it had around 240,000 votes from over 90,000 users in more than 100 languages, covering over 50 models, both closed ones such as GPT-4, Claude and Gemini and open ones.

Keep reading with Pro

This paper is part of Pro. Pro and Architect open the full breakdown, with animated diagrams, interactive elements you can play with and a quiz. A few papers stay free to read in full.

Still to read

3 more sections, 1 more diagram and 2 interactive elements to play with

  • From votes to a ranking
  • Smart pairing and honest checks
  • What to keep in mind

Found this breakdown useful?

Share it with someone else wrestling with this paper.

Discussion0

Join the Discussion

Sign in to leave comments, reply to others, or like insights.

Sign In to ScaleDojo

No comments yet. Be the first to start the thread!

Enjoyed this? Get more like it.

New paper breakdowns, levels, and one concept worth knowing, straight to your inbox.

No spam, ever. Unsubscribe in one click.

Enjoyed this content?

Your support keeps us creating free resources

We put a lot of hours into researching and writing these guides. If it helped you, consider buying us a coffee. Every bit goes toward keeping ScaleDojo's content free and growing.

$

One-time payment via Stripe. ScaleDojo account required.