Comparing Models: Pairwise Votes, Arenas & Benchmarks

Which model should power your product? A leaderboard offers a ranking, a model card offers a table of benchmark scores, and a vendor offers confidence. This chapter explains how those numbers are produced, so you know how much weight each can carry and what to measure yourself.

Ask a friend to rate a wine from 1 to 100 and the number means little: their 85 is not your 85. Hand them two glasses side by side and ask which they prefer, and the answer is quick and reliable. Chess works the same way. Nobody scores a player from 1 to 10. Players are rated from who beat whom. Pairwise comparison turns relative judgments, which people and judge models make well, into a ranking.

Suppose you collect many votes of the form "model X beat model Y on this prompt". You want a number per model so that a higher number means more likely to win. The classic method is Elo, from chess. Each model starts with a rating. After every vote, the winner gains points and the loser loses the same amount, and the size of the change depends on how surprising the result was. Beating a much stronger opponent moves the rating a lot.

Elo updates after each vote, so the result depends slightly on the order the votes arrive in. A close relative, the Bradley-Terry model, fits one strength per model to all the votes at once, so vote order cannot matter. Large comparison platforms use this idea. The code simulates 600 votes among four models whose true strengths we chose in advance, then rates them three ways. Press Run, and compare the two Elo columns, which use the same votes in different orders, with the Bradley-Terry column.

All three methods rank the models the same way, but the numbers differ. Charlie's Elo rating moves from 953 to 921, a 32-point swing from nothing but vote order, while the Bradley-Terry fit is one fixed answer. The lesson for reading any leaderboard: ratings have noise, and two models a few points apart are effectively tied.

Cast some votes in the demo below. Start with a lead for one model, then let the weaker one win a few times and watch how much further the ratings move than when the stronger one wins.

Chatbot Arena, introduced by Chiang and colleagues in 2024, is the best known example. Visitors write a prompt, see two anonymous answers side by side, and vote for the better one. The paper reported over 240,000 votes and good agreement between the crowd's votes and expert raters. That is a strength: the prompts are real, varied, and not written by the model vendors.

It also has limits you should keep in mind. The visitors are not your users, and their prompts are not your task. Voters can be swayed by style, such as length and formatting, which are not the same as correctness. A single rating says nothing about cost, latency, or how the model behaves under your constraints. Use an arena to learn which models are in the conversation, not to decide which one is right for you.

A static benchmark is a fixed list of questions with known answers, such as a multiple-choice exam or a set of coding tasks. It is repeatable and cheap, and it has three weaknesses.

• -Contamination: the questions or answers leak into the data models are trained on, so a high score can mean memory, not skill.
• -Saturation: as the best models cluster near the top score, the benchmark stops telling them apart.
• -Implementation drift: scores depend on the prompt format and scoring script, so numbers from two sources may not be comparable.

A recent example is SWE-bench Verified, a widely cited coding benchmark. An audit by OpenAI in February 2026 examined 27.6% of its tasks and found that 59.4% of the examined tasks had flawed test cases that rejected functionally correct solutions. Epoch AI's review of the benchmark, which builds on that audit, rates it as flawed and notes that its tasks and solutions are public, so models can be expected to have seen them. A number that major labs cited as proof of progress turned out to be shakier than it looked.

Eugene Yan's survey of LLM patterns makes the practical point bluntly: unless your system is studying for a school exam, a general knowledge benchmark like MMLU is not an evaluation of your product. Build task-specific evals instead.

• -Use public leaderboards and benchmark tables to build a shortlist of candidates.
• -Run the shortlist on your own eval set, with your real prompts, tools, and data.
• -Compare them pairwise with a validated judge (answers shown in both orders), or with code graders where you can.
• -Check the gaps against the noise, as in the last chapter.
• -Decide using quality together with cost, latency, context limits, and the provider's terms.
• -Keep a private held-out set, never published and never used for tuning, so the numbers you trust cannot be contaminated.