LLM-as-Judge, Done Right

Some qualities no rule can check: whether an explanation is clear, whether a summary stays faithful to its source, whether a reply is respectful to an upset customer. A person could judge these, but a person cannot read ten thousand outputs after every change. An LLM judge is a model that reads an output and scores it against criteria you wrote. It is powerful and easy to misuse, so this chapter is about using it carefully.

A football referee does not decide whether the match was "good". They apply specific rules to specific events: was that a foul, was the ball over the line. Referees still make mistakes, so leagues added video review and track how often decisions are overturned. An LLM judge is a referee. To work, it needs three things.

• -Clear rules: a narrow question with a definition of pass and fail, not "is this answer good?".
• -A clear view: only the information it needs to decide, such as the policy text and the answer.
• -A video review: a check of its decisions against human decisions, which the next chapter covers in detail.

• -Single output, pass or fail: "does this answer invent a policy that is not in the document?" The most reliable form, and the one to start with.
• -Single output, scored against a rubric: "rate clarity from 1 to 5". Flexible, but scores drift and are hard to compare, so use sparingly.
• -Pairwise: "which of these two answers is better?" Useful for comparing two versions or two models, and the basis of arenas, covered later in this module.

Most judge failures come from a vague prompt. These habits come from practitioners who run evals for a living, including Hamel Husain and Shreya Shankar and Anthropic's engineering team.

• -One judge per failure mode. A single judge asked to check tone, accuracy, and safety does all three poorly. Build separate, small judges.
• -Binary verdicts. Pass or fail forces a decision. A 1 to 5 scale invites the middle and means different things to different runs.
• -Define the verdicts in words, and add a few real examples of each, taken from labeled data. A common approach, called critique shadowing, has a domain expert mark pass or fail and write a short critique, then uses those examples to build and test the judge.
• -Give a way out. An UNSURE option stops the judge from guessing, and unsure verdicts can be sent to a person.
• -Ask for the reason first, then the verdict, in a structured format your code can parse.
• -Show only what is needed. Extra context can distract the judge. If you are unsure whether a piece of context helps, test with and without it.

The prompt below follows all of these habits. Read it as a template, not a recipe. The wording of your definitions matters more than any trick of format.

A 2023 study by Zheng and colleagues, which introduced the MT-Bench benchmark and the Chatbot Arena comparison, is the standard reference on how judges go wrong. It found that a strong judge, GPT-4, agreed with human raters more than 80% of the time, about as often as two humans agree with each other. It also catalogued systematic biases:

• -Position bias: the judge favors an answer because of where it appears, often the first one. In that study, with the default prompt, one judge gave the same verdict after the two answers were swapped only 23.8% of the time, while GPT-4 did so 65% of the time. Models have changed since 2023, so measure your own judge rather than assuming.
• -Verbosity bias: longer answers win, even when the extra words add nothing.
• -Self-enhancement bias: a model tends to prefer text that resembles its own style, so using one model to grade itself can flatter it.

• -**Position bias**: Run each pair twice with the order swapped. Count a win only when the same answer wins both times, and treat the rest as ties.
• -**Verbosity bias**: Say in the prompt that length is not quality, compare answers of similar length, or include length as a check of its own.
• -**Self-enhancement**: When comparing models, avoid using one of the contenders as the only judge. Use another model, or several judges that must agree.

The code below is a toy: a simulated judge that usually prefers the better answer but leans toward whichever it sees first. It judges 278 pairs with a clear winner, each twice with the order swapped. Press Run and compare three rows with different amounts of bias. Look at two columns: how often the verdict survives the swap, and how accurate the judge is on the pairs where it does.

As the bias grows from 0 to 2.0, accuracy from a single order falls from 88.5% to 80.6%, and the share of verdicts that survive the swap falls from 80.9% to 59.4%. The useful finding is in the last column: where the verdict does survive the swap, the judge is right 97% to 99% of the time. Disagreement under a swap is a cheap warning sign. Count those pairs as ties, or send them to a person, instead of trusting a coin flip.

Advice differs, and it is worth knowing both views. Some practitioners recommend starting with the strongest model you can afford, to learn whether the task is judgeable at all, then testing cheaper models against it. Some warn against grading a model with itself. Others find that for a narrow, binary check the same model works fine. You do not need to settle the argument. Whichever model you pick, measure its agreement with human labels, which is the subject of the next chapter.