Why AI Systems Need Evals

A team ships a support bot after trying about a dozen questions by hand. Two weeks later someone edits the prompt to fix one angry customer's complaint. The complaint is fixed. Three other behaviors quietly break, and nobody notices for days, because nothing in the system measures those three behaviors. This chapter is about the missing piece: an eval.

A new driver can feel confident after a few easy trips around a quiet block. A driving test is different. It has fixed routes, a list of specific skills (parallel parking, merging, an emergency stop), a scoring sheet, and the same standard for every candidate. Passing says more than feeling confident ever could.

An eval is the driving test for an AI feature. Mapping the pieces:

• -The fixed routes are the test cases: real situations the system must handle, chosen on purpose.
• -The scoring sheet is the grader: a rule that decides pass or fail for each case.
• -The pass mark is your threshold: how many cases must pass before you ship.
• -Retaking the test after more lessons is re-running the eval after every change to the prompt, model, or code.

A normal unit test says: given this input, the output must equal exactly this value. That works because ordinary code is deterministic. An LLM feature breaks that assumption in three ways.

• -The same question can produce different words on different runs, so there is no single exact string to compare against.
• -Many different wordings are all correct. "Returns are accepted for 30 days" and "You have a month to send it back" mean the same thing.
• -A wrong answer rarely raises an error. It arrives fluent, confident, and well formatted, which is exactly why it gets past a quick look.

Setting the temperature to zero makes outputs more repeatable, but it does not fix the second and third problems. And the ground moves under you: a provider updates a model, or someone edits a prompt, and behavior shifts without a single line of your code changing.

Teams use slightly different words, but one vocabulary has become common, and it is the one used throughout this module. It follows the terms Anthropic's engineering team uses for agent evals.

• -**Task**: one test. It has an input and a definition of success.
• -**Trial**: one attempt at a task. Because outputs vary, you often run several trials of the same task.
• -**Grader**: the logic that scores a trial. A task can have more than one grader.
• -**Transcript** (also called a trace): the full record of a trial, including the output, any tool calls, and the steps in between.
• -**Outcome**: the final state of the world after the trial. Did the refund actually appear in the database, or did the bot only say it did?
• -**Harness**: the code that runs every task, collects the results, and reports them.

• -**Code**: A rule you wrote: exact match, a schema check, a regex, a test run. Fast, cheap, repeatable, but blind to nuance.
• -**Model (LLM as judge)**: Another model scores the output against a rubric. Flexible and scalable, but it can be wrong and must itself be checked.
• -**Human**: A person reads and decides. The gold standard for quality, but slow and expensive, so it is best used to check the other two.

The next chapters teach each kind. For now, one principle: use the cheapest grader that can answer the question. If a rule in code can decide it, do not pay for a judge.

Here is a small but complete eval. A returns-policy assistant has two versions: v1 is the current prompt, and v2 is a tweak that asks for shorter answers. The recorded answers from both are graded by plain code against eight test cases. Press Run, then look at two things: the pass rate of each version, and which individual cases changed.

v2 looks better on average, 75% against 62.5%. If you only read the average, you would ship it. The case-by-case table tells a more useful story: v2 fixed two cases and broke one, and the one it broke is a promise about when customers get their money back. Whether that trade is acceptable is a product decision, but you can only make it if you can see it.

Evals are not one test. They form a ladder, and each rung catches a different kind of problem. This three-level framing follows Hamel Husain's widely read essay on evals.

• -Quick assertions: cheap code checks that run on every change, like "the answer is valid JSON" or "no internal ID leaks into the reply". They catch plain breakage in seconds.
• -Offline evals: a curated set of cases graded by code, models, and sometimes people, run before release and when comparing candidates. They catch quality regressions and tell you which version is better.
• -Online checks: scoring a sample of real traffic, reading user feedback, and A/B tests. They catch what your test set never imagined, because real users are more inventive than any test writer.

The production chapters later in this course cover the top rung in detail, with offline and online evaluation and with A/B testing. This module builds the lower two rungs properly.

Good teams run a loop. They look at real failures, fix the system, add the failure to the eval set as a new case, and re-run everything. Each turn of the loop makes the next change safer, which is why people call it a flywheel.

The trap is writing evals for problems you only imagine. Practitioners differ on how much to write before launch, but a sensible middle path is this: write the cheap, deterministic checks you already know you need, and write the expensive ones (judges, big datasets) only for failures you have actually seen. The next chapter shows how to see them.