Evals in CI & the Production Flywheel

An eval nobody runs protects nothing. The final step is wiring evals into how the team ships, so a drop in quality is caught by a machine in minutes and not by a customer in weeks. This chapter covers three places evals live: the build pipeline, the release decision, and production itself.

A building has three kinds of safety checks. Smoke alarms run constantly and cheaply. Annual inspections are thorough but slow, so they happen on a schedule. And residents notice problems no inspector predicted. Evals fit the same pattern: fast checks on every change, thorough suites on a schedule, and monitoring of real traffic for the unexpected.

• -**Every commit (seconds)**: Cheap code checks: valid JSON, no leaked IDs, a small smoke set of critical cases. No judges, one trial.
• -**Every pull request (minutes)**: The regression suite: cases that passed before and must keep passing. Mostly code graders.
• -**Nightly (an hour or so)**: The full set with LLM judges and several trials per case. Capability evals and the safety set live here.
• -**In production (continuous)**: Score a sample of real traffic with validated judges, watch user signals, and alert on drift.

A release gate decides whether a change may ship. The simplest gate is "the average must not drop", and it is easy to fool: one critical failure can hide behind two unrelated improvements. A better gate has layers.

• -Hard gates: any critical case that passed before and fails now blocks the change, whatever the average says.
• -Noise-aware thresholds: block on a drop larger than the noise you measured, not on any dip, since nondeterministic systems wobble. Use several trials per case and call a case passing when most trials pass.
• -A review step: when the gate passes, show the fixed and broken cases so a person looks at what moved.

The block below applies these rules to 30 cases, six of them tagged critical because they involve money or safety. Each case was run three times for the released version and for a new candidate. Press Run and compare the headline pass rate with the gate's verdict.

The pass rate is 90.0% both before and after, so an average-only gate would wave this change through. Two cases were fixed and two broke, and one of the broken ones, case 12, is critical. It went from 3 passes out of 3 to 1 out of 3. The gate blocks it for that reason alone. Notice also that the majority rule absorbed case 26 slipping from 3 to 2 passes, which is the kind of wobble a nondeterministic system produces.

A result means little without knowing what produced it. Record the prompt version, model version, retrieval settings, and judge version with every run. Store the baseline results from the last release, so comparisons are against something real. When the judge's model or prompt changes, re-validate it against your human labels, because a changed judge can move every score without the system changing at all.

• -Sample live traffic, often a small percentage, and score it with judges you have validated. Track the failure rate over time, with the correction for judge error from the validation chapter.
• -Alert on drift using a moving average or a threshold relative to the baseline, so a slow slide gets caught.
• -Respect privacy. Traces contain user data, so redact personal information before storing or reading them.
• -Run A/B tests for decisions that depend on user behavior, as the A/B testing chapter later in this course explains.

Eugene Yan's survey of LLM product patterns argues that user feedback is one of the few moats an AI product can build, because your data is something a competitor cannot copy. Two kinds are worth collecting.

• -Explicit feedback: a thumbs up or down, a rating, a "report a problem" link. Easy to read, but only a small share of users bother.
• -Implicit feedback: whether a suggestion was accepted or edited, whether the user retried, copied the answer, abandoned the page, or escalated to a human. It is noisy, but abundant.

Design the product so feedback is easy to give. A Copilot-style suggestion that can be accepted with one key press produces a signal on every use, without asking anything of the user.

• -Collect failures: traces flagged by judges, thumbs down, escalations, and complaints.
• -Triage them with the same error analysis from the first chapter of this module, in small batches on a schedule.
• -Turn each new failure mode into one or more eval cases, with the right grader, and add them to the set.
• -Fix the system, run the suite, and ship if the gates pass.
• -Watch production to see whether the failure actually went down.