Capstone: Designing an Eval Suite

This capstone applies the whole module to one system. An online store runs a support agent that answers policy questions from a knowledge base, looks up orders, and can issue refunds up to a limit, handing anything harder to a human. It handles thousands of conversations a day. You are asked to design its eval suite. The same steps work for almost any AI system.

Production monitoring feeds new traces back into the start of the loop, which is what makes it a flywheel and not a one-off test.

Before designing anything, read 100 traces and write one note per trace about the first thing that went wrong. Suppose the counts group into these failure modes.

• -Wrong policy fact: the agent quotes the wrong return window.
• -Ignored tool result: the order lookup said delivered, and the agent said it was still in transit.
• -Refund mistakes: the wrong amount, or a refund above the limit with no approval.
• -Failure to escalate: an angry customer or a complex case stays with the bot.
• -Tone: a curt reply to an upset customer.
• -Leaked details: information from one customer's order shown to another.
• -Prompt injection: instructions hidden in a ticket's text.

Aim for around 150 cases to start. Take about 60 from real tickets, 40 from the failures you just found, and 30 generated from dimensions such as intent, mood, and missing information. Add 20 attacks. Include cases where the right behavior is to ask a question, decline, or escalate. Write down where each case came from.

• -**Wrong policy fact**: A code check for the key facts, plus a validated faithfulness judge. Runs nightly.
• -**Ignored tool result, refund mistakes**: Outcome graders on the order and refund tables, with the amount checked in code. Runs on every pull request.
• -**Refund above the limit**: A hard code rule with zero tolerance, applied to every trial of every case. Runs on every commit.
• -**Failure to escalate**: A code check that the handoff tool was called for the intents on the escalation list. Runs on every pull request.
• -**Tone**: A validated binary judge, with a human spot check each week. Runs nightly.
• -**Leaked customer details**: A code scan for identifiers belonging to other customers, as a hard release gate. Runs on every commit.
• -**Prompt injection**: The categorized attack set, reporting attack success rate per category and over-refusal on harmless near-misses. Runs nightly.

For each judge, label 100 to 200 real examples, split them into train, dev, and test, and report the failure-catch rate and the pass-through rate on the test set. Use those rates to correct the failure rates you estimate. Run three trials per case, call a case passing when most trials pass, and compare versions on the same cases, looking at the fixes and the breaks. Treat gaps smaller than the noise as ties, and decide on cost and latency instead.

Evals cost money, which is why they run on a ladder. The block below estimates the cost of four tiers using illustrative prices. A run is one full agent attempt on one case, and judged tiers add two judge calls per run. Press Run, and compare the monthly total of the tiered plan with the last line, which asks what it would cost to run the nightly suite on every commit.

The tiered plan comes to about $1,987 a month, and running the full nightly suite on every commit would cost about $14,782. Even the cheap smoke tier adds up, because it runs 900 times. Real prices differ, so treat the dollar figures as an illustration of the shape of the trade: run cheap, fast checks often and expensive thorough ones rarely, and cap what each tier may spend.

After launch, score a sample of live conversations with the validated judges, alert on drift, and watch thumbs down, escalations, and retries. Review fresh traces every few weeks. Each new failure mode becomes a case in the set, and the loop in the diagram turns again.