Building Eval Datasets

An eval is only as good as the cases inside it. A hundred easy, repetitive cases give you a comfortable number and no information. A few dozen well-chosen ones can protect a product for months. This chapter is about choosing them.

Imagine stocking a test kitchen to judge a new chef. If you only supply eggs and bread, you learn how they cook eggs and bread. A fair test includes the awkward ingredients: a tough cut of meat, a customer with an allergy, a rush of ten orders at once. An eval dataset is that pantry. It should hold the situations that are hard, rare, or expensive when they go wrong.

• -**Real failures**: Traces you read during error analysis, user complaints, support tickets, incident reports. Each one is a case that actually happened.
• -**Pre-release checks**: What your team already tries by hand before shipping. Writing those down is a free first version of the set.
• -**Production samples**: A random slice of real traffic, so the set reflects how people really talk, not how you imagine they do.
• -**Synthetic cases**: Generated inputs that fill gaps, such as rare combinations. Useful, but weaker than real data for judging how often a failure occurs.

Start small. Anthropic's guidance for agent evals suggests 20 to 50 simple tasks drawn from real failures is a good first suite. It is far better to have 30 cases you trust than 3,000 you never looked at.

A good case has a clear definition of success, and two domain experts looking at it independently would reach the same verdict. If experts disagree about whether an answer is right, the problem is the case, not the system. Where you can, include a reference solution that you know passes, so you can confirm the grader and the case are not broken.

If every case tests that the bot should help, the cheapest way to score 100% is a bot that answers everything, including requests it should refuse. Include cases where the right behavior is to decline, ask a clarifying question, or hand off to a human. Test both when a behavior should happen and when it should not, or you will tune the system in one direction forever.

When real data is thin, you can generate cases, but an open request like "give me 50 test questions" produces generic, repetitive output. A better method is to list the dimensions along which real inputs differ, pick values for each, and combine them. The model's job is then only to phrase each combination naturally.

• -List the dimensions that matter, such as the intent of the request, the customer's mood, and whether the key detail was provided.
• -Write down the possible values for each dimension.
• -Combine them, and remove combinations that cannot happen.
• -Turn each combination into a realistic message, by hand or with a model in a separate step.
• -Run the messages through the real system and read the traces, exactly as in error analysis.

The code below does steps 1 to 3 and a simple phrasing step with templates. Press Run and check three things: how many combinations there are, how many are left after removing impossible ones, and what share of cases expect the bot to decline.

Of the 22 valid cases, 16 expect an answer from policy, 4 expect the bot to ask for the missing order number, and 2 expect it to decline. That mix is deliberate. A bot that never asks and never declines would still pass 16 of the 22 cases and fail the other 6, which is exactly the signal you want. Notice what this method cannot do: it tells you the set covers the combinations, but not how often each one happens in real life. For frequency, only real traffic will do.

• -Add every new real failure as a case. This is how the set stays tied to reality.
• -Keep a held-out slice you never tune against, so you can tell improvement from memorizing the test.
• -Watch for saturation. When a suite hits 100% it stops showing progress, so retire easy cases or add harder ones.
• -Give the set an owner, and let domain experts and product people contribute. Treat it like code: reviewed, versioned, and maintained.