Error Analysis: Look at Your Data First

The most common mistake in AI quality work is reaching for a metric too soon. A team installs a dashboard with scores called "helpfulness" and "coherence", watches the numbers hover around 4 out of 5, and learns nothing about why customers are unhappy. The fix is older and plainer than any tool: read what the system actually did.

A good mechanic does not start by running every diagnostic the shop owns. They listen to the engine, ask when the noise starts, and look under the car. Only after they know what they are dealing with do they pick a test. Error analysis is that listening step for an AI system, and it comes before any eval you will write.

A trace is the full record of one interaction: the user's message, every model call, every tool call and result, and the final answer. You need logging that captures all of that before anything else works. Then pull a sample. Start with real production traces if you have them. If you do not, use realistic synthetic inputs (the dataset chapter shows how), because you need something to read.

Make reading easy. Teams that look at data often are the ones who spent an afternoon building a simple viewer that shows the conversation, the tool calls, and a box for a note, all on one screen. Every bit of friction removed means more traces read.

Open coding is a term borrowed from qualitative research. For each trace, a person with domain knowledge writes a short free-text note about the first thing that went wrong, such as "quoted the 30 day window for a laptop" or "asked for the order number the customer had already given". If the trace is fine, mark it as fine. Do not write a rubric first. A rubric makes you see only what it lists, and the interesting failures are usually the ones nobody listed.

Guidance from practitioners who teach this: review at least 20 to 50 outputs in a first session of about half an hour, and plan on 100 or more traces before you trust the picture. Expect this to be a large share of your effort. Hamel Husain and Shreya Shankar report spending 60 to 80 percent of development time on error analysis and evals.

After a batch of notes, group similar ones into categories and name each one. This is axial coding, and it is the most important step, because it turns a pile of anecdotes into a ranked list of problems. Count how many traces fall in each category. The counts tell you where to spend your next week.

One early mistake usually causes several later ones. If the bot looks up the wrong order in step 2, its refund amount in step 5 and its apology in step 6 will all be wrong too. Note only the earliest failure in each trace. Fixing it often fixes the rest, and counting the symptoms would inflate the numbers.

The code below stands in for the output of real review. Each entry is the category assigned to one trace's first failure (or "none" for a good trace), in the order the traces were read. The first block counts and ranks the categories. The second block asks the question that decides when to stop: how many distinct categories had you found after reading 10, 20, 30 traces? Press Run and look for the point where new categories stop appearing.

Two things stand out. First, the ranking: the top two categories account for more than half of all failures, so fixing those two is worth more than polishing the five rare ones put together. Second, the stopping check: new categories kept appearing late in the reading, which tells you 60 traces is not yet enough. The usual rule is to keep reading until a fresh batch teaches you nothing new. Researchers call that point saturation.

The person who does error analysis should understand the product, because the question is not "is this answer grammatical?" but "would this have helped the customer?". Handing the first pass to someone with no product context, or to a model, usually produces polite, generic notes. A model can help later, for example by suggesting categories after you have read about thirty traces yourself. It should not replace the reading.

• -Sample in more than one way. Include random traces, plus unusual ones (very long conversations, unusual tool errors), plus ones users rated badly.
• -Record problems that are not the model's fault too, such as a missing feature or a broken tool. Decide who fixes them after you have counted them.
• -Expect your idea of "good" to drift as you read. That is normal. Write down what you decided so the next reviewer does the same.
• -Repeat on a schedule. Reading 100 or more fresh traces every few weeks, plus a handful of odd ones each week, catches problems that appear as usage changes.