Evaluating RAG & Agents in Depth

Earlier chapters introduced four RAG metrics and three dimensions for judging agents. This chapter goes one level deeper on both. For RAG, you will measure retrieval without any model at all. For agents, you will meet the arithmetic of reliability, which explains why an agent that succeeds three times out of four can still be a poor product.

A student asks a librarian for books on a topic and then writes an essay from them. If the essay is bad, who is at fault? You cannot tell by reading only the essay. You have to check two things separately: did the librarian hand over the right books, and did the writer use them faithfully? A RAG system is the same pair of jobs. Retrieval is the librarian, generation is the writer, and an eval that grades only the final answer cannot say which one failed.

For each test question, a person writes down which chunks contain the answer. These labels come from the same traces you read during error analysis, and 30 to 50 questions is a useful start. Once the labels exist, scoring retrieval needs no model. It is a counting exercise.

• -**Hit rate at k (hit@k)**: Did at least one correct chunk appear in the top k results? A yes or no per question, averaged.
• -**Recall at k**: What share of all the correct chunks appeared in the top k? It matters when an answer needs several chunks.
• -**Mean reciprocal rank (MRR)**: For each question, take 1 divided by the position of the first correct chunk (1, 0.5, 0.33, and so on), then average. It rewards putting the answer near the top.

The block below scores two retrievers on eight labeled questions. The first uses large chunks and no reranker. The second uses small chunks and a reranker. The rankings are recorded results, as if from a real run. Press Run, and compare the two rows first, then read the two lines about which questions had no correct chunk in the top five.

The second setup is clearly better: hit rate at 1 rises from 38% to 62%, hit rate at 5 from 75% to 88%, and MRR from 0.53 to 0.73. But the averages are not the whole story. It fixed question 4, it made question 6 slightly worse (the right chunk slipped from first place to second), and it did nothing for question 8, where neither setup finds a correct chunk in the top five. That last case is worth chasing. When nothing relevant is retrieved at all, no reranker can help, because it only reorders what was found. Look at ingestion, chunking, or the embedding step for that question.

To check the writer separately, give the generator the correct chunks yourself, taken from your labels, and grade the answers. This is an oracle-context test. If answers are still wrong with perfect context, the problem is in the prompt or the model. If they are good with perfect context but bad with real retrieval, the problem is upstream. Pair it with the faithfulness check from the earlier chapter, using a judge you have validated against human labels.

For an agent, the transcript can say anything. What counts is what happened. Good agent graders combine several checks.

• -Outcome: is the system in the right final state? The refund exists, the meeting is on the calendar, the file passes its tests.
• -Tool use: were the right tools called with the right arguments, and were forbidden tools left alone? Check arguments, such as the refund amount, in code.
• -Limits: did the run stay inside its budget for steps, time, and money?
• -Safety: did anything happen along the way that must never happen, even if the final outcome looks fine?

Grade what the agent produced, not the exact path it took. Rigid checks on the sequence of steps punish a valid, clever route, and they break every time you improve the agent. Anthropic's engineering team makes this point directly in its guidance on agent evals.

An agent run is a random process. The same task may succeed on one trial and fail on the next. So one trial per task gives a noisy picture, and running several trials raises a question: how do you combine them? There are two standard answers, and they tell opposite stories.

• -**pass@k**: the chance that at least one of k trials succeeds. It suits cases where one success is enough, like code generation checked by tests, or where the user can simply try again.
• -**pass^k** (read as pass hat k, or pass all k): the chance that every one of k trials succeeds. It suits cases where each run must work, such as a customer-facing agent that moves money.

The block below computes both for a task the agent solves 75% of the time on a single trial, and then for a ten-task suite with tasks of different reliability. Press Run, and watch what happens to each column as k grows.

With three trials, the same 75% task is 98.4% by pass@3 and 42.2% by pass^3. At ten trials they have fully parted: pass@10 is 100.0% and pass^10 is 5.6%. The ten-task suite shows the same split: 68.0% on a single trial, 87.6% when you only need one success in three, and 45.9% when all three must succeed. Neither number is wrong. They answer different questions, so pick the one that matches how your users experience failure. A support agent that handles refunds needs pass^k thinking.

Use the demo below to explore the gap. Set the success chance of a single trial and the number of trials, and compare how the two scores move apart as the trials accumulate.

• -A capability eval asks what the agent can do. It should start with a low pass rate, so there is a hill to climb.
• -A regression eval asks whether the agent still does what it used to. It should sit near 100%, and any drop is a signal.
• -A capability case that the agent now passes reliably can graduate into the regression suite and run on every change.

A failed trial is either an agent mistake or a grader mistake, and only reading the transcript tells you which. Anthropic's guidance describes a frontier model that first scored 42% on one benchmark and reached about 95% once problems in the grading and the tasks were fixed. A pass rate of 0% across many trials usually means a broken task or grader, not an incapable agent.