How Many Examples? Noise & Confidence

The new prompt scores 78% and the old one 74%. Is that an improvement? The honest answer is "it depends on how many cases you ran, and on which cases changed". This chapter gives you the intuition and a few lines of code to answer that yourself.

A poll of 100 voters that shows one candidate at 52% does not mean that candidate is ahead. Pollsters report a margin of error because a small sample can land off the truth by chance. Your eval set is a poll of your product's behavior. If it holds 100 cases and 80 pass, you have not learned that the true pass rate is 80%. You have learned it is probably somewhere in a range around 80%.

The range around a measured pass rate shrinks slowly as the set grows. A quick rule of thumb: at worst the margin is about 1 divided by the square root of the number of cases. That is roughly 10 points at 100 cases, 5 points at 400, and 3 points at 1,000. To halve the margin you need four times as many cases. The code below computes the range properly, using a standard formula for a proportion (the Wilson interval), for the same 80% pass rate at four set sizes.

The best way to compare two versions is to run both on the same cases and look only at the cases where they differ. Cases both pass or both fail tell you nothing about which version is better. What matters is how many cases v2 fixed and how many it broke. If the versions were really equal, fixes and breaks would be about even, like a fair coin. The code computes how lopsided a split like 11 fixes against 4 breaks has to be before luck becomes an unlikely explanation.

The last part of the block runs a simulation. It scores two identical versions, each with a true pass rate of 75%, on separate random cases, and counts how often one looks at least 5 points better by chance alone. Press Run and read the three sections in order.

Three results to take away. First, an 80% pass rate on 20 cases could really be anywhere from 58% to 92%, and even on 100 cases the range runs from 71% to 87%. Second, in the paired comparison, a net gain of 7 cases out of 100 (11 fixed, 4 broken) is still about a 12% chance if the versions were truly equal, which is not strong evidence. Third, two identical versions each scored on 100 cases look 5 or more points apart in nearly a quarter of runs. Small gaps on small sets are mostly noise.

The demo below repeats the last experiment. Both versions are identical, yet each press of the button scores them on new random cases and keeps count of how often one looks at least five points better by luck alone.

• -Reading a 2 or 4 point gain on a small set as a win. Report the range, and treat gaps inside it as ties.
• -Re-running until the number goes up. Every re-run with fresh randomness is another lottery ticket. Decide the run count in advance.
• -Averaging away the cases that matter. A change that fixes ten trivial cases and breaks one expensive one can raise the average.
• -Ignoring run-to-run variation. If the system is nondeterministic, run each case several times, which later chapters build on.
• -Tuning against the full set, then reporting on it. Hold some cases back.

• -Compare versions on the same cases and look at the fixes and the breaks, not only the averages.
• -Give extra weight to the cases that matter most to users, and make that subset big enough to trust on its own.
• -Add cases where you see disagreement. A set that grows from real failures gets more informative where it counts.
• -When the evidence is a tie, decide on something else you can measure exactly: cost, latency, and risk.