Online Experiments for ML Systems

Offline metrics tell you which model is worth testing. Only a live experiment tells you whether it helps real people. The earlier chapter on A/B testing prompts covered sample size and significance. This chapter covers what is specific to ML systems: the checks that tell you whether an experiment can be trusted, and the metrics that stop a good-looking launch from doing quiet damage.

A drug trial randomizes patients into groups, watches for side effects as well as benefits, checks that the groups really were comparable at the start, and follows patients long enough to see lasting effects. An ML experiment needs the same four habits: random assignment, safety metrics, a fairness check on the split, and enough time.

Pick one primary metric that the change is meant to move, such as watch time or orders. Then list guardrail metrics, the things the change must not hurt, each with a limit decided in advance. A ranking change that raises watch time but also raises unsubscribes, crashes, or latency has not won.

If you intended a 50/50 split, the group sizes should be close to equal. A split that is off by more than chance allows, called a sample ratio mismatch, means something is wrong with assignment or logging: a bug in the hash, bots filtered from one arm only, a page that crashes for treatment users before they are logged. Kohavi, Tang, and Xu, in their book on online experiments, treat this check as a guardrail every experiment should include. When it fails, the results cannot be trusted, whatever they say.

The code runs a chi-square check on three splits and then reads out a launch decision against guardrail limits. The percentages are invented. Press Run and look at which splits pass and which guardrail blocks the launch.

The first split, 50,200 against 49,800, has a p-value of about 0.21, which is ordinary random variation. The other two are only 0.9 and 1.5 percentage points away from the intended 50% share, but with 100,000 users that is far outside what chance produces, with p-values of about 1e-8 and 1e-21. The check uses a strict threshold, 0.001 here, so it rarely raises a false alarm when it runs on every experiment. In the read-out, watch time is up 1.4, which is the goal. Latency and crashes are within their limits. But unsubscribes rose 8.0 against a limit of 5.0, so the launch is blocked. The change made people watch more and, for a group of them, want to leave.

• -Novelty effects: a new ranking excites users for a week, then the effect fades. Run long enough to see it settle, and keep a long-term holdout, a small group that never gets the change.
• -Interference: in a marketplace or social network, users affect each other, so the control group is not untouched. Randomize by cluster or region where this matters.
• -Peeking: checking results every day and stopping when they look good inflates false wins. Decide the duration or use a method built for sequential looks.
• -Mismatched population: a result measured on active users may not hold for new ones. Check segments.
• -Offline wins that vanish: the offline metric was a proxy, and the online metric is the truth. Expect some offline gains not to survive.

Rankers can be compared faster with interleaving. Instead of showing user A the old list and user B the new one, show every user one merged list in which results from both rankers are mixed, and count which ranker's results get clicked more. Because each user judges both, far fewer users are needed. Interleaving is good for choosing between rankers and not a replacement for a full experiment on the business metric.