Model Serving Basics: Latency & Rollouts

A model that is accurate but slow is a model nobody uses. A recommendation that arrives after the page has rendered is wasted, and a fraud check that takes two seconds blocks the checkout. Serving is where a model meets real traffic, and it has its own engineering problems: speed, reliability, and the ability to change the model safely.

The average wait at airport security might be six minutes, and still some travelers wait forty and miss their flights. They do not care about the average. They remember their own wait. A serving system is judged the same way: not by how long a typical request takes, but by how long the slow ones take.

• -The gateway authenticates the call and routes it.
• -The feature lookup fetches the user's and item's features from the online store, often the slowest step because it talks to other systems.
• -The model runs inference. For a small model this is a few milliseconds, for a large one it needs a GPU or several.
• -Post-processing applies filters, business rules, and formatting.

Teams give each step a share of a total latency budget, for example 50 milliseconds for the whole call, and then watch where the time actually goes.

A percentile says how long the slowest share of requests take. The p50 is the median: half of requests are faster. The p99 is the time that 99 out of 100 requests beat, so it describes the slowest 1%. At 10,000 requests a second, 1% is 100 requests every second, and a few of them will be the ones that matter. That is why teams set targets on p95 or p99, not on the average.

A well-known 2013 paper by Jeffrey Dean and Luiz Andre Barroso, "The Tail at Scale", made the point that in large systems the slow tail dominates, because one user request often fans out into many parallel calls. The block below simulates both ideas. The first part sends 20,000 requests through four steps, each with a slow tail. The second part makes a request wait for 10 parallel feature lookups and takes the slowest. Press Run, and compare the single lookup with the slowest of ten.

The whole request has a p50 of 26.5 ms and a p99 of 62.0 ms. The feature lookup has the heaviest tail, with a p99 of 41.2 ms against a p50 of 7.9 ms, so it is the first place to look. Notice that the four p99 values add up to 81.2 ms, more than the request's own p99. Percentiles do not add, because the steps are rarely slow in the same request. The second part is the surprise. One lookup has a median of 8.0 ms, but waiting for the slowest of ten pushes the median to 22.8 ms. The more calls a request waits on, the more likely one of them is slow.

Use the demo below to see why waiting on many calls hurts. Choose how many parallel calls a request makes and how rare a slow call is, and read off how often a request is slowed by at least one of them.

• -Cache lookups and, where inputs repeat, predictions themselves.
• -Reduce fan-out: fetch features in one batched call, or precompute combined features.
• -Shrink the model with quantization or distillation, or use a smaller model for easy cases.
• -Batch requests together on a GPU. Larger batches raise throughput but add waiting time, so set a short maximum wait.
• -Set timeouts and fallbacks. A request that waits forever hurts everyone behind it. Serve a default when a step is too slow.
• -Autoscale on load, and keep headroom, since a cold instance is slow.

A model release can fail in ways code releases do not, because the code can run perfectly while the predictions get worse. Use staged rollouts and keep a fast way back.

• -Shadow mode: run the new model on live traffic, log its predictions, and do not show them to anyone. You see how it would have behaved at no risk.
• -Canary: send a small share of real traffic, say 1% to 5%, to the new model and compare its metrics with the old one before widening.
• -Blue-green: run old and new side by side and switch traffic over all at once, with an instant switch back.
• -A/B test: split users and measure a business metric, covered in the experiments chapter.

Keep a model registry that records, for each version, the code, the training data, the features, and the evaluation results. When something breaks at night, rolling back to the last good version should be one command.