Batch vs Online Inference

Once a model exists, a question decides much of the architecture: do you compute predictions ahead of time, or when someone asks? The answer affects cost, how fresh the predictions are, and how much can go wrong at 3 a.m.

Meal prep on Sunday is cheap per meal, but by Friday the food is stale, and you may have cooked dishes nobody eats. Cooking to order is fresh and tailored, but you need a kitchen ready for the rush and every guest waits. Batch inference is meal prep. Online inference is cooked to order. Most real systems do both.

• -Batch: score everything on a schedule, store the results, and look them up when needed. Cheap and simple at serving time, but results can be a day old and cover users who never show up.
• -Online: score at request time with fresh features. The answer reflects what just happened, but you pay for every request and must meet a latency target, often tens of milliseconds.
• -Near-real-time or streaming: update predictions or features within seconds as events arrive. A middle path when freshness matters but per-request scoring is too costly.
• -Hybrid: precompute the heavy parts (candidate lists, embeddings) in batch and finish online with fresh signals, such as a ranker that reads the last five minutes of clicks.

Batch scoring has a fixed cost: it scores every user, whether or not they visit. Online scoring costs a little per request, and only when someone arrives. Which is cheaper depends on how many of your users are active and how often each one is scored. The code below assumes 10 million users and 5 milliseconds of compute per prediction. Each active user opens the feed about six times a day. Press Run and read the last column, then the break-even line.

The nightly batch costs 50,000 compute-seconds whatever happens. When only 200,000 users are active, that bill is shared by few people, 0.25 seconds each, against 0.03 seconds each for online scoring, so online is cheaper. The lines cross at about 1.67 million active users, 17% of all users. Past that point batch is cheaper. This is a simplified model, because it ignores the cost of storing predictions, but it shows the shape of the decision: batch pays off when most of what you precompute is used, and each result is used many times.

Cost is only one axis. Batch is also stale. If a user bought a phone ten minutes ago, a batch recommender computed last night still shows phones. When freshness drives the value, you move at least the final step online.

• -Restrict batch to likely users. Score only people active in the last 30 days, and fall back to a popular list for the rest.
• -Precompute candidates, rank online. Retrieval is heavy and changes slowly, so build candidate lists in batch. Ranking needs fresh context, so do it per request.
• -Cache predictions with a time limit. Reuse a result for minutes when the same input repeats.
• -Plan the fallback first. If the online model times out, serve the latest batch result, and if that is missing, serve a safe default. A slow or missing prediction should degrade the product, not break it.

• -A decision is needed in tens of milliseconds, on fresh data: online.
• -The answer is used later, in bulk, such as a nightly email or report: batch.
• -Freshness matters, but you can afford a few seconds of delay: streaming.
• -Retrieval is heavy and ranking needs fresh context: hybrid.