Monitoring, Drift & Retraining

Launching a model is not the finish line. The world it was trained on keeps changing, and the model does not know. A forecasting model that worked in spring can be wrong by autumn without a single line of code being touched. Monitoring is how you find out, and retraining is how you respond.

A map that was perfect when it was printed slowly becomes wrong. New roads open, an old bridge closes, a town grows. The map has not changed, the territory has. A model is a map made from past data. Monitoring is checking, from time to time, whether the roads still match.

Chip Huyen's book on ML system design, in its chapter on data distribution shifts, separates the common cases. The names are used widely.

• -Covariate shift: the inputs change, but the relationship between inputs and outcome stays the same. A delivery app launches in a new city, so it sees streets and speeds the model never learned.
• -Label shift: the mix of outcomes changes. A fraud model trained when 0.5% of orders were fraud now faces a wave of attacks and 3% are fraud, so its thresholds and calibration are off.
• -Concept drift: the relationship itself changes. A new road opens, and the same distance now takes less time. Or customers change what they want, so the same features now predict a different outcome.

• -System health: latency, errors, and throughput, as for any service.
• -Input data: feature distributions, missing values, and out-of-range values compared with the training data. These warnings arrive immediately, since no labels are needed.
• -Predictions: the distribution of scores. A sudden change in the share of high-risk scores is an early sign of trouble.
• -Outcomes: real accuracy, once labels arrive. Labels often come late, such as chargebacks weeks later, so this check is slow but it is the ground truth.
• -Business metrics: the number the model exists to move. A drop here is the final alarm.

When labels are slow, input and prediction monitoring are your early warning, and outcome monitoring is the confirmation. Use the first to raise questions and the second to answer them.

One common way to measure how far a feature has moved is the population stability index, or PSI. You divide the training values into ten equal-sized bins, count what share of a new batch falls in each bin, and add up how far each share has moved from the expected one. A rule of thumb used widely, for example in credit scoring, treats below 0.1 as stable, 0.1 to 0.25 as worth watching, and above 0.25 as a major shift. These are conventions, not laws, so tune them on your own data.

The block simulates average trip speed for a delivery service. The training data averages 30 km/h. In production, traffic slowly gets worse over five weeks, and in the last two weeks a road closure also makes speeds vary more. Press Run and watch the PSI column cross the thresholds.

The PSI is 0.003 and 0.010 in the first two weeks, and 0.056 in week 3, when the mean speed has already slipped from 30.1 to 28.6 km/h. It reaches 0.241 in week 4, just under the major-shift line, and 0.681 in week 5, a clear alert. A drift score tells you that something changed, not why, and not whether the model got worse. It is a prompt to investigate.

The demo below lets you drift the feature yourself. Shift the average speed and widen the spread, or jump to a week from the table, and watch the ten bins fill unevenly as the score crosses its thresholds.

• -Scheduled: retrain every day, week, or month. Simple and predictable, and good when the world drifts steadily.
• -Triggered: retrain when drift or an accuracy metric crosses a threshold. Efficient, but it needs reliable monitoring.
• -Continual: update the model continuously as new data arrives. Very fresh and very demanding to operate.

Whatever the trigger, treat a retrained model like any new model. Train it on a time-based split, evaluate it offline against the current one, and release it in stages. A champion and challenger setup keeps the current model, the champion, serving while the retrained one, the challenger, runs in shadow or on a small share of traffic. Promote it only if it is better.