Training-Serving Skew & Data Leakage

Two bugs sink more ML projects than any modeling mistake, and both are invisible in a notebook. In the first, the model looks excellent offline because it was accidentally shown the answer. In the second, the model that looked excellent offline behaves differently in production because the data it receives there is not the data it learned from. They are called data leakage and training-serving skew.

Leakage is studying with the answer key. You will ace a test on those exact questions and learn nothing, and your score says nothing about how you will do on new questions. Skew is training on a clay tennis court and playing the match on grass. You practiced hard, but the surface behaves differently from the one you will play on.

Google's guide defines skew as a difference between performance during training and performance during serving. It names three causes.

• -A discrepancy in how data is handled in the training and serving pipelines, such as two different pieces of code computing the same feature.
• -A change in the data between when you trained and when you serve, because the world moves.
• -A feedback loop between the model and the algorithm, where the model's own outputs shape its future inputs.

The first cause is the one you can fix with engineering. A classic case is a clock. A training job groups orders by UTC date, while the serving code groups them by the customer's local date. For a customer in India, 5.5 hours ahead of UTC, the two disagree about "today" for any order placed in the last 5.5 hours of the UTC day.

Leakage means the training data contains information that will not exist when the model makes a real prediction. It takes several forms.

• -Target leakage: a feature created by, or after, the outcome, such as a "cancellation survey sent" flag when predicting who will cancel.
• -Future data in a join: a lookup that returns today's value of a feature instead of the value on the day of the example.
• -Preprocessing on the whole data set: computing averages or scaling factors using test rows.
• -Duplicates across the train and test split, so the model is tested on examples it has memorized.

The code builds 5,000 synthetic customers and asks a practical question for each feature: if the retention team contacted the 500 customers the feature ranks as riskiest, what share would really churn? One feature, the cancellation survey flag, is a leak, because the survey is only sent after someone cancels. The second part simulates the clock mismatch described above. Press Run and compare the leak with the honest features.

Overall, about one customer in five churns (21.9%). The honest features lift the hit rate: 49.4% for the top 500 by support tickets, and 29.0% for the top 500 by short tenure. That is what real signal looks like: useful, not magical. The survey flag scores 100% because it is the outcome in disguise, and a model trained with it would look flawless offline and be useless in production, where the flag does not exist yet. The clock mismatch is the second lesson. Roughly 23% of requests disagree about what "today" means, so a feature such as orders today would reach the model with a different value than the one it was trained on.

• -Log the features at serving time, then train on the logged values. Google's guide recommends this directly, and it removes whole classes of mismatch.
• -Use one implementation of each feature. Share the code, or compute the feature once in a store that both training and serving read.
• -Split by time, not at random, when the data has a time order. Train on the past and test on the future.
• -Join with point-in-time correctness: each example may only see feature values that existed on its own date. The next chapter shows how.
• -Review every feature with one question: will this value exist, with this meaning, at the moment of prediction?
• -Measure skew. Compare performance on training data, on held-out data, on the next day's data, and on live traffic. A big drop between the last two points to an engineering error.