Train / Validation / Test Splits & Cross-Validation
You'll learn to
- -Understand why data must be split into separate training, validation, and test sets
- -Explain what would go wrong if these sets were not kept separate
- -Implement a reproducible train/validation/test split from scratch
- -Implement k-fold cross-validation and explain when it beats a single validation split
A model needs to be evaluated on data it has never seen during training. Otherwise you are not measuring how well it generalizes, you are measuring how well it memorized. This is the reasoning behind one of the most important, and most frequently violated, practices in ML: splitting your data into three separate sets before you do anything else.
The Three Sets
- -The training set, typically 60 to 80 percent of the data, is what the model actually learns from.
- -The validation set is used during development to tune choices like the learning rate, model size, or which features to include: essentially, making decisions about the model without touching the final exam.
- -The test set is touched exactly once, at the very end, to report a final, honest estimate of how the model will perform on genuinely new data.
Why Not Just Use One Set for Everything?
If you evaluated the model on the same data it trained on, a model that had simply memorized every training example, a failure mode called overfitting that the next module covers fully, would score perfectly, and you would have no way to detect that it had failed to actually generalize. The validation set exists because even tuning decisions based on repeated peeks at the test set will slowly leak information about the test set into your choices, quietly inflating your confidence.
A subtle but common mistake is data leakage, where information from the validation or test set accidentally influences training. Cleaning or feature-engineering the entire dataset together before splitting it is a classic example, since it can let statistics from the test set quietly leak backward into the training process.
Splitting Correctly Matters More Than It Sounds
How you split matters too. Randomly shuffling and splitting works for many problems, but for time-series data, splitting must respect chronological order, training on the past and testing on the future. Otherwise the model gets an unrealistic advantage by seeing the future during training that it will never have in actual production use.
The Problem With a Single Validation Split
A single train/validation split has a hidden weakness. Your estimate of how good the model is depends on exactly which examples happened to land in the validation set. On a small dataset especially, a lucky validation split can make a mediocre model look great, and an unlucky one can make a good model look bad, pure noise from the split itself, not a real signal about the model.
K-Fold Cross-Validation: Averaging Over Many Splits
K-fold cross-validation fixes this by not committing to just one split. It divides the training data into K equal-sized folds, then trains and evaluates the model K separate times, each time holding out a different fold as validation and training on the remaining K minus one folds. The final performance estimate is the average across all K runs, far less sensitive to any single lucky or unlucky split.
When to Use K-Fold vs. a Single Split
The test set stays exactly as before, untouched by any of this. K-fold cross-validation replaces the single validation split for model development and tuning, but the final, once-only test set evaluation at the end is unchanged.
Interview Signal is part of Pro
See a real weak answer next to a real strong one for this exact topic.
Quiz is part of Pro
Test what you just read with a short quiz, and bank the XP.