Overfitting & Underfitting
You'll learn to
- -Define overfitting and underfitting as two opposite generalization failures
- -Recognize the telltale sign of each in training and validation performance
- -See both failure modes happen live by fitting the same data at different model complexities
Getting a model to fit its training data well is only half the goal. The real goal is generalization: performing well on new data it has never seen. Overfitting and underfitting are the two opposite ways a model can fail at exactly that goal.
Overfitting: Memorizing Instead of Learning
An overfit model has essentially memorized its training data, including its noise and quirks, rather than learning the genuine underlying pattern. Picture fitting a wildly wiggling curve that passes through every single training point exactly. It looks like a perfect fit on the data you can see, but it will make wild, nonsensical predictions on new data, because it learned the specific noise of this dataset rather than the real signal underneath it.
Underfitting: Not Learning Enough
An underfit model is the opposite failure. It is too simple to capture the real pattern in the data at all, performing poorly even on the training data it was directly shown. Trying to fit a straight line to data that clearly curves is a classic case. The model is not flexible enough to represent the actual relationship, no matter how much you train it.
The demo below fits the same 14 noisy data points with polynomials of increasing degree. Drag from degree 1, too simple, underfit, up to degree 9, memorizes every wiggle, overfit, and watch both the curve and the train/validation error respond.
Fitting the Same Data at Different Complexities
A degree-2 polynomial fit to 10 training points (filled) - 4 held-out validation points (hollow) never influence the fit.
train error (MSE)
0.305
validation error (MSE)
0.553
Train vs. Validation Error, All Degrees
Validation error is clipped at the top of the chart for degrees where it explodes - the shape past that point is "very bad," not literally flat.
Spotting Each One
This is precisely why the train/validation/test split from the previous module matters so much in practice. Overfitting is invisible if you only ever look at training performance. Comparing training performance against validation performance is the single most common way practitioners diagnose it.
Interview Signal is part of Pro
See a real weak answer next to a real strong one for this exact topic.
Quiz is part of Pro
Test what you just read with a short quiz, and bank the XP.