The Bias-Variance Tradeoff
You'll learn to
- -Understand bias and variance as the two underlying sources of a model's prediction error
- -Write the full bias-variance decomposition of expected test error
- -Connect bias-variance directly back to underfitting and overfitting
Overfitting and underfitting are the symptoms. Bias and variance are the underlying causes. This is one of the most useful frameworks in all of machine learning for reasoning about why a model is failing and what to actually do about it.
Bias: Being Systematically Wrong
Bias is error from a model being too simple to represent the true underlying relationship. It makes systematic mistakes because its assumptions are too restrictive to ever fit the real pattern well, no matter how much data it sees. High bias is the direct cause of underfitting.
Variance: Being Unstable
Variance is error from a model being too sensitive to the exact training data it happened to see. Retrain it on a slightly different sample of the same underlying population and you would get a meaningfully different model, one that latched onto that particular sample's noise. High variance is the direct cause of overfitting.
The Decomposition, Precisely
This is not just a loose metaphor. A model's expected error on new data can be mathematically decomposed into exactly three additive terms. This is a genuinely popular whiteboard interview question. Being able to write this formula and explain each term signals real understanding, not memorized vocabulary.
Why "Tradeoff"?
These two error sources typically pull in opposite directions as you adjust a model's complexity. A very simple model, like plain linear regression, tends toward high bias and low variance. It is stable but systematically wrong. A very complex model, like a deep, unpruned decision tree, tends toward low bias and high variance. It can represent the true pattern well but is unstable and prone to fitting noise. The practitioner's job is finding the sweet spot: just enough complexity to capture the real pattern, not so much that it starts chasing noise.
This tradeoff is also why the ensemble methods from the Classical ML Toolbox module work so well. Averaging many individually high-variance models, like deep decision trees, tends to cancel out their individual instabilities, lowering overall variance while keeping the low bias each individual tree provides.
Interview Signal is part of Pro
See a real weak answer next to a real strong one for this exact topic.
Quiz is part of Pro
Test what you just read with a short quiz, and bank the XP.