Loss Functions: Measuring Wrongness
You'll learn to
- -Understand a loss function as a single number quantifying how wrong a model's predictions are
- -Write and implement mean squared error and binary cross-entropy
- -Extend binary cross-entropy to categorical cross-entropy for more than two classes
Every training process needs a single number to answer one question: how wrong is the model right now? That number is called the loss (sometimes the cost), and training is, at its core, an attempt to make that number as small as possible.
Comparing Prediction to Reality
A loss function takes the model's prediction and the actual correct answer and outputs a number representing how far apart they are. Predict a house at $410,000 when it actually sold for $400,000, and a simple loss function might just report the $10,000 difference, or more commonly that difference squared, which penalizes big misses much more than small ones.
Mean Squared Error: The Default for Regression
Binary Cross-Entropy: The Default for Two-Class Problems
For classification, squared error is not the natural choice. You want to punish a model that was confidently wrong far more than one that was merely uncertain. Binary cross-entropy does exactly that.
What About More Than Two Classes?
Real problems rarely stop at two classes. Is this photo a cat, a dog, or a bird? Binary cross-entropy generalizes cleanly into categorical cross-entropy, which compares a full predicted probability distribution across every class against a true distribution that puts all its weight on the one correct class.
Categorical cross-entropy is almost always paired with a softmax output layer, which the next chapter covers in depth. Softmax turns a network's raw scores into a valid probability distribution across all the classes, and categorical cross-entropy is the loss that scores how good that distribution is.
Loss Is Not the Same as the Metric You Actually Care About
One subtlety worth remembering: the loss function used during training is not always the same thing as the business metric you actually care about. A model might be trained to minimize squared error on price predictions, while the business cares about the percentage of predictions within 5% of the true price. These can pull in slightly different directions, which is worth watching for.
Interview shorthand worth memorizing: MSE for regression, binary cross-entropy for two classes, categorical cross-entropy for more than two. You will rarely be wrong reaching for these as your default answer, as long as you can explain why the shape of each penalty fits its task.
Interview Signal is part of Pro
See a real weak answer next to a real strong one for this exact topic.
Quiz is part of Pro
Test what you just read with a short quiz, and bank the XP.