Regularization Techniques, Briefly

If overfitting comes from a model being too free to chase noise in its training data, regularization is the family of techniques that deliberately constrain that freedom, trading a small amount of fit on the training data for meaningfully better generalization to new data.

Many regularization techniques work by adding a penalty to the loss function, recall the Math module, based on how large or complex the model's parameters are. This nudges training toward simpler solutions unless the added complexity genuinely earns its keep by meaningfully reducing error, directly discouraging the kind of overly flexible fitting that leads to overfitting.

• -L1 regularization is like L2, but penalizes the sum of absolute weight values instead of squared values, a subtle but important difference. L1 tends to push some weights all the way to exactly zero, effectively selecting features, while L2 shrinks weights smoothly without usually zeroing them out.
• -Dropout, specific to neural networks, randomly turns off a fraction of neurons during each training step, preventing the network from becoming overly reliant on any single neuron or narrow path through the network.
• -Early stopping simply stops training once validation performance stops improving, even if training performance would keep improving further, directly using the overfitting-detection signal from the previous chapter as a training control.
• -Data augmentation artificially expands the effective training set, for example by slightly rotating or cropping training images, so the model sees more variety and has less opportunity to memorize any single exact example.