Training Stability: Initialization, Normalization & Gradient Problems

Backpropagation explained how a gradient gets computed. This chapter is about what can go wrong with that process once a network gets deep, and the two standard fixes every practitioner reaches for. This is genuinely practical territory. Get initialization or normalization wrong and a network that should train perfectly well simply does not.

It seems natural to initialize every weight to zero, but that choice quietly breaks training before it even starts. If every weight in a layer starts identical, every neuron in that layer computes the exact same output and receives the exact same gradient during backpropagation. They stay identical forever, no matter how long you train. This is called the symmetry problem, and it is why weights always start as small random numbers instead of a fixed value.

Recall from the backpropagation chapter that a hidden layer's error signal is the output error multiplied by a weight and an activation derivative, layer after layer. Stack ten or twenty layers, and that gradient gets multiplied by roughly the same small factor ten or twenty times in a row. A factor just below 1, repeated enough times, shrinks toward zero fast. By the time the signal reaches the earliest layers, there may be almost nothing left to learn from. Those layers barely update at all, even while later layers train just fine.

The same multiplication can run the other way. A factor just above 1, repeated across many layers, grows exponentially instead of shrinking. The gradient can blow up to enormous values, causing wild, unstable weight updates that can wreck a training run in a single bad step.

Try all three below and watch the same ten-layer network behave in completely different ways depending on nothing but how it was set up.

The fix is to choose the initial random weights carefully, scaled to the size of each layer, so the signal neither shrinks nor grows much as it passes through. Two standard schemes handle this, and they are genuinely named after the papers that introduced them.

Good initialization only guarantees a stable start. As training proceeds and weights change, the distribution of values flowing into each layer can still drift. Batch normalization addresses this directly, at every layer, throughout training, not just at the start. For each mini-batch, it rescales that layer's inputs to have a mean of 0 and a standard deviation of 1, then lets the network learn a small correction on top if needed.