By 2015, deep learning for images had a clear trend: deeper networks win. AlexNet in 2012 had 8 layers. Later networks had 16 or so, then 22. The reasoning was sensible. Each layer builds on the previous one, from edges to textures to parts to whole objects, so more layers should mean richer features. The authors write that the depth of representations is of central importance for visual recognition.
But when researchers simply stacked on more layers, something strange happened. Past a certain depth, the network got worse, and not because it was memorizing. A team at Microsoft Research proposed a small change that made networks over a hundred layers trainable and won the 2015 ImageNet competition. It is called a residual connection, and it is now in nearly every large neural network.
Instead of asking a block of layers to learn the full output, let it learn only the difference from its input and add the input back with a shortcut. Doing nothing becomes easy, so adding layers can no longer hurt.
The degradation problem
You might expect that a deeper network can never do worse than a shallower one. After all, the extra layers could simply copy their input forward and leave the answer unchanged. The authors build this exact argument: a deeper model has at least one good solution, namely the shallow model plus identity layers.
Yet in experiments the deeper plain networks did worse. On ImageNet, a 34-layer plain network had higher error than an 18-layer one. And the extra error showed up on the training data, so it was not overfitting. The optimizer simply could not find the good solution that provably existed. The authors call this the degradation problem.
