Skip to content
Deep Residual Learning for Image Recognition 3 min

Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun · CVPR 2016

Deep Residual Learning for Image Recognition

The shortcut connection that made 152-layer networks trainable: the degradation problem, learning F(x) + x, and a toy that shows why deep stacks need it.

Read the original paper

AI & ML · White Belt · Pro · 3 min read

Deep Residual Learning for Image Recognition

By 2015, deep learning for images had a clear trend: deeper networks win. AlexNet in 2012 had 8 layers. Later networks had 16 or so, then 22. The reasoning was sensible. Each layer builds on the previous one, from edges to textures to parts to whole objects, so more layers should mean richer features. The authors write that the depth of representations is of central importance for visual recognition.

But when researchers simply stacked on more layers, something strange happened. Past a certain depth, the network got worse, and not because it was memorizing. A team at Microsoft Research proposed a small change that made networks over a hundred layers trainable and won the 2015 ImageNet competition. It is called a residual connection, and it is now in nearly every large neural network.

Instead of asking a block of layers to learn the full output, let it learn only the difference from its input and add the input back with a shortcut. Doing nothing becomes easy, so adding layers can no longer hurt.

The degradation problem

You might expect that a deeper network can never do worse than a shallower one. After all, the extra layers could simply copy their input forward and leave the answer unchanged. The authors build this exact argument: a deeper model has at least one good solution, namely the shallow model plus identity layers.

Yet in experiments the deeper plain networks did worse. On ImageNet, a 34-layer plain network had higher error than an 18-layer one. And the extra error showed up on the training data, so it was not overfitting. The optimizer simply could not find the good solution that provably existed. The authors call this the degradation problem.

The degradation problem: after a point, adding layers to an ordinary network makes training error go up, not down. With shortcut connections the deeper network becomes easy to train and beats the shallower one.

Keep reading with Pro

This paper is part of Pro. Pro and Architect open the full breakdown, with animated diagrams, interactive elements you can play with and a quiz. A few papers stay free to read in full.

Still to read

2 more sections, 1 more diagram and 2 interactive elements to play with

  • The fix: learn the residual
  • Results that changed the field

Found this breakdown useful?

Share it with someone else wrestling with this paper.

Discussion0

Join the Discussion

Sign in to leave comments, reply to others, or like insights.

Sign In to ScaleDojo

No comments yet. Be the first to start the thread!

Enjoyed this? Get more like it.

New paper breakdowns, levels, and one concept worth knowing, straight to your inbox.

No spam, ever. Unsubscribe in one click.

Enjoyed this content?

Your support keeps us creating free resources

We put a lot of hours into researching and writing these guides. If it helped you, consider buying us a coffee. Every bit goes toward keeping ScaleDojo's content free and growing.

$

One-time payment via Stripe. ScaleDojo account required.