Optimizers: Beyond Vanilla Gradient Descent

Every gradient descent example so far in this course has used the plain update rule from two chapters ago: subtract the learning rate times the gradient. In practice, almost no real network is trained with plain gradient descent. It has two specific weaknesses that a family of smarter optimizers were built to fix, and knowing them by name, especially the one you should reach for by default, is standard interview territory.

Plain gradient descent treats every step independently. It has no memory of which direction it was moving in previously. On a loss surface shaped like a narrow ravine, this causes visible zig-zagging. The gradient keeps pointing back and forth across the narrow direction while making frustratingly slow progress along the long, gentle direction toward the actual minimum.

Momentum fixes this by keeping a running velocity, an exponentially weighted average of recent gradients, and stepping in that direction instead of the raw current gradient alone. Like a heavy ball rolling downhill, it builds up speed in a consistent direction and resists sudden changes, smoothing out the zig-zag and accelerating through gentle, consistent slopes.

Plain gradient descent also applies the exact same learning rate to every single parameter, even though different parameters can have wildly different-sized gradients. A parameter with a consistently large gradient can overshoot, while one with a consistently tiny gradient barely moves at all, both problems happening simultaneously, with no single learning rate that is right for both.

RMSprop keeps a running average of each parameter's recent squared gradients, then divides that parameter's learning rate by the square root of that average. A parameter that has been seeing large gradients gets its effective learning rate shrunk. A parameter seeing small gradients gets a relative boost. Every parameter effectively gets its own, self-adjusting learning rate.

Adam, short for Adaptive Moment Estimation, tracks both quantities at once: a momentum-style running average of the gradient itself, and an RMSprop-style running average of the squared gradient, then combines them into a single update. This gives Adam both benefits simultaneously, a smoothed, momentum-driven direction and a per-parameter adaptive step size. It is, by a wide margin, the most commonly used optimizer in deep learning today, and "just use Adam" is a genuinely reasonable default answer in most practical situations.

Watch all three race on the same landscape below. The gap between them is not subtle.