Adam: A Method for Stochastic Optimization

Training a deep neural network means repeatedly asking "which direction should I nudge millions of parameters to reduce the loss?" This is a classic problem in stochastic optimization, and the plain stochastic gradient descent approach answers it with one number, a single global learning rate applied uniformly to every parameter. That is a strange simplification: some parameters see huge, frequent gradients, others see tiny, rare ones (this is common with sparse features, or embeddings for rare words), and forcing them to share one step size is rarely ideal.

In 2015, Diederik Kingma and Jimmy Ba published Adam, short for Adaptive Moment Estimation, which gives every single parameter its own learning rate, computed automatically from that parameter's own gradient history. It combines two ideas that already existed separately, momentum and per-parameter adaptive rates, into one update rule, and it is very likely the single most widely used optimizer in deep learning, including in the Transformer paper that popularized modern NLP.

The two ingredients Adam combines

Adam is best understood as a blend of two optimizers that came before it.

Momentum: smoothing out noisy gradients

A single mini-batch gives you a noisy estimate of the true gradient. Momentum keeps an exponential moving average of past gradients, so a consistent downhill direction accumulates speed while noise from batch to batch tends to cancel out:

The raw gradient bounces around from one mini-batch to the next. The exponential moving average m_t rides through that noise and tracks the underlying trend instead.

Adaptive learning rates: RMSProp

AdaGrad and later RMSProp took a different idea: track a moving average of each parameter's squared gradients, and divide the update by the square root of that average. A parameter with consistently large gradients gets a smaller effective step, one with small or sparse gradients gets a relatively larger step:

Adam simply keeps both moving averages, $m_t$ (the first moment, mean of the gradient) and $v_t$ (the second moment, uncentered variance of the gradient), for every parameter.

The bias correction that makes it actually work

Both moving averages start at zero: $m_0 = v_0 = 0$. Early in training, with $\beta_1$ and $\beta_2$ close to 1 (defaults are 0.9 and 0.999), the running averages are still overwhelmingly weighted toward that initial zero, so they badly underestimate the true mean and variance for the first several steps.

Just after initialization, the raw second-moment estimate is a tiny fraction of the true value. Dividing by (1 − β₂ᵗ) corrects for it, and the correction fades out automatically as t grows.

The fix is a bias-corrected estimate, applied at every step t:

As t grows, $\beta_1^t$ and $\beta_2^t$ both shrink toward zero, so the correction factor shrinks toward 1 and quietly disappears, exactly when it is no longer needed.

Putting it together: the update rule

The final parameter update combines the two corrected estimates:

Read this as: take the smoothed gradient direction ($\hat{m}_t$), and scale the step size down for parameters whose gradients have historically been large (dividing by $\sqrt{\hat{v}_t}$), while a tiny $\epsilon$ (commonly 1e-8) just prevents division by zero. The paper's suggested defaults, $\alpha = 0.001$, $\beta_1 = 0.9$, $\beta_2 = 0.999$, are still the starting point most training scripts reach for today, almost a decade later.

On an ill-conditioned loss surface (steep in one direction, shallow in another), plain SGD zigzags, momentum overshoots past the minimum, and Adam's per-parameter scaling heads there directly.

Known limitations, and what came after

Adam was not the final word on optimization. A widely cited 2018 paper, "On the Convergence of Adam and Beyond," showed that in some settings Adam's exponential moving average can forget important large gradients too quickly, in principle preventing convergence on certain convex problems, and proposed a fix called AMSGrad that keeps a running maximum of past $v_t$ values instead of an exponential average.

A separate, more practically important fix came from Loshchilov and Hutter's 2017 paper "Decoupled Weight Decay Regularization," which showed that combining Adam with the traditional L2 weight decay trick (adding a penalty term to the gradient) interacts badly with Adam's adaptive scaling, effectively weakening regularization for parameters with large gradients. Their fix, AdamW, applies weight decay directly to the parameters, decoupled from the adaptive gradient update. AdamW, not the original Adam, is the default optimizer in most modern large-model training today, including for training Transformers.

Why this paper still matters

Adam's real contribution was making adaptive optimization robust and simple enough to become the default choice with almost no tuning required. Before it, getting SGD with momentum to train a deep network well usually meant carefully hand-tuning a learning rate schedule. Adam, with its default hyperparameters, reliably got most models training without that ritual.

That reliability is a large part of why deep learning research accelerated as quickly as it did in the years that followed: an optimizer that "just works" removes one entire axis of manual tuning from every new architecture experiment, letting researchers iterate faster on the actual model instead of chasing a fragile hyperparameter schedule.

Related reading

• Paxos Made Simple

• Brewer's Conjecture and the Feasibility of Consistent, Available, Partition-Tolerant Web Services (The CAP Theorem)