Gradient Descent, Intuitively, Applied to Networks

The Math module introduced gradient descent as walking downhill on a loss landscape in the fog. This chapter reconnects that picture specifically to neural networks, where the landscape is far stranger than the simple 2D or 3D pictures anyone can actually visualize.

Every single weight and bias in the network is one dimension of the loss landscape. A modest neural network might have hundreds of thousands of these parameters. Large language models have billions. The landscape gradient descent walks across is not a hill you could picture in three dimensions. It is a shape with as many dimensions as the network has parameters, a genuinely different regime than the simple analogy suggests, even though the core idea (find the downhill direction, take a step, repeat) stays exactly the same.

Each single step of gradient descent computes a downhill direction and adjusts every one of those millions of weights simultaneously by a small amount, based on the current training examples. Then it repeats, over and over, often millions of steps in total, gradually reshaping the entire network from random noise into something that produces useful predictions.

The smallest possible version of this, two parameters instead of millions, is already enough to see the whole loop in action. The code below fits a line, slope w and intercept b, exactly like linear regression's closed-form solution from the Classical ML Toolbox module, using gradient descent instead of the closed-form formula, and prints the loss shrinking step by step.

In practice, each step of gradient descent typically looks at only a small batch of training examples at a time rather than the entire dataset, computing the downhill direction from just that batch before moving to the next one. This makes each step far cheaper to compute and, somewhat counterintuitively, often trains better in practice than using the full dataset for every single step. This variant is called mini-batch gradient descent, or with a batch size of exactly one, stochastic gradient descent.