Optimization Intuition, Without the Calculus

Once you have a loss function measuring wrongness, training becomes the process of adjusting the model's internal numbers, its parameters, to make that loss as small as possible. This chapter builds the intuition for how that adjustment happens. The full mechanics wait until the Neural Networks module, but the picture here applies broadly across ML.

Imagine every possible setting of a model's parameters as a point on a landscape, with the loss at that point as the landscape's height. Lower is better. Training is like standing somewhere on that landscape in thick fog, unable to see the whole terrain, trying to walk downhill to the lowest point you can find. At each step, you can only feel the slope right where you are standing.

The standard approach, gradient descent, repeatedly checks which direction is downhill from the current position and takes a small step that way, then repeats. Over many thousands of small steps, the parameters gradually settle into a low-loss region. Gradient is just the mathematical term for the direction and steepness of the slope right here, formally the derivative of the loss with respect to the parameter.

How far to step at each iteration is controlled by α (alpha), the learning rate. Too large, and you might leap right over the lowest point and bounce around without ever settling, like sprinting downhill in the fog and overshooting the valley floor. Too small, and training crawls forward so slowly it might never finish, or gets permanently stuck in a shallow dip that is not actually the best possible spot. Try all three below and watch the same starting point lead to three very different outcomes.