Backpropagation: How Networks Learn From Mistakes
You'll learn to
- -Understand backpropagation as the specific technique that computes gradients efficiently in a network
- -Derive every gradient in a small network by hand, via the chain rule
- -Implement full backpropagation from scratch and verify it against a live diagram
Gradient descent needs to know the gradient, the downhill direction, at every single one of a network's millions of parameters, for every single step. Backpropagation is the specific algorithm that makes computing all of those gradients efficient, and it is arguably the single most important algorithm in all of deep learning.
The Problem It Solves
A weight buried deep in the middle of a network affects the final output only indirectly, through a long chain of subsequent layers. Figuring out exactly how much that one weight contributed to the final error, and therefore how it should be adjusted, is not obvious. Computing this the naive way, one weight at a time from scratch, would be far too slow to be practical for any real network.
The Idea: Propagate the Error Backward
Backpropagation solves this with a clever shortcut. After a forward pass produces a prediction and the loss function measures the error, that error signal is propagated backward through the network, one layer at a time, from the output layer back toward the input layer. At each layer, the algorithm efficiently computes exactly how much each weight in that layer contributed to the final error, reusing calculations from the layer after it rather than starting over from scratch each time, an application of the chain rule from calculus.
The interactive walkthrough below uses the exact same tiny network as the forward pass chapter, now running backward. Given a true label, it computes the loss, then derives every gradient layer by layer, right to left.
Backpropagation, Flowing Right to Left
The error at the output propagates backward, one layer at a time, via the chain rule.
a_out = 0.775, true label y = 1
loss L = 0.5 × (a_out − y)² = 0.5 × (-0.225)² = 0.0253
The Chain Rule, Applied Layer by Layer
Notice dW_h2 comes out as all zeros when you run the code above. That is not a bug. For this particular input, z_h2 was negative during the forward pass, so ReLU'(z_h2) = 0, which zeroes out h2's entire error signal, the exact "dead ReLU" case from the quiz below. Try Input B or Input C in the live diagram above to see h2 receive nonzero gradients instead.
From Perceptrons to Reasoning Models
The papers, breakthroughs, and origin stories behind every idea in this course - from the 1958 Perceptron to modern LLM agents.
When machines first tried to think
Scale + GPUs = the deep learning revolution
The paper that changed everything
From 117M parameters to 100M users
Grounding AI in facts, not fiction
From text generators to autonomous workers
From notebook to 10 million users
Why This Mattered So Much
Backpropagation, popularized in the mid-1980s, is precisely what made training networks with multiple hidden layers computationally practical, directly answering the limitation that caused the first AI Winter. Before backpropagation, there was no efficient way to train the deeper, more powerful networks that could overcome a single perceptron's straight-line-only limitation from earlier in this module.
Put together, the training loop for a neural network is exactly this: forward pass to get a prediction, loss function to measure how wrong it was, backpropagation to compute how each individual weight should change, and gradient descent to actually make those small changes. Repeat this loop many thousands of times, and a randomly initialized network gradually becomes a trained one.
Interview Signal is part of Pro
See a real weak answer next to a real strong one for this exact topic.
Quiz is part of Pro
Test what you just read with a short quiz, and bank the XP.