Layers, Activations, and Why Nonlinearity Matters
You'll learn to
- -Understand how stacking neurons into layers overcomes a single neuron's limitations
- -Understand why a nonlinear activation function is essential, not optional
- -Know softmax and how it extends sigmoid to more than two classes
The previous chapter ended on a cliffhanger. A single neuron can only draw a straight-line boundary. The fix that unlocked everything since is deceptively simple: stack many neurons together into layers, and connect one layer's outputs to the next layer's inputs.
Layers: Neurons Feeding Neurons
A neural network arranges neurons into an input layer for the raw data, one or more hidden layers in between, and an output layer for the final prediction. Every neuron in one layer typically connects to every neuron in the next, each connection carrying its own learned weight. Data flows forward through the layers, transformed a little at each step, until it emerges as a final prediction.
The Catch: Stacking Alone Is Not Enough
Here is a genuinely surprising fact. If every neuron just computes a weighted sum with nothing else, stacking any number of layers together mathematically collapses back down into the exact same thing as a single layer. Pure weighted sums, no matter how many you chain together, can still only represent straight-line boundaries. Depth alone buys you nothing without one more ingredient.
Activation Functions: The Missing Ingredient
That ingredient is a nonlinear activation function applied to each neuron's output before passing it to the next layer: a function that bends or reshapes the signal rather than just scaling it. Adding this single nonlinear step between layers is what allows a network to represent curved, complex decision boundaries instead of only straight lines.
This is one of the most commonly misunderstood facts about neural networks. It is not depth alone that gives them power, and it is not any single neuron being clever. It is specifically the combination of depth and nonlinear activation functions between layers, as the code above demonstrates directly.
Softmax: One-vs-Rest, All at Once
Binary classification only needs a single sigmoid output: one number, the probability of the positive class. With more than two classes, softmax generalizes that idea. It takes a raw score for every class and converts the whole set into a valid probability distribution: every value between 0 and 1, and all of them summing to exactly 1.
Drag the sliders below and watch three competing raw scores turn into a genuine probability distribution in real time.
Softmax Playground
Drag any raw score. The bars always resettle so they sum to exactly 1.
Sum of all three bars: 1.0000. No matter what the raw scores are, softmax always redistributes them into a valid probability distribution that adds up to 1.
Softmax and categorical cross-entropy, from the Loss Functions chapter, are almost always used together: softmax produces the probability distribution, and categorical cross-entropy scores how good that distribution is against the true label.
Interview Signal is part of Pro
See a real weak answer next to a real strong one for this exact topic.
Quiz is part of Pro
Test what you just read with a short quiz, and bank the XP.