The Forward Pass: How a Network Predicts

With layers and activation functions in place, this chapter walks through the actual mechanical process of turning an input into a prediction, called the forward pass, since data flows forward through the network, layer by layer, from input to output. Step through the diagram below. It uses a small, fixed network, two inputs, two ReLU hidden neurons, one sigmoid output, that the code sample beneath it reproduces exactly.

• -The input layer receives the raw data as a vector: pixel values, numeric features, or a text representation, depending on the task.
• -Each neuron in the first hidden layer computes its weighted sum plus bias, then applies its activation function.
• -That layer's outputs become the next layer's inputs, and the same weighted-sum-plus-activation process repeats.
• -This continues through every hidden layer until reaching the output layer, whose format matches the task: a single number for regression, a probability per category for classification.

Every one of these steps is pure arithmetic: multiplication, addition, and a fixed activation function, repeated many times. There is nothing mysterious happening inside a forward pass. The apparent intelligence of the output emerges from the accumulated effect of millions, or billions, of individually simple weighted sums, using weights that were carefully shaped by training.

It is worth being precise about a distinction that trips people up. The forward pass is what happens every time the network makes a prediction, whether during training or in production. Training additionally runs backpropagation, the next chapter, after each forward pass, to figure out how to adjust the weights. In production, only the forward pass runs. The weights are frozen at whatever training left them, and each prediction is just this same forward flow of arithmetic through fixed weights.