Broadcasting & Vector Norms

Real models never process one example at a time - they process a batch of thousands. Two ideas make that efficient: broadcasting, which lets one operation apply to every row of a batch at once, and norms, which measure a vector's size in ways that have real consequences for training and for comparing vectors to each other.

Suppose every example in a batch needs the same bias vector added to it. Looping over each row and adding the bias manually works, but numpy can do the entire batch in one call: shapes are compared starting from the right, so a bias of shape (d,) lines up automatically with the last dimension of a batch shaped (n, d), and gets "broadcast" across every row for free. The same trick, run the other way, computes every pairwise combination between two sets of vectors at once - the exact operation behind an all-pairs distance matrix, or a batch of attention scores.

A vector's "norm" is just a rule for measuring its size, and the rule you choose has real consequences. The L2 norm is the familiar straight-line length: square every element, sum, take the square root. The L1 norm sums the absolute values instead, with no squaring at all.