Broadcasting & Vector Norms
You'll learn to
- -Explain broadcasting as numpy's way of applying one operation to many rows at once, and work out the result shape
- -Compute L1 and L2 norms and explain when each one matters
- -Show why L1 pushes weights to exactly zero while L2 only shrinks them
- -Understand why embeddings get normalized before comparison
Real models never process one example at a time - they process a batch of thousands. Two ideas make that efficient: broadcasting, which lets one operation apply to every row of a batch at once, and norms, which measure a vector's size in ways that have real consequences for training and for comparing vectors to each other.
An Analogy: One Sticker Design on Every Parcel
On a conveyor belt of parcels, you do not design a new sticker for every parcel. You design one sticker and the machine stamps it on each one as it goes by. Broadcasting is that machine: you hand numpy one small piece, such as a bias vector, and it applies it to every row of a big batch without you writing a loop.
Broadcasting: One Line, Every Row
Suppose every example in a batch needs the same bias vector added to it. Looping over each row and adding the bias manually works, but numpy can do the entire batch in one call: shapes are compared starting from the right, so a bias of shape (d,) lines up automatically with the last dimension of a batch shaped (n, d), and gets "broadcast" across every row for free. The same trick, run the other way, computes every pairwise combination between two sets of vectors at once - the exact operation behind an all-pairs distance matrix, or a batch of attention scores.
The rule is simple to state. Compare the two shapes starting from the right. Dimensions match if they are equal, or if one of them is 1, in which case it is stretched to fit. So a bias of shape (d,) against a batch of shape (n, d) matches on d, and the bias is repeated across the n rows. Shapes (2, 1) and (1, 3) also work: each size-1 dimension is stretched, and the result has shape (2, 3), one value for every pair.
That second case is the all-pairs trick behind distance matrices and attention scores. Here it is in plain Python, with two points compared against three:
L1 vs L2: Two Different Ideas of "Size"
There is a good way to picture the two rules. The L2 norm is the straight line as the crow flies. The L1 norm is a taxi's route through a grid of city blocks, which can only go along the streets. To get from one corner to another that is 3 blocks east and 4 north, the crow flies 5 (the square root of 3 squared plus 4 squared), while the taxi drives 3 + 4 = 7.
A vector's "norm" is just a rule for measuring its size, and the rule you choose has real consequences. The L2 norm is the familiar straight-line length: square every element, sum, take the square root. The L1 norm sums the absolute values instead, with no squaring at all.
Why L1 Pushes Weights to Exactly Zero
This is the classic interview question, and the answer comes from how hard each penalty pulls on a small weight. The L1 penalty pulls by a fixed amount, however small the weight is, so a weight near zero is dragged the rest of the way and stays there. The L2 penalty pulls in proportion to the weight, so as the weight shrinks, so does the pull, and the weight gets closer to zero without ever quite arriving.
Normalizing Embeddings: A Worked Example
Cosine similarity cares about the direction of two vectors, not their length. If you divide each vector by its L2 norm, every vector has length 1, and the cosine similarity becomes a plain dot product. Take (3, 4) and (4, 3). Their norms are both 5, so they normalize to (0.6, 0.8) and (0.8, 0.6), and the dot product is 0.6 x 0.8 + 0.8 x 0.6 = 0.96. That is a very similar direction, which matches what you would see if you drew them.
AI Lab's Level 43 (Broadcasting a Batch) and Level 44 (L1 vs L2 Norms) implement both of these directly in numpy, including the exact reshape trick that makes an all-pairs comparison a single broadcasted expression.
Interview Signal is part of Pro
See a real weak answer next to a real strong one for this exact topic.
Quiz is part of Pro
Test what you just read with a short quiz, and bank the XP.
Implement both from scratch: Level 43 (Broadcasting a Batch) and Level 44 (L1 vs L2 Norms) in AI Lab.
Discussion0
Join the Discussion
Sign in to leave comments, reply to others, or like insights.
No comments yet. Be the first to start the thread!