Skip to content
GenAI Learn/The Math You Actually Need
Browsing as a guest. Sign in to save your progress and earn XP as you complete chapters.

Broadcasting & Vector Norms

5 min read

You'll learn to

  • -Explain broadcasting as numpy's way of applying one operation to many rows at once, and work out the result shape
  • -Compute L1 and L2 norms and explain when each one matters
  • -Show why L1 pushes weights to exactly zero while L2 only shrinks them
  • -Understand why embeddings get normalized before comparison

Real models never process one example at a time - they process a batch of thousands. Two ideas make that efficient: broadcasting, which lets one operation apply to every row of a batch at once, and norms, which measure a vector's size in ways that have real consequences for training and for comparing vectors to each other.

An Analogy: One Sticker Design on Every Parcel

On a conveyor belt of parcels, you do not design a new sticker for every parcel. You design one sticker and the machine stamps it on each one as it goes by. Broadcasting is that machine: you hand numpy one small piece, such as a bias vector, and it applies it to every row of a big batch without you writing a loop.

Broadcasting: One Line, Every Row

Suppose every example in a batch needs the same bias vector added to it. Looping over each row and adding the bias manually works, but numpy can do the entire batch in one call: shapes are compared starting from the right, so a bias of shape (d,) lines up automatically with the last dimension of a batch shaped (n, d), and gets "broadcast" across every row for free. The same trick, run the other way, computes every pairwise combination between two sets of vectors at once - the exact operation behind an all-pairs distance matrix, or a batch of attention scores.

The broadcasting idea, expressed with plain Python (numpy does this in one call, in compiled C)

The rule is simple to state. Compare the two shapes starting from the right. Dimensions match if they are equal, or if one of them is 1, in which case it is stretched to fit. So a bias of shape (d,) against a batch of shape (n, d) matches on d, and the bias is repeated across the n rows. Shapes (2, 1) and (1, 3) also work: each size-1 dimension is stretched, and the result has shape (2, 3), one value for every pair.

That second case is the all-pairs trick behind distance matrices and attention scores. Here it is in plain Python, with two points compared against three:

All pairs at once: row i, column j holds the distance from a[i] to b[j]. Numpy does this in one broadcasted expression

L1 vs L2: Two Different Ideas of "Size"

There is a good way to picture the two rules. The L2 norm is the straight line as the crow flies. The L1 norm is a taxi's route through a grid of city blocks, which can only go along the streets. To get from one corner to another that is 3 blocks east and 4 north, the crow flies 5 (the square root of 3 squared plus 4 squared), while the taxi drives 3 + 4 = 7.

A vector's "norm" is just a rule for measuring its size, and the rule you choose has real consequences. The L2 norm is the familiar straight-line length: square every element, sum, take the square root. The L1 norm sums the absolute values instead, with no squaring at all.

L1 and L2 Norms
‖x‖₁ = Σ|xᵢ| ‖x‖₂ = √(Σxᵢ²)
L1 sums absolute values. L2 is the familiar Euclidean length.
L1 norm, L2 norm, and normalizing a vector to unit length

Why L1 Pushes Weights to Exactly Zero

This is the classic interview question, and the answer comes from how hard each penalty pulls on a small weight. The L1 penalty pulls by a fixed amount, however small the weight is, so a weight near zero is dragged the rest of the way and stays there. The L2 penalty pulls in proportion to the weight, so as the weight shrinks, so does the pull, and the weight gets closer to zero without ever quite arriving.

The same start (0.5), a fixed pull against a proportional pull: L1 hits zero on step 5, L2 never does

Normalizing Embeddings: A Worked Example

Cosine similarity cares about the direction of two vectors, not their length. If you divide each vector by its L2 norm, every vector has length 1, and the cosine similarity becomes a plain dot product. Take (3, 4) and (4, 3). Their norms are both 5, so they normalize to (0.6, 0.8) and (0.8, 0.6), and the dot product is 0.6 x 0.8 + 0.8 x 0.6 = 0.96. That is a very similar direction, which matches what you would see if you drew them.

Where Each Norm Actually Matters
Tends to push many weights to exactly zero - produces sparse models
L1 regularization (Lasso)
Shrinks weights smoothly toward zero without eliminating them
L2 regularization (Ridge)
Dividing by the L2 norm rescales a vector to unit length - the standard prep step before comparing embeddings by cosine similarity
Embedding normalization

AI Lab's Level 43 (Broadcasting a Batch) and Level 44 (L1 vs L2 Norms) implement both of these directly in numpy, including the exact reshape trick that makes an all-pairs comparison a single broadcasted expression.

Interview Signal is part of Pro

See a real weak answer next to a real strong one for this exact topic.

Quiz is part of Pro

Test what you just read with a short quiz, and bank the XP.

Ready to Build This?

Implement both from scratch: Level 43 (Broadcasting a Batch) and Level 44 (L1 vs L2 Norms) in AI Lab.

Discussion0

Join the Discussion

Sign in to leave comments, reply to others, or like insights.

Sign In to ScaleDojo

No comments yet. Be the first to start the thread!