RMSNorm & a Complete Transformer Block

You now have every mathematical piece a Transformer block is made of. This chapter is pure assembly: normalization, attention, a feedforward layer, and the residual connections that make it all trainable when stacked dozens of layers deep.

You met BatchNorm back in Phase 1 - normalize a batch's activations to zero mean, unit variance, then let the network learn its own scale back. RMSNorm, used by LLaMA, Mistral, and most modern open LLMs, is a cheaper variant: skip mean-centering entirely, and rescale purely by the root-mean-square.

Stack enough layers without care and gradients either vanish or explode on their way back through the network, and training simply fails. The fix that made very deep networks (and Transformers stacking dozens of blocks) practical is deceptively simple: instead of a layer's output replacing its input, add the layer's output TO its input. x = x + sublayer(x). Even a sublayer contributing nothing useful yet cannot break the signal flowing through - it just adds zero.

Real Transformer blocks combine this with "pre-norm": normalize BEFORE a sublayer runs, but the residual add still uses the original, un-normalized input. That combination is what keeps training stable across many stacked blocks.