How Attention Actually Works
You'll learn to
- -Explain attention with the query, key and value picture, and say what each one is
- -Trace one attention step by hand: scores, scaling, softmax weights, blended output
- -Explain why a Transformer uses several attention heads in parallel
- -Explain the KV cache, and why Grouped-Query Attention shrinks it, using real numbers
You have called an LLM through an API for most of this course, treating what happens inside it as someone else's problem. This chapter opens the box on the one operation that makes modern language models work: attention. Every Transformer layer, in every model from GPT-2 to today's frontier models, runs the same small recipe, and by the end you will have implemented it.
The Problem Attention Solves
Consider "the animal didn't cross the street because it was too tired". To understand the word "it", a reader looks back and weighs the other words: "animal" matters a lot, "street" much less. A language model needs the same ability: for each word, decide how much to draw on every other word, and blend what it learns. Attention is that decision, done with arithmetic.
An Analogy: A Filing Cabinet
Picture every word as a folder in a filing cabinet. Each folder has a label on the front saying what it is about, and notes inside. When a word needs context, it writes down what it is looking for, compares that to every label, scores how well each one matches, and then reads the notes of the best matches, taking more from the better ones.
- -The question a word writes down is its query (Q).
- -The label on each folder is its key (K).
- -The notes inside each folder are its value (V).
- -How well a question matches a label is the score, computed as a dot product.
- -Taking more notes from better matches is a weighted blend of the values.
One Attention Step, by Hand
Here is that sentence in miniature, with three words and made-up two-number vectors. A dot product multiplies matching numbers and adds them up, so the query [2, 0] against the key [1, 0] gives 2, against [0, 1] gives 0, and against [0.5, 0.5] gives 1. Dividing each by the square root of the vector length (here the square root of 2, about 1.41) gives scores of 1.41, 0 and 0.71.
Softmax turns scores into weights that are all positive and add up to 1. Here the weights are about 0.58 for animal, 0.14 for street and 0.28 for tired. The output is then each word's value multiplied by its weight and added up: 0.58 x [8, 1] + 0.14 x [1, 9] + 0.28 x [4, 5] is about [5.88, 3.26]. The answer lies closest to animal's value, because animal earned the biggest share. Run it.
- -Q @ K.T scores every query against every key. It is the same dot-product similarity from the Math module, computed between every pair of positions at once.
- -Dividing by the square root of d_k (the vector length) keeps scores from growing large as vectors get longer. Without it, softmax would give almost all the weight to one position and training would stall.
- -softmax turns each query's row of scores into a probability distribution over the keys.
- -Multiplying those weights by V blends every position's value by how much attention the current position pays it. That blend is the attention output.
Now try the same three-word example yourself. Drag the query and watch the scores and weights change, then turn off the division by the square root of the vector length to see how much sharper softmax becomes without it.
Where Does the Word "it" Look?
Each word offers a key, and the word "it" sends out a query. The better a query matches a key, the higher the score, and a softmax turns scores into attention weights. Move the query and see which word wins.
The chapter's example is [2, 0]: "it" is looking for something animate.
"it" pays most attention to animal (58%) and blends the three values into [5.88, 3.26]. Dividing by the square root of the vector length keeps the scores small, so the weights stay soft.
Multi-Head Attention: Several Readers at Once
One reader can only look for one thing at a time. A real Transformer layer runs attention several times in parallel, each time on a narrower slice of the vector, called a head, so different heads can specialize: one might track grammar, another who did what, another the overall topic. The formula does not change at all. The work is bookkeeping: split a wide vector into narrow ones, run the same attention on each, then join the results back together.
AI Lab's Level 49 (Scaled Dot-Product Attention) and Level 50 (Multi-Head Attention's Reshape Trick) implement exactly this in numpy: the vectorized attention formula across a whole sequence at once, then the reshape and transpose trick that splits and recombines attention heads.
The KV Cache and Grouped-Query Attention
When a model generates text one token at a time, each new token needs the keys and values of every earlier token. Recomputing them at every step would be wasteful, so the model stores them: the KV cache. For a long conversation this cache is often the biggest memory cost of serving the model. How big? Take Llama 2 70B, which has 80 layers and a head size of 128, storing 16-bit numbers for a 4,096-token conversation.
With one KV head for each of its 64 query heads the cache would be 10 GiB for one conversation. Grouped-Query Attention (GQA) keeps many query heads for representational power but far fewer key and value heads, letting groups of query heads share each one. Llama 2 70B uses 64 query heads and only 8 KV heads, so the cache is 8 times smaller: 1.25 GiB.
- -Ordinary multi-head attention is the special case where every query head has its own KV head, one to one.
- -Multi-Query Attention (MQA) is the opposite extreme: every query head shares a single KV head.
- -GQA sits in between: num_q_heads divided by num_kv_heads query heads share each KV head.
- -Once the KV heads are repeated to match the query heads, the attention math is unchanged. GQA is a memory and serving optimization, not a different formula.
AI Lab's Level 62 (Grouped-Query Attention) builds this exact mechanism in numpy: repeating KV heads, then running the same batched attention formula across all query heads at once.
Interview Signal is part of Pro
See a real weak answer next to a real strong one for this exact topic.
Quiz is part of Pro
Test what you just read with a short quiz, and bank the XP.
Implement it in numpy: Level 49 (Scaled Dot-Product Attention), Level 50 (Multi-Head Attention's Reshape Trick), and Level 62 (Grouped-Query Attention) in AI Lab.
Discussion0
Join the Discussion
Sign in to leave comments, reply to others, or like insights.
No comments yet. Be the first to start the thread!