Skip to content
GenAI Learn/Transformers Under the Hood
Browsing as a guest. Sign in to save your progress and earn XP as you complete chapters.

Causal Masking & Positional Encoding

6 min read

You'll learn to

  • -Explain why a language model must be stopped from looking at future tokens, and apply a causal mask by hand
  • -Say why attention on its own cannot tell word order, using "dog bites man" versus "man bites dog"
  • -Read sinusoidal positional encoding as a set of clock hands moving at different speeds
  • -Explain RoPE's rotation and check, by running it, that scores depend only on the distance between positions

The last chapter showed how attention decides which words matter to each other. Two problems remain before it can generate text. First, nothing stops a word from attending to words that come after it, which would let the model cheat by peeking at the answer during training. Second, attention has no idea what order the words are in. Both problems have small, elegant fixes.

Problem 1: No Peeking

During training, the model sees a whole sentence at once but must learn to predict each word using only the words before it. If it could look ahead, it would simply copy the next word and learn nothing. Think of an exam where the answers are printed further down the page and you hold a sheet of paper over them. The mask is that sheet of paper.

  • -The exam paper is the table of attention scores between positions.
  • -The sheet of paper is the causal mask.
  • -The covered answers are the positions after the current one, which are marked minus infinity.
  • -Minus infinity is paper so thick that no light gets through: after softmax the weight is exactly zero.
Causal Mask
mask[i, j] = 0 if j ≤ i, else −∞
Position i may attend to position j only if j is not in the future. The mask is added to the raw scores before softmax.

A worked case with three tokens. Each row below is one token's raw scores for tokens 1, 2 and 3. Token 1 may only see itself, so its weights must be [1, 0, 0]. Token 2 may see tokens 1 and 2: with scores 1 and 3 softmax gives about 0.12 and 0.88. Token 3 sees all three. Why minus infinity and not zero? Because exp of minus infinity is exactly 0, so the blocked positions vanish and the rest still add up to 1. Run it to see.

A causal mask, and its effect once added to raw scores

The demo below uses the same three tokens and the same raw scores as the code. Switch the mask on and off and watch the attention paid to later tokens appear and disappear, while every row still adds up to 1.

The Mask That Stops a Token Peeking Ahead

Each row shows how one token spreads its attention over tokens 1, 2 and 3. In training the model sees the whole sentence at once, so without a mask a token could read the very words it is meant to predict.

looks at token 1looks at token 2looks at token 3
token 10.630.230.14
token 20.110.840.05
token 30.090.240.67

Attention on the future

14%

average over the three tokens

Each row adds up to

1.00

blocked cells are exactly 0

With the mask off, on average 14% of each token's attention lands on tokens that come after it. A model allowed to do that would simply copy the next word and learn nothing.

Problem 2: Attention Cannot Tell Order

"Dog bites man" and "man bites dog" contain the same words and mean opposite things. Attention, as built so far, treats its input as an unordered set: swap two tokens and it produces the same set of outputs, just shuffled. Word order obviously matters, so we must inject position into the model. The fix is to add a position signal to every token's embedding.

Sinusoidal Positional Encoding: Clock Hands

Think of how a clock tells time. One hand alone is ambiguous, but the second, minute and hour hands moving at different speeds together give every moment a unique reading. The original Transformer paper does the same. Each position gets a fixed pattern of sines and cosines, with some pairs changing quickly and others slowly.

  • -A position in the sentence is the time on the clock.
  • -Each sine and cosine pair is one clock hand.
  • -Low dimensions are the fast hands, good at telling neighboring positions apart.
  • -High dimensions are the slow hands, good at telling distant positions apart.
  • -Reading all the hands together identifies the position exactly, and also hints at how far apart two positions are.
Sinusoidal Positional Encoding
PE[pos, 2i] = sin(pos / 10000^(2i/d)), PE[pos, 2i+1] = cos(pos / 10000^(2i/d))
Even embedding dimensions get a sine and odd ones get a cosine, at a speed that slows as i grows. d is the size of the embedding.
Sinusoidal positional encoding for the first three positions, with a 4-number embedding

The first pair of numbers swings quickly, going from (0, 1) at position 0 to (0.84, 0.54) at position 1 and (0.91, -0.42) at position 2. The second pair barely moves, from 0.01 to 0.02. Together they make every position's pattern different.

RoPE: Rotating Instead of Adding

Most modern open models, including Llama, Mistral and DeepSeek, do not add a pattern. They use Rotary Positional Embeddings (RoPE), which keep the same speeds but change the operation. Each pair of numbers in the query and key vector is treated as a point on a flat plane and rotated by an angle that grows with position, like a clock hand turned further for later words.

RoPE (one dimension pair)
rotate(x₂ᵢ, x₂ᵢ₊₁) = (x₂ᵢcos θ − x₂ᵢ₊₁sin θ, x₂ᵢsin θ + x₂ᵢ₊₁cos θ), θ = pos · base⁻²ⁱ⁄ᵈ
Each pair is rotated by an angle that grows with position, at a speed that depends on the pair, exactly like the sine and cosine speeds above.

The payoff is subtle and worth seeing for yourself. Attention scores compare a rotated query with a rotated key. When two hands are both turned, the comparison depends only on the angle between them, which is the distance between their positions, not on where they sit in the sentence. Run the code: a query at position 5 against a key at position 3, and a query at 7 against a key at 5, both a distance of 2 apart, give identical scores.

Rotating a query and a key: the score depends on the distance, not the absolute positions

The first three lines print the same number, 0.9252, even though the positions differ. The last line, at distance 3, gives a clearly different one, 0.6979. This relative-distance behavior is what attention wants, and an added pattern gives it less cleanly, which is why RoPE displaced the sinusoidal encoding.

To see this without reading code, use the demo below. Change the distance between the query and the key and the score moves. Then slide the whole pair along the sentence: both vectors spin, but the score does not change.

Rotating by Position: Only the Distance Matters

RoPE turns each query and key by an angle that grows with its position. Their dot product, the attention score, depends only on how far apart they are. Slide the pair along the sentence and the vectors spin, but the score holds still.

2
3
keyquery

Score by distance (the dot is yours)

Attention score

0.9252

distance 2

Same distance, elsewhere

0.9252

query at 2, key at 0

Query at position 5, key at position 3: distance 2, score 0.9252. The same distance at positions 2 and 0 gives 0.9252, identical. Change the distance and the score changes; change only where the pair sits and it does not.

AI Lab's Level 61 (RoPE: Rotating Instead of Adding) implements RoPE across a full (seq_len, d_model) array at once, using the same broadcasting pattern as Level 53's sinusoidal encoding. Only the final rotation step differs.

Interview Signal is part of Pro

See a real weak answer next to a real strong one for this exact topic.

Quiz is part of Pro

Test what you just read with a short quiz, and bank the XP.

Ready to Build This?

Implement all three from scratch: Level 51 (Causal Masking), Level 53 (Sinusoidal Positional Encoding), and Level 61 (RoPE) in AI Lab.

Discussion0

Join the Discussion

Sign in to leave comments, reply to others, or like insights.

Sign In to ScaleDojo

No comments yet. Be the first to start the thread!