Eight Engineers, One Paper, a New Era
In June 2017, eight researchers at Google published a paper with a deceptively simple title: 'Attention Is All You Need.' The paper proposed a new architecture called the Transformer that replaced the dominant LSTM-based approach with a mechanism based entirely on attention. Within two years, it would become the foundation of GPT, BERT, and every large language model that followed. Within five years, it would underpin the most significant wave of AI products in history.
The Transformer was not an incremental improvement. It was a complete architectural rethinking that solved the LSTM's biggest problems while enabling capabilities no one anticipated. To understand why it matters, you need to understand what it replaced and what it does differently.
What Was Wrong With LSTMs
By 2017, LSTMs were the standard for sequence processing. Translation, summarization, question answering. But they had two fundamental limitations:
First, sequential processing. An LSTM reads words one at a time, left to right. To process word 100, it first needs to process words 1 through 99. This means you cannot parallelize across the sequence, each step waits for the previous one. On modern GPUs that can do thousands of operations simultaneously, this is a massive waste. Training an LSTM on a long document uses a tiny fraction of the GPU's capacity.
Second, long-range dependencies. Despite the LSTM's gating mechanism, information from early in a sequence degrades as it passes through more and more time steps. In a 500-word document, the LSTM's representation of word 1 has been transformed through 499 gating operations by the time it reaches the end. Important details get diluted. Self-attention solves this by letting any word directly attend to any other word, regardless of distance.
Self-Attention: Every Word Looks at Every Word
The core innovation of the Transformer is self-attention. Instead of processing words sequentially, self-attention lets every word in a sequence compute a relationship score with every other word, all at once.
Here is how it works. Each word is represented as a vector (its embedding). For each word, the Transformer computes three new vectors:
- Query (Q): 'What am I looking for?'
- Key (K): 'What do I contain?'
- Value (V): 'What information do I provide?'
For each word, the attention score between that word and every other word is computed by taking the dot product of its Query vector with every other word's Key vector. High dot product = high relevance. These scores are normalized (softmax) into a probability distribution, then used to take a weighted sum of all Value vectors. The result is a new representation of each word that incorporates context from every other word in the sequence.
Consider the sentence: 'The animal didn't cross the street because it was too tired.' What does 'it' refer to? Self-attention computes that 'it' has a high attention score with 'animal' and a low score with 'street.' The output vector for 'it' will therefore contain information from 'animal,' effectively resolving the reference. An LSTM might or might not propagate this connection, depending on the distance and the learned gate values.
The Math (Simplified)
The attention formula: Attention(Q, K, V) = softmax(QK^T / sqrt(d_k)) * V. That is it. Q, K, and V are matrices where each row is a word's query, key, or value vector. QK^T computes all pairwise dot products in a single matrix multiplication. Division by sqrt(d_k) prevents the dot products from getting too large (which would make the softmax too peaky). The softmax turns scores into probabilities. Multiplying by V produces the weighted sum of values.
The beauty is that this is entirely matrix multiplication, and GPUs are designed for matrix multiplication. Unlike LSTMs, where you process one time step at a time, self-attention processes the entire sequence in a handful of matrix multiplies. This makes Transformers dramatically faster to train on GPUs.
Multi-Head Attention: Looking at Multiple Things at Once
A single self-attention operation captures one type of relationship between words. But language has many simultaneous relationships. Syntactic (subject-verb agreement), semantic (synonyms and antonyms), positional (nearby words), and referential (pronoun resolution). Multi-head attention runs multiple self-attention operations in parallel, each with different learned Q, K, V transformations.
The original Transformer used 8 attention heads. Each head learns to focus on different types of relationships. One head might learn to attend to the previous word. Another might learn to attend to the verb associated with each noun. A third might focus on co-reference (matching pronouns to their referents). The outputs of all heads are concatenated and linearly projected back to the model dimension.
In practice, researchers found that different heads do specialize, though not always in easily interpretable ways. Some heads attend to positional patterns (the word immediately before), while others attend to semantic patterns (words with related meanings regardless of position). This diversity is part of why Transformers are so effective. They capture multiple aspects of language simultaneously.
Positional Encoding: Teaching Position Without Recurrence
Self-attention has a problem: it treats all words as a bag of tokens with no inherent order. The sentence 'dog bites man' and 'man bites dog' produce the same attention scores because the attention mechanism is permutation-invariant. Word order matters in language, so the Transformer needs a way to inject positional information.
The solution: add a positional encoding vector to each word embedding before feeding it into the attention layers. The original Transformer used sinusoidal functions at different frequencies:
PE(pos, 2i) = sin(pos / 10000^(2i/d_model)), PE(pos, 2i+1) = cos(pos / 10000^(2i/d_model))
Each position gets a unique encoding, and the relative distance between positions can be computed from the encodings. The encoding uses different frequencies so that the model can distinguish both nearby positions (high-frequency components) and distant positions (low-frequency components). Think of it like a clock: the second hand tells you about short time scales, the hour hand about longer ones.
Modern Transformers (including GPT-4) use learned positional embeddings or rotary position embeddings (RoPE) instead of sinusoidal, but the principle is the same: inject position information so the model knows word order.
The Full Architecture
The complete Transformer has two halves:
- Encoder: 6 identical layers, each containing multi-head self-attention followed by a feed-forward neural network. Every word attends to every other word in the input. Used for understanding.
- Decoder: 6 identical layers with masked self-attention (each word can only attend to earlier words, so no peeking at the future), cross-attention to the encoder output, and a feed-forward network. Used for generation.
Each layer also has residual connections (adding the input to the output) and layer normalization, both borrowed from earlier work. Residual connections solve the vanishing gradient problem by letting gradients flow directly through the addition operation. Layer normalization stabilizes training.
The 'base' Transformer had 65 million parameters. By modern standards, that is tiny. GPT-4 is estimated at over 1 trillion. But the architecture scaled in ways no one expected.
The Transformer replaced sequential processing with parallel attention, recurrence with position embeddings, and handcrafted features with learned representations. It was simpler than what came before, faster to train, and better at the task. That is a rare combination.
Why This Changed Everything
The Transformer was not just a better architecture for translation. It became the universal architecture for AI. Within three years:
- BERT (2018): encoder-only Transformer for understanding tasks (classification, Q&A, NER)
- GPT-1 (2018): decoder-only Transformer for text generation
- GPT-2 (2019): larger decoder Transformer that generated surprisingly coherent text
- GPT-3 (2020): 175 billion parameter Transformer that could do few-shot learning
- Vision Transformer (2020): Transformers applied to images, matching or beating CNNs
- DALL-E (2021): Transformers generating images from text descriptions
- GPT-4 (2023): multi-modal Transformer processing text and images
The Transformer succeeded because it scaled. More parameters, more data, more compute, and the model kept getting better. LSTMs did not scale this way. Doubling the parameters of an LSTM did not reliably double its capability. Transformers exhibited predictable scaling laws: performance improved as a smooth power law of model size and training data. This predictability made it rational to invest billions of dollars in building bigger Transformers, because you could predict the return.
What This Means for GenAI Engineers
Every LLM you use in production is a Transformer. Understanding the architecture helps in several practical ways.
Context windows and attention: when you configure a PromptTemplate in a RAG pipeline, you are deciding what goes into the Transformer's attention mechanism. Too much context dilutes attention, the model struggles to find the relevant information among noise. Too little context starves the model of information. The art of prompt engineering is partly about curating what the attention mechanism sees.
Token limits and chunking: Transformers have quadratic attention complexity. Computing attention over N tokens requires N^2 operations. This is why context windows have limits. When you configure a Chunker in your pipeline, you are working within this constraint. Smaller chunks mean each one fits in the context window with room for the query and instructions. Larger chunks provide more context but leave less room.
Model selection trade-offs: different LLMs use different Transformer configurations. More attention heads capture more relationship types but cost more compute. More layers give deeper reasoning but increase latency. When you choose between GPT-4o and GPT-4o-mini in the pipeline lab, you are choosing between different points on this trade-off curve.
Try It in the Lab
Every component in the GenAI Pipeline Lab connects to the Transformer architecture. The EmbeddingModel creates the input representations. The Chunker manages what fits in the attention window. The PromptTemplate structures what the decoder attends to. The LLM is the Transformer itself. Understanding this end-to-end connection helps you design pipelines that work with the architecture instead of against it.
Further Reading
- Vaswani et al. (2017): 'Attention Is All You Need,' the original Transformer paper
- Jay Alammar's 'The Illustrated Transformer,' the single best visual explanation, widely considered essential reading
- Lilian Weng's 'The Transformer Family' blog post, comprehensive overview of Transformer variants
- Andrej Karpathy's 'Let's build GPT from scratch' YouTube lecture. Builds a Transformer step by step in code
- Harvard NLP's 'The Annotated Transformer,' the paper re-implemented line by line in PyTorch