Two Papers That Made Machines Translate
In 2014, two papers published within months of each other transformed machine translation, and set the stage for everything leading to ChatGPT. The first, from Google, introduced the encoder-decoder architecture for sequence-to-sequence learning. The second, from Bahdanau, Cho, and Bengio, introduced the attention mechanism. Together, they solved the fundamental problem of converting one sequence (French) into another sequence (English) using neural networks.
Before these papers, machine translation relied on statistical methods: align words between languages, compute translation probabilities from huge parallel corpora, and use complex beam search to find the most likely translation. These systems were brittle, struggled with long sentences, and required extensive language-specific engineering. Seq2Seq replaced all of that with a single neural network trained end-to-end.
The Encoder-Decoder Architecture
The seq2seq idea from Sutskever, Vinyals, and Le at Google is elegant. Use one LSTM (the encoder) to read the input sequence one word at a time, compressing the entire sentence into a single fixed-length vector, a 'thought vector' that captures the meaning of the sentence. Then use a second LSTM (the decoder) to generate the output sequence one word at a time, starting from that thought vector.
Think of it like this: the encoder reads 'Je suis etudiant' and converts it into a vector that means something like [person, be, student, present tense, first person]. The decoder takes that vector and generates 'I am a student.' Neither LSTM knows anything about French or English grammar. They learn the translation mapping purely from seeing millions of example pairs.
The encoder processes the input left to right, updating its hidden state at each step. After the last input word, the final hidden state contains (in theory) everything the decoder needs to generate the translation. The decoder then generates one word at a time, feeding each generated word back as input for the next step, until it produces an end-of-sentence token.
The Bottleneck Problem
Seq2seq worked surprisingly well for short sentences. But it had a critical flaw: the entire meaning of the input sentence had to be compressed into a single fixed-length vector. For a 5-word sentence, that is fine. For a 50-word sentence, you are trying to squeeze all the nuance, word order, and relationships into maybe 256 or 512 numbers. Information gets lost. Long sentences translated poorly. Details from early in the sentence were forgotten by the time the decoder reached them.
This is the same vanishing information problem that plagued RNNs in general, just at a higher level. The LSTM's gates helped, but they could not fully compensate for the information bottleneck of a fixed-size vector.
Asking a fixed-length vector to represent any sentence of any length is like asking a single Post-it note to summarize an entire book. It works for short books, but important details inevitably get lost.
Attention: Looking Back at the Input
Bahdanau, Cho, and Bengio's attention mechanism solved the bottleneck elegantly. Instead of forcing the entire input into a single vector, let the decoder look back at all the encoder hidden states at every decoding step and focus on the most relevant parts.
Here is how it works. The encoder processes 'Je suis etudiant' and produces a hidden state vector for each word: h1 (Je), h2 (suis), h3 (etudiant). When the decoder generates 'I,' it computes an attention score for each encoder state. How relevant is each input word for generating 'I'? It might assign high attention to h1 (Je) because 'Je' directly translates to 'I.' When generating 'student,' it assigns high attention to h3 (etudiant).
The attention scores are computed using a small neural network that takes the decoder's current state and each encoder state, producing a relevance score. These scores are normalized into a probability distribution (using softmax), and the encoder states are combined using these probabilities into a context vector. This context vector. Different at every decoding step. Is what the decoder uses to generate the next word.
Why This Is a Breakthrough
Attention solves the bottleneck because the decoder now has access to all encoder states, not just the final one. It can pull specific information from specific positions in the input. For a 100-word sentence, the decoder can focus on word 3 when translating the subject, word 47 when translating a clause reference, and word 98 when translating the final predicate. Information no longer needs to survive compression into a single vector.
The attention weights are also interpretable. You can visualize which input words the model 'looks at' when generating each output word. This creates natural word-to-word alignments between languages. Something that statistical machine translation systems had to compute separately using complex alignment models.
From Translation to Everything
Attention quickly spread beyond translation. Researchers realized it was a general mechanism for relating elements in sequences. Within a year, attention was being used for:
- Text summarization: attend to the most important sentences when generating a summary
- Question answering: attend to the relevant parts of a document when answering a question
- Image captioning: attend to different regions of an image when generating each word of the caption
- Speech recognition: attend to different parts of the audio signal when outputting each character
The key insight that would lead to transformers was this: if attention is so useful between the encoder and decoder, why not also use it within the encoder itself? Let every word in a sentence attend to every other word. This 'self-attention' idea would become the foundation of the transformer architecture in 2017. Instead of processing words one at a time (like LSTMs), self-attention processes all words simultaneously, with each word computing relevance scores against every other word.
What This Means for GenAI Engineers
The encoder-decoder architecture and attention mechanism are not just history. They are the building blocks of modern GenAI systems.
First, every transformer-based model uses attention as its core mechanism. When you use GPT-4, Claude, or Gemini, the model is computing attention scores between tokens at every layer. Understanding attention helps you understand why LLMs handle some queries well (the relevant context is clearly attended to) and struggle with others (the relevant information is buried among noise).
Second, the encoder-decoder pattern lives on in RAG pipelines. Your embedding model is an encoder. It compresses text into a fixed-length vector. Your VectorDB search is a simple form of attention. Finding which stored embeddings are most relevant to the query. And the LLM generation step is the decoder. Producing output conditioned on retrieved context. The seq2seq architecture from 2014 maps directly onto the RAG pattern from 2020.
Third, the attention bottleneck is still relevant. Modern LLMs have context windows (128K tokens for GPT-4) that are much larger than the old fixed-length vectors. But when your context window fills up with too much retrieved text, the model's attention gets diluted. It cannot focus on the relevant parts. This is why Reranker components in RAG pipelines matter: they filter the context down to the most relevant chunks so the LLM's attention can focus effectively.
Try It in the Lab
When you add a Reranker component to your GenAI pipeline, you are implementing a form of attention filtering. The Reranker scores retrieved chunks by relevance and sends only the top-K to the LLM. Giving the model a focused context instead of a diluted one. Experiment with different top-K values in Act 2 levels to see how retrieval quality affects generation quality.
Further Reading
- Sutskever, Vinyals & Le (2014): 'Sequence to Sequence Learning with Neural Networks,' the original seq2seq paper
- Bahdanau, Cho & Bengio (2014): 'Neural Machine Translation by Jointly Learning to Align and Translate,' the attention paper
- Jay Alammar's 'Visualizing A Neural Machine Translation Model,' excellent visual walkthrough of attention
- Luong et al. (2015): 'Effective Approaches to Attention-based Neural Machine Translation,' simplified attention variants
- Lilian Weng's 'Attention? Attention!' blog post, comprehensive survey of attention mechanisms