Two Teams, One Architecture, Opposite Directions
In 2018, two teams independently realized that the Transformer architecture had a superpower nobody fully appreciated: pre-training. OpenAI took the decoder half and trained GPT-1 to predict the next word. Google took the encoder half and trained BERT to fill in masked words. Both showed that pre-training on massive amounts of unlabeled text, then fine-tuning on a small labeled dataset, dramatically outperformed training from scratch. This insight, that you can learn language structure first and specialize later. Became the foundation of modern NLP.
GPT-1: Predict the Next Word
GPT-1 (Generative Pre-trained Transformer) was deceptively simple. Take the Transformer decoder, pre-train it on a huge corpus of books (BooksCorpus, about 800 million words), and the training objective is just: given all the words so far, predict the next word. That is it. No labels, no human annotation, no task-specific design.
The model reads 'The cat sat on the' and learns to predict 'mat' (or 'floor' or 'couch'. Any likely next word). It reads 'In 1969, Apollo 11 successfully' and learns to predict 'landed.' Through billions of these predictions, the model learns grammar, facts, reasoning patterns, and stylistic conventions. All from the structure of raw text.
After pre-training, GPT-1 was fine-tuned on specific tasks: sentiment analysis, textual entailment, question answering. Fine-tuning meant adding a small task-specific layer and training on labeled examples for a few epochs. The pre-trained weights provided such a strong starting point that even a few hundred labeled examples were enough to achieve competitive performance.
GPT-1 had 117 million parameters. Large for 2018, trivial by today's standards. But it proved the concept: language modeling as pre-training transfers to downstream tasks. The paper's title said it all: 'Improving Language Understanding by Generative Pre-Training.'
BERT: The Bidirectional Breakthrough
Google's BERT (Bidirectional Encoder Representations from Transformers) took a different approach. Instead of predicting the next word (left-to-right), BERT randomly masks 15% of words in the input and trains the model to predict them from both sides. This is called Masked Language Modeling (MLM).
For example, given 'The [MASK] sat on the mat,' BERT learns to predict 'cat' (or 'dog' or 'child') using context from both left ('The') and right ('sat on the mat'). This bidirectional context was BERT's key advantage. GPT-1 could only use left context because it was trained to generate left-to-right.
BERT also used a second pre-training objective: Next Sentence Prediction (NSP). Given two sentences, predict whether the second sentence actually follows the first in the original text. This taught the model about discourse structure and coherence between sentences. Later research showed NSP was less important than MLM, and subsequent models like RoBERTa dropped it.
BERT came in two sizes: BERT-base (110 million parameters, 12 layers) and BERT-large (340 million parameters, 24 layers). It achieved state-of-the-art results on 11 NLP benchmarks simultaneously, including question answering (SQuAD), sentiment analysis, and natural language inference. Google integrated BERT into Search in 2019, calling it 'the biggest leap forward in search in five years.'
GPT-1 and BERT proved the same insight from opposite directions: if you train a Transformer on enough text, it learns the structure of language well enough that task-specific fine-tuning becomes almost trivially easy.
Encoder vs Decoder: When to Use Which
GPT-1 (decoder-only) and BERT (encoder-only) optimized for different use cases, and understanding this split matters for GenAI engineers:
- Encoder models (BERT, RoBERTa, DeBERTa): best for understanding tasks where you need to analyze existing text. This includes classification, NER, semantic similarity, and embeddings for search. They see the full input bidirectionally.
- Decoder models (GPT series, LLaMA, Mistral): best for generation tasks where you need to produce new text. This includes chatbots, code generation, and creative writing. They generate one token at a time, left to right.
- Encoder-decoder models (T5, BART): handle both understanding and generation. Think summarization, translation, and question answering with generation.
In modern RAG pipelines, you typically use an encoder model for creating embeddings (to store and search documents) and a decoder model for generation (to produce the final answer). The embedding model understands your documents; the LLM generates responses. Both descended from the same 2018 moment.
What This Means for GenAI Engineers
The pre-training revolution directly shapes how you build AI systems today.
First, the embedding models in your RAG pipeline are BERT descendants. When you choose text-embedding-3-small for your pipeline, you are using a model that was pre-trained on massive text corpora using a masked language modeling objective (or a similar variant). It learned what words mean by predicting masked words from context. Bengio's 2003 idea, scaled up through BERT's architecture.
Second, the LLMs you use for generation are GPT descendants. GPT-4, Claude, LLaMA. All are decoder-only Transformers pre-trained to predict the next token, exactly like GPT-1 but with more parameters and more data. The training objective has not changed since 2018. What changed is scale.
Third, fine-tuning is the bridge between pre-training and your specific use case. When companies fine-tune LLaMA on their proprietary data, they are doing exactly what GPT-1 demonstrated: take a pre-trained model and adapt it with a small amount of task-specific data. The GenAI Pipeline Lab's later acts cover this pattern in detail.
Try It in the Lab
Notice how your pipeline uses two different model types: an EmbeddingModel (encoder, BERT lineage) for search and an LLM (decoder, GPT lineage) for generation. These are not the same kind of model doing the same thing. They are two sides of the 2018 split, each optimized for its role. Understanding this helps you choose and configure them correctly.
Further Reading
- Radford et al. (2018): 'Improving Language Understanding by Generative Pre-Training,' the GPT-1 paper
- Devlin et al. (2018): 'BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding'
- Jay Alammar's 'The Illustrated BERT, ELMo, and co.,' visual guide to pre-trained language models
- Liu et al. (2019): 'RoBERTa: A Robustly Optimized BERT Pretraining Approach,' showed BERT was undertrained
- Google AI Blog: 'Understanding searches better than ever before,' BERT in Google Search