What Happens When You Make It Bigger?
After GPT-1 proved that pre-training works, the obvious question was: what happens if we make the model bigger and train it on more data? OpenAI's answer was GPT-2 (2019) and GPT-3 (2020), and the results surprised everyone. Including the researchers who built them.
GPT-2: Too Dangerous to Release (Or So They Said)
GPT-2 scaled GPT-1 from 117 million to 1.5 billion parameters and trained on WebText. 8 million web pages (about 40GB of text) curated from Reddit links with at least 3 upvotes. The training objective was identical to GPT-1: predict the next word. Nothing else changed architecturally.
The results were qualitatively different. GPT-2 could generate coherent multi-paragraph text on any topic. Give it a prompt about unicorns and it would write a plausible news article. Give it a math problem and it would sometimes solve it. Give it Python code and it would complete the function. None of these abilities were explicitly trained. They emerged from next-word prediction at scale.
OpenAI initially refused to release the full model, claiming it was 'too dangerous' due to potential misuse for generating fake news and spam. This was controversial. Critics called it a publicity stunt, and the staged release (releasing larger versions over several months) generated massive media attention. Regardless of the PR strategy, GPT-2 demonstrated that scale unlocked qualitatively new capabilities.
GPT-3: Few-Shot Learning Without Fine-Tuning
GPT-3 was an order of magnitude larger: 175 billion parameters, trained on a filtered version of Common Crawl plus books and Wikipedia (about 570GB of text). Training cost an estimated $4.6 million in compute.
GPT-3's most surprising capability was few-shot learning. Instead of fine-tuning on task-specific data (like GPT-1 required), you could just show GPT-3 a few examples in the prompt and it would learn the pattern:
Translate English to French:
sea otter => loutre de mer
peppermint => menthe poivree
cheese => GPT-3 would output 'fromage' without ever being explicitly trained on translation. This in-context learning ability. Learning new tasks from examples provided at inference time, with no weight updates. Was GPT-3's defining contribution. It meant a single model could handle thousands of tasks without any task-specific training.
The paper coined three levels of prompting: zero-shot (just describe the task), one-shot (one example), and few-shot (a handful of examples). Each required no gradient updates, the model adapted purely from the prompt context. This discovery fundamentally changed how people thought about using language models.
GPT-2 showed that scale produces emergent capabilities. GPT-3 showed that at sufficient scale, you don't even need to fine-tune, the model learns from examples in the prompt.
Scaling Laws: Predicting Before Training
In 2020, Kaplan et al. at OpenAI published 'Scaling Laws for Neural Language Models,' which showed that model performance improves as a predictable power law of three variables: the number of parameters (N), the size of the training dataset (D), and the amount of compute (C). The relationship was smooth and predictable across several orders of magnitude.
This was enormously significant for practical reasons. It meant you could predict how well a model would perform before spending millions of dollars training it. You could plot the scaling curve from smaller experiments and extrapolate. This turned model development from 'train it and hope' into 'compute the expected performance and decide if the investment is worth it.'
Chinchilla: The Scaling Laws Were Wrong (Sort Of)
In 2022, DeepMind's Chinchilla paper revealed that the original scaling laws had a blind spot. OpenAI's scaling laws suggested making models bigger was the primary lever. Chinchilla showed that the optimal strategy is to scale model size and training data equally. A 70 billion parameter model trained on 1.4 trillion tokens (Chinchilla) outperformed the 280 billion parameter Gopher trained on 300 billion tokens.
The practical implication: most models were undertrained relative to their size. You get more performance per dollar by training a smaller model on more data than by training a bigger model on less data. This finding influenced every subsequent model architecture decision. LLaMA (Meta, 2023) was explicitly designed following Chinchilla-optimal ratios.
What This Means for GenAI Engineers
Scaling laws are not academic curiosities. They directly affect your daily decisions.
First, model selection in RAG pipelines involves the same trade-off. GPT-4o is bigger and smarter. GPT-4o-mini is smaller, cheaper, and faster. For most RAG queries where the answer is directly in the retrieved context, the smaller model performs nearly as well. The scaling laws tell you that the capability gap narrows for tasks that do not require deep reasoning.
Second, few-shot prompting is a core technique in GenAI applications. When you design a PromptTemplate in your pipeline, including 2-3 examples of the desired output format is a direct application of GPT-3's few-shot learning discovery. The model adapts its behavior based on the examples in the prompt, with no retraining required.
Third, Chinchilla's insight about data efficiency matters for fine-tuning. If you are fine-tuning a model on your company's data, you will often get better results from training longer on your existing data (with augmentation) than from using a larger base model with less training.
Try It in the Lab
The cost-quality trade-off in the pipeline lab directly reflects scaling laws. A pipeline using GPT-4o might score higher on quality but blow the budget. Switching to GPT-4o-mini and adding a Router to escalate only complex queries mirrors real-world patterns where engineers use Chinchilla-optimal thinking: spend compute where it matters, save where it doesn't.
Further Reading
- Radford et al. (2019): 'Language Models are Unsupervised Multitask Learners,' the GPT-2 paper
- Brown et al. (2020): 'Language Models are Few-Shot Learners,' the GPT-3 paper
- Kaplan et al. (2020): 'Scaling Laws for Neural Language Models'
- Hoffmann et al. (2022): 'Training Compute-Optimal Large Language Models,' the Chinchilla paper
- Gwern Branwen's 'The Scaling Hypothesis' essay, excellent long-form analysis of what scaling means for AI
