In 2020 a rule had settled in the large-language-model world: make the model bigger and it gets better. GPT-3 had 175 billion parameters. DeepMind's Gopher had 280 billion. Megatron-Turing NLG had 530 billion. Training these giants took enormous amounts of computing, and everyone assumed the answer to "how do we do better?" was "build an even bigger model."
A DeepMind team asked a sharper question. Suppose you have a fixed amount of computing to spend. How should you split it between making the model bigger and training it on more text? Their answer, backed by more than 400 training runs, was that nearly everyone had been getting the split wrong.
For a given compute budget, model size and training data should grow together. Double the model, double the tokens. Most big models were far too large and trained on far too little text. Chinchilla, a 70-billion-parameter model trained on 1.4 trillion tokens, beat the much bigger Gopher.
The question: where should the compute go?
Training compute is roughly proportional to the number of parameters N multiplied by the number of training tokens D. The paper uses the common estimate:
So with a fixed budget, you can pick a bigger N and fewer tokens, or a smaller N and more tokens. Every point on that trade-off costs the same to train. An earlier study, by Kaplan and colleagues in 2020, found that model size mattered most, so as budgets grew, the field made models larger while keeping the data around 300 billion tokens.
The Chinchilla team trained over 400 models, from 70 million to over 16 billion parameters, on 5 to 500 billion tokens. They analyzed the results in three different ways. All three gave the same message: the best model size and the best number of tokens should be scaled equally. For every doubling of model size, double the training tokens.
