Skip to content
Training Compute-Optimal Large Language Models 4 min

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark · NeurIPS 2022

Training Compute-Optimal Large Language Models

The Chinchilla paper: with a fixed compute budget, model size and training data should grow together. See why a 70B model beat a 280B one.

Read the original paper

AI & ML · Black Belt · Pro · 4 min read

Training Compute-Optimal Large Language Models

In 2020 a rule had settled in the large-language-model world: make the model bigger and it gets better. GPT-3 had 175 billion parameters. DeepMind's Gopher had 280 billion. Megatron-Turing NLG had 530 billion. Training these giants took enormous amounts of computing, and everyone assumed the answer to "how do we do better?" was "build an even bigger model."

A DeepMind team asked a sharper question. Suppose you have a fixed amount of computing to spend. How should you split it between making the model bigger and training it on more text? Their answer, backed by more than 400 training runs, was that nearly everyone had been getting the split wrong.

For a given compute budget, model size and training data should grow together. Double the model, double the tokens. Most big models were far too large and trained on far too little text. Chinchilla, a 70-billion-parameter model trained on 1.4 trillion tokens, beat the much bigger Gopher.

The question: where should the compute go?

Training compute is roughly proportional to the number of parameters N multiplied by the number of training tokens D. The paper uses the common estimate:

FLOPs≈6×N×D\text{FLOPs} \approx 6 \times N \times D

So with a fixed budget, you can pick a bigger N and fewer tokens, or a smaller N and more tokens. Every point on that trade-off costs the same to train. An earlier study, by Kaplan and colleagues in 2020, found that model size mattered most, so as budgets grew, the field made models larger while keeping the data around 300 billion tokens.

From the paper's Table 1. Before Chinchilla, big models were trained on about 300 billion tokens regardless of size. Chinchilla has a quarter of Gopher's parameters and four times the data, for about the same compute.

The Chinchilla team trained over 400 models, from 70 million to over 16 billion parameters, on 5 to 500 billion tokens. They analyzed the results in three different ways. All three gave the same message: the best model size and the best number of tokens should be scaled equally. For every doubling of model size, double the training tokens.

Keep reading with Pro

This paper is part of Pro. Pro and Architect open the full breakdown, with animated diagrams, interactive elements you can play with and a quiz. A few papers stay free to read in full.

Still to read

2 more sections, 1 more diagram and 2 interactive elements to play with

  • Proof by building a model
  • What changed after this paper

Found this breakdown useful?

Share it with someone else wrestling with this paper.

Discussion0

Join the Discussion

Sign in to leave comments, reply to others, or like insights.

Sign In to ScaleDojo

No comments yet. Be the first to start the thread!

Enjoyed this? Get more like it.

New paper breakdowns, levels, and one concept worth knowing, straight to your inbox.

No spam, ever. Unsubscribe in one click.

Enjoyed this content?

Your support keeps us creating free resources

We put a lot of hours into researching and writing these guides. If it helped you, consider buying us a coffee. Every bit goes toward keeping ScaleDojo's content free and growing.

$

One-time payment via Stripe. ScaleDojo account required.