Scaling Laws, Quantization & Mixture-of-Experts
You'll learn to
- -Compute a compute-optimal model size and training-set size from a budget, using C = 6ND and the Chinchilla ratio
- -Explain quantization, and run an int8 round trip to see the small error it introduces
- -Explain how a router sends each token to only a few experts, and why that makes a huge model cheap to run
- -Say what the three ideas have in common: they decide which model you end up running and what it costs
Everything so far in this module explains how a Transformer block computes its output. This chapter is about three decisions that shape which model you are running at all: how big it should be for a given training budget, how cheaply it can be served once trained, and how it can grow bigger without costing proportionally more per token. Three small formulas answer them.
Scaling Laws: Why a Model Is the Size It Is
Training a model means spending a fixed amount of computing effort. Think of it as study hours before an exam. You can spend them skimming a huge textbook once, or reading a smaller one many times. DeepMind's Chinchilla paper found, by experiment, how to split the hours best.
- -Parameters (N) are the numbers a model learns: its size, like the thickness of the textbook.
- -Tokens (D) are the pieces of text it trains on: how much it reads.
- -Compute (C) is the total arithmetic, counted in FLOPs, floating-point operations: the study hours.
Training costs about 6 floating-point operations for every parameter and every token, roughly 2 for the forward pass and 4 for the backward pass, so C is about 6 x N x D. Chinchilla's finding was that, for a fixed C, the best results come when tokens and parameters grow together at about 20 tokens per parameter. Earlier models such as GPT-3, with 175 billion parameters trained on about 300 billion tokens, saw fewer than 2 tokens per parameter. They were big but under-read, which is a large part of why smaller, longer-trained models like Llama can beat much larger older ones.
A worked case: a budget of 10^21 FLOPs. N is the square root of 10^21 / (6 x 20), about 2.9 billion parameters, and D is 20 times that, about 58 billion tokens. Run it.
Try other budgets in the demo below. Raise or lower the compute, change the tokens-per-parameter ratio, or jump to the budget used to train GPT-3, and see how a smaller model trained on more text spends the same compute.
How Big Should a Model Be for a Training Budget?
With a fixed amount of training compute you can train a bigger model on less text or a smaller one on more. Pick a budget and the tokens-per-parameter ratio, and see the split that spends it best.
Chinchilla's result is about 20.
Best model size
2.9 billion
parameters
Best training text
57.7 billion
tokens
Same budget, 175B model
0.01
tokens per parameter
A budget of 10^21.0 FLOPs buys a model of about 2.9 billion parameters trained on 57.7 billion tokens. Spending the same budget on a 175-billion-parameter model would leave it only 952 million tokens, 0.01 per parameter: large, but under-read.
Quantization: Serving a Model for a Quarter of the Memory
A model's weights are normally stored as 16 or 32 bits each. Quantization stores them as 8-bit (or even 4-bit) whole numbers instead, plus one shared scale factor to turn them back into decimals. It is like storing prices in whole cents instead of fractions of a cent: a tiny rounding error in return for much less space. For a 70-billion-parameter model, the weights alone take about 140 GB at 16 bits, 70 GB at 8 bits and 35 GB at 4 bits.
Take four weights: 0.12, -0.5, 0.33 and 0.99. The scale is 0.99 / 127, about 0.0078. Dividing and rounding gives 15, -64, 42 and 127. Multiplying back gives 0.1169, -0.4989, 0.3274 and 0.99, and the largest error is about 0.003. Methods such as GPTQ and AWQ choose the roundings more cleverly to protect accuracy, especially at 4 bits, but the core idea is exactly this.
The demo below uses the same four weights. Lower the number of bits and watch the stored whole numbers get coarser and the rounding error grow, which is the trade every quantization method is managing.
Store Weights in Fewer Bits, Lose a Little Precision
Quantization replaces each weight with a whole number and one shared scale factor. Choose how many bits each weight gets and watch the rounding error grow as the steps get coarser.
| Original | Stored | Restored | Error |
|---|---|---|---|
| 0.12 | 15 | 0.1169 | 0.0031 |
| -0.50 | -64 | -0.4989 | 0.0011 |
| 0.33 | 42 | 0.3274 | 0.0026 |
| 0.99 | 127 | 0.9900 | 0.0000 |
Scale factor
0.0078
largest weight maps to the top number
Largest error
0.0031
70B model weights
70 GB
16-bit would be 140 GB
At 8 bits the largest error is 0.0031, tiny next to weights of this size. A 70-billion-parameter model would shrink to about 70 GB, half of its 16-bit size.
Mixture-of-Experts: A Bigger Model at the Same Per-Token Cost
Picture a hospital with eight specialist doctors and a triage nurse at the door. Each patient is sent to the two specialists most relevant to them, not to all eight. The hospital has enormous total expertise, but each patient uses only a small part of it. A Mixture-of-Experts (MoE) layer works the same way.
- -The experts are many separate sub-networks inside one layer.
- -The router is the triage nurse: a small network that scores every expert for the current token.
- -Top-k gating keeps only the k highest scores and turns them into shares with softmax. Every other expert gets a weight of exactly 0.
- -An expert with weight 0 is never even run for that token, so compute grows with k, not with the total number of experts.
In the example the router scores four experts [1, 5, 3, 2]. With k = 2, only experts 1 and 2 survive, with gate weights 0.88 and 0.12. DeepSeek-V3 uses this idea at scale: it has over 600 billion parameters in total (671 billion) but activates only about 37 billion per token, around 5.5 percent. That is a huge model at the running cost of a much smaller one.
AI Lab's Levels 63-65 (the Chinchilla Scaling Law, 8-bit Quantization and Mixture-of-Experts Routing) implement all three of these in numpy: solving the Chinchilla equations directly, real int8 quantize and dequantize round-tripping, and vectorized top-k gating across a whole batch of tokens at once.
Interview Signal is part of Pro
See a real weak answer next to a real strong one for this exact topic.
Quiz is part of Pro
Test what you just read with a short quiz, and bank the XP.
Implement all three: Level 63 (Chinchilla Scaling Law), Level 64 (8-bit Quantization), and Level 65 (Mixture-of-Experts Routing) in AI Lab.
Discussion0
Join the Discussion
Sign in to leave comments, reply to others, or like insights.
No comments yet. Be the first to start the thread!