Prefill vs Decode: Two Phases, Two Bottlenecks

An LLM answer does not arrive at a steady pace. There is a pause, then words stream out one after another. Those two behaviors come from two different phases of the same computation, and they are limited by two different parts of the hardware. Knowing which phase is hurting is the first skill in LLM serving.

When a long letter arrives, you take in the whole page at once. Your eyes and mind process many words in parallel, and the limit is how fast you can think. Then you write the reply one word at a time, and for each word you must hold the whole letter in mind. Reading is prefill. Writing is decode.

• -Prefill: the model processes every token of the prompt in parallel, builds the internal memory it will need (the KV cache, covered in the next chapter), and produces the first output token. How long this takes is the time to first token, or TTFT.
• -Decode: the model generates the rest one token at a time. Each new token needs a full pass through the model. How long each token takes is the time per output token, or TPOT, also called inter-token latency.

A response's total time is roughly TTFT plus TPOT multiplied by the number of tokens generated. A long prompt hurts TTFT, and a long answer is paid for token by token. Products care about different ones. A chat window cares about a low TTFT and a TPOT faster than reading speed. A summarizer of long documents may tolerate a slow TTFT but wants cheap prefill.

Picture a chef. If the chef has a mountain of onions to chop and the pantry is next to the counter, the limit is how fast their hands move: that is compute-bound. If the chef needs one pinch of saffron from a pantry across the building for every dish, the limit is the walking: that is memory-bound. A GPU is the same. It has an enormous number of calculations per second and a large but finite rate at which it can read memory.

• -Prefill is compute-bound. Thousands of prompt tokens are processed together, so each weight read from memory is used for thousands of multiplications.
• -Decode is memory-bound. Each step handles one token per request, so almost every weight must be read from memory to do just a small amount of math before it is discarded.

This yields a useful rule. In decode, speed is roughly the memory bandwidth divided by the bytes of the weights, because each token requires reading all of them once. In prefill, time is roughly the total floating point operations divided by the GPU's peak math speed, with about 2 operations per parameter per token. The code below applies both rules to an H100, which has about 3.35 terabytes per second of memory bandwidth and about 989 trillion dense 16-bit operations per second. Press Run and read the decode column down the rows.

For the 8B model with 16-bit weights, the ceiling for one request is about 209 tokens per second, since each token reads 16 GB. Halving the bytes with 8-bit weights roughly doubles it to 419, and 4-bit doubles it again to 838. The 70B model has 141 GB of 16-bit weights, so it needs at least two GPUs just to hold them, and still tops out near 47 tokens per second. Prefill behaves differently: a 2,000-token prompt costs about 32 ms of peak math on the 8B model, and quantizing the weights does not change that column in this simple model, because prefill is limited by arithmetic. The prefill column assumes the 16-bit peak for every row, and real systems fall short of all of these ceilings, so treat them as upper bounds.

Because decoding is limited by reading the weights, reading them once and using them for many requests at the same time is nearly free. A batch of 32 requests takes almost as long per step as a batch of 1, but produces 32 tokens. The last lines of the output show how far this goes: with 16-bit weights, a GPU only becomes compute-bound when the batch reaches roughly 295 requests. That is why serving systems batch aggressively, why the earlier chapter on continuous batching matters, and why a lightly loaded GPU is an expensive way to serve.

The demo below turns this arithmetic into controls. Pick a model, change how its weights are stored, add users to the batch, and see the speed each user gets, the total output, and whether it all fits in memory.