Speeding Up Decoding: Speculation & Prefix Caching

Decoding is slow because each token needs a full pass through a big model, and that pass is limited by memory. If you cannot make one pass faster, you can try to get more than one token out of each pass. And if the start of every prompt is the same, you can avoid redoing that work. These are two of the most effective tricks in serving.

A senior editor is slow and careful. A junior assistant is quick, and usually right about the obvious next words. Instead of the editor writing every word, the assistant drafts the next several words, and the editor reads the whole draft at once and keeps everything up to the first mistake. Checking several words in one glance costs about the same as writing one. That is speculative decoding.

• -A small, fast draft model proposes the next γ tokens (γ is the number of guesses).
• -The big model checks all of them in one parallel pass, which costs about the same as producing one token, because the pass is memory-bound.
• -The system keeps the proposed tokens up to the first one the big model disagrees with, and uses the big model's own token at that point.
• -Repeat from there.

Leviathan, Kalman, and Matias introduced the method in 2022 and reported a 2 to 3 times speedup on a T5-XXL model, with outputs identical to what the big model would have produced alone. The acceptance rule is built so that the distribution of outputs does not change, so this is a speedup without a quality trade. The size of the gain depends on two things: how often the draft's guess is accepted, and how cheap the draft is.

The code evaluates the formula for a draft model that costs 5% as much as the big model, across acceptance rates and guess lengths. The second part shows prompt caching, which is the next section. Press Run and, for each row, find where the speedup peaks.

When the draft is right half the time, the speedup never exceeds about 1.6 times, and it peaks at four guesses ahead. At 70% it reaches 2.35 times, at 80% about 3.1 times, and at 90% over 4 times. More guesses only pay when they are likely to be accepted, because a wrong guess wastes everything after it. Real acceptance rates depend on the text. Code, templates, and structured output are predictable, and the draft does well. Open-ended creative writing at high temperature is not, and the gain is smaller. The result in the table also assumes a spare, well-matched draft model, which has to be hosted and kept in sync.

Use the demo below to find the best number of guesses. Raise the chance each guess is accepted and the best length grows, or make the draft expensive and the gain disappears.

Many requests start with the same text: a long system prompt, a set of examples, a document that several questions refer to. The KV cache for that shared start is identical every time, so a server can compute it once and reuse it. This is called prefix caching, or prompt caching. In the second part of the output, a 1,500-token system prompt in front of a 100-token user message means prefill drops from 1,600 tokens to 100, 94% less, once the prefix is cached.

• -It helps when requests share a long, stable prefix, and it cuts both time to first token and prefill cost.
• -The prefix must match exactly. A single changed token, such as a timestamp near the top of the prompt, invalidates the cache from that point on.
• -Put stable content first (instructions, examples, documents) and variable content last (the user's message).
• -Hosted APIs usually offer cached tokens at a discount and for a limited time. Check the current terms, since they change.

Storing weights in 8 or 4 bits cuts the bytes read per token, which speeds up the memory-bound decode phase and lets a model fit on fewer GPUs. The cost is a possible drop in quality, usually small at 8 bits and larger at 4 bits or in sensitive tasks. Measure it on your own evaluation set, as the earlier module on evals explains, and do not assume. Quantized models also need hardware that supports the lower precision to realize the full speedup.