Capstone: The Inference Bill

You are handed a bill. A chat product serves 8 million requests a day, and it needs 88 GPUs at peak, the plan from the back-of-the-envelope chapter. The finance team asks for it to be cut by a large factor without hurting quality. This capstone applies the techniques from the whole module as a ladder, one step at a time, and reads each step's saving and its risk.

You do not cut a household budget by one dramatic move. You renegotiate the phone plan, stop paying for subscriptions you do not use, cook at home more often, and buy in bulk. Each step saves a little, the savings multiply, and each has a cost in convenience. Inference cost reduction is the same: several modest improvements that compound.

• -Baseline: 16-bit weights, no caching, plain decoding. This is the 88-GPU plan.
• -8-bit weights: half the bytes to read per token, so decoding gets faster, assumed here at 1.7 times in practice.
• -Prompt caching: 1,200 of the 1,500 input tokens are a shared system prompt, so only 300 need prefilling.
• -Speculative decoding: assumed 1.8 times faster decoding for typical text.
• -Routing: send 60% of requests, the easy ones, to a small model that is assumed to be four times cheaper per token, and the rest to the big one.

The multipliers are assumptions, not measurements, and each must be tested on your workload. The code applies the steps in order and prints the GPUs and cost after each. Press Run and look at the last column, the share of the baseline cost.

The baseline needs 88 GPUs, $190,080 a month. 8-bit weights bring it to 68 GPUs, 77% of the baseline. Prompt caching is the largest single step, to 36 GPUs and 41%, because it removes most of the prefill work. Speculative decoding takes it to 24 GPUs and 27%, and routing to a small model for easy requests ends at 14 GPUs and $30,240, 16% of where it began.

• -8-bit weights: a small quality loss is possible. Run the evaluation set before and after, with attention to the cases that matter most.
• -Prompt caching: it only helps with a stable prefix, so the saving depends on prompt design and on how often the cache hits. Measure the hit rate, and do not assume it.
• -Speculative decoding: the gain depends on how predictable the text is, and it needs a draft model that matches the big one. The output is unchanged, but the saving varies by workload.
• -Routing: a router must decide which requests are easy. Misrouting hard questions to the small model hurts quality, so evaluate the router itself and keep a fallback to the big model.

• -Autoscaling to the daily curve, so you do not pay for peak capacity overnight.
• -Batch or offline processing, which can use idle GPUs and cheaper terms for work that is not time-sensitive.
• -Shorter outputs and prompts. Asking for concise answers cuts decode time directly.
• -Caching whole responses for repeated questions, as in the semantic cache chapter.
• -Buying versus renting, and comparing with hosted API prices, which may be cheaper at low volume.