Back-of-the-Envelope Math for AI Systems

"How many GPUs would this need?" is a favorite question in AI design interviews and a daily one in real planning. Nobody expects an exact answer. They want to see a clear chain of reasoning from users to dollars, with stated assumptions and a sanity check. This chapter gives you that chain and a worked example.

To decide how many cooks to hire, you do not guess. You estimate how many guests arrive at the busiest hour, how long each meal takes to cook, and how busy you want your cooks to be. Multiply, divide, round up. Sizing an AI service is the same chain with different units: requests instead of guests, GPU-seconds instead of cook-minutes.

• -Users to requests: daily users times requests per user.
• -Requests to rate: divide by seconds in a day for the average, then multiply by a peak factor for the busiest moment.
• -Tokens per request: input tokens (system prompt, history, retrieved context) and output tokens. Count them separately, since they cost differently.
• -GPU time per request: prefill time for the input plus decode time for the output.
• -GPUs at peak: rate times GPU time per request, divided by the utilization you are willing to run at.
• -Dollars: GPUs times hours times price per GPU-hour. Divide by requests for cost per thousand requests.

One more tool: Little's law says the average number of things in a system equals the rate they arrive at times how long each stays. If 278 requests arrive each second and each takes 6 seconds to stream, about 278 × 6 = 1,668 requests are in flight at once. That number tells you how much memory the KV cache needs, as in the earlier chapter on memory math, and how many connections your servers must keep open.

The block below sizes a chat product with one million daily users who send eight messages each. Each request carries 1,500 input tokens and produces 300 output tokens. The throughput of one GPU is an assumption for an 8B model with continuous batching: 2,500 output tokens per second and 15,000 prompt tokens per second. Real numbers must be measured on your own stack. Press Run, then change a number and run again.

Eight million requests a day is an average of 93 per second and a peak of 278. Each request takes 0.220 GPU-seconds: 0.120 decoding its 300 tokens and 0.100 prefilling its 1,500. Prefill is almost half the bill even though it produces no output, which is a reminder that long prompts have a cost. At a 70% target utilization the peak needs 88 GPUs, about $190,080 a month if you keep them all running, or $0.79 per thousand requests. At the same moment, 1,667 requests are in flight.

• -Dominant term: which line of the calculation matters most? Here, input tokens and output tokens cost about the same, so reducing either one helps.
• -Memory: 1,667 requests in flight, at about 1,800 tokens each, with 128 KiB per token for a small model, is about 390 GB of KV cache, spread over the fleet. Check that your GPU count leaves room for it, since memory, not compute, may be the real limit.
• -Compare with the alternative: a hosted API charges per token. At what volume does running your own GPUs beat it, given that you must pay for idle capacity?
• -Peak versus average: sizing for the peak all day overpays. Autoscaling, a smaller overnight fleet, and a queue for batch work all cut the bill.