Model Serving Infrastructure Basics

A company serving Llama-3.1-70B for enterprise clients gets 5 requests per second on a $30,000 GPU. A competitor claims 50 on the exact same hardware, ten times the throughput. Neither number is about the model itself. It is entirely about how the serving layer schedules work onto the GPU, and getting this layer wrong is one of the most expensive mistakes in production GenAI, since it is pure wasted hardware spend.

Calling a model directly in a loop, one request fully finishes before the next one starts, badly underuses an expensive GPU, since most of the time is spent waiting rather than computing. A real model server (vLLM and TGI are the two names worth knowing) exists specifically to keep the GPU busy: batching multiple requests together, managing memory carefully, and scheduling work so the hardware is rarely sitting idle.

The naive approach, static batching, waits for a fixed-size batch of requests to accumulate, runs them together, and only starts the next batch once every single request in the current one has finished. That last part is the problem: one unusually long generation in the batch holds the entire batch, and the whole GPU, hostage until it finishes, even while every other slot in that batch sits idle waiting.

Adjust the slot count below and watch two real scheduling algorithms, run on the same 30 requests, diverge.

Continuous batching (sometimes called in-flight batching) fixes exactly this. The instant any single request in the batch finishes, a new request immediately takes its slot, rather than waiting for every other slot to finish too. No request is ever blocked by a stranger's long generation. This alone is typically a 3 to 5x throughput improvement over static batching on the same hardware.

Batching solves scheduling. The other constraint is GPU memory, specifically the KV cache, the growing memory each in-progress generation needs to hold onto as it produces more tokens. vLLM's signature technique, PagedAttention, manages this memory the way an operating system manages virtual memory: allocating it in small fixed-size pages rather than one large contiguous block per request, so memory is not wasted reserving worst-case space up front. This is what lets a single GPU hold meaningfully more concurrent requests in flight at once.