vLLM, PagedAttention, and Serving LLMs at Scale

Training Is Hard. Serving Is Harder.

Everyone talks about training large language models. The massive datasets, the thousands of GPUs, the months of compute time. But fewer people talk about the equally hard problem that comes after: serving those models to millions of users with acceptable latency and cost.

Training happens once (or a handful of times). Serving happens billions of times. A model that takes 10 seconds to respond and costs $0.03 per query is fine for a demo. At a million queries per day, that is $30,000 daily and users staring at loading spinners. The economics and the user experience both demand efficient serving.

The Memory Bottleneck

The bottleneck in LLM serving is not compute. It is memory. Understanding why requires knowing what happens during inference.

A 7 billion parameter model in FP16 precision requires 14GB of GPU memory just for the model weights. Those weights are the same for every request. But during inference, each request ALSO needs memory for its KV cache: the stored attention key-value pairs that accumulate as the model generates tokens.

Here is the problem. With a 4,096 token context window, a single request's KV cache can consume 2 to 4GB of GPU memory. If you are serving 10 concurrent users, you need 30 to 50GB just for KV caches, on top of the 14GB for model weights. An 80GB A100 GPU fills up fast. You can serve maybe 10 to 15 concurrent users per GPU, which means you need a lot of expensive GPUs to handle real traffic.

Worse, the memory management was terrible. Traditional serving systems allocated a fixed block of memory for each request's maximum possible context length, even if the request only used a fraction of it. A request with a 100-token response got the same memory allocation as one with a 4,096-token response. This waste of memory meant 60 to 80 percent of GPU memory sat unused at any given time.

PagedAttention: Virtual Memory for LLMs

In June 2023, researchers at UC Berkeley published vLLM with PagedAttention. The insight came from a surprisingly old source: operating systems. In the 1960s, computer scientists solved the RAM fragmentation problem with virtual memory. Physical memory was divided into fixed-size pages that could be mapped flexibly to virtual addresses. Programs got the illusion of contiguous memory even though the physical pages were scattered.

PagedAttention applies exactly the same idea to the KV cache. Instead of allocating one contiguous block of GPU memory for each request, it divides the KV cache into fixed-size pages (typically 16 tokens each). Pages can be non-contiguous in GPU memory, allocated on demand as the response grows, and freed immediately when a request finishes.

This simple change has enormous implications:

• No more over-allocation: memory is allocated page by page as the response grows. A 100-token response uses memory for 100 tokens, not 4,096.
• No more fragmentation: pages are fixed-size, so freed pages can be reused immediately by other requests. No wasted gaps.
• Memory sharing: requests with the same system prompt can share the KV cache pages for the prompt portion. If 100 users hit the same chatbot, the system prompt's KV cache is stored once and shared, not duplicated 100 times.
• Efficient memory usage goes from 20 to 40 percent up to 90+ percent, meaning the same GPU serves 2 to 4 times as many concurrent users.

The results were dramatic. vLLM achieved 2 to 4x higher throughput than previous serving solutions like HuggingFace's text-generation-inference. Same hardware, same model, multiple times more users served. For companies spending millions on GPU infrastructure, this was transformative.

Continuous Batching: Never Waste a GPU Cycle

PagedAttention solved the memory problem. Continuous batching solved the compute problem.

Traditional batching (static batching) works like this: collect 8 requests, process them as a batch, wait for ALL of them to finish generating, then accept the next batch. The problem is that LLM requests have wildly different output lengths. One request might generate 10 tokens in 100ms. Another might generate 500 tokens in 5 seconds. With static batching, the short request finishes and then sits idle, wasting GPU cycles, until the longest request in the batch completes.

Continuous batching (also called iteration-level batching) takes a different approach. Instead of processing requests in fixed batches, it operates at the individual token level. Each iteration generates one token for every active request. When a short request finishes (hits its stop token), a new request immediately takes its slot in the batch. The GPU is never idle waiting for the slowest request.

Combined with PagedAttention, continuous batching can increase serving throughput by another 2 to 3x. The two techniques are complementary: PagedAttention makes memory allocation efficient, and continuous batching makes compute utilization efficient. Together, they can deliver 5 to 10x improvement over naive serving approaches.

Key Serving Metrics

When evaluating LLM serving performance, two metrics matter most:

• Time to First Token (TTFT): how long the user waits before seeing the first word of the response. This is the perceived latency. Users start reading immediately when TTFT is low, making even long generations feel fast.
• Tokens per second (TPS): how fast the model generates subsequent tokens. This determines how quickly the full response appears. Streaming responses at 30+ tokens per second feels like real-time writing.

These two metrics often trade off against each other. Techniques that improve throughput (serving more users) can increase TTFT (each individual user waits longer). Finding the right balance depends on your application. A chatbot needs low TTFT. A batch processing pipeline cares about total throughput.

The Serving Landscape

Several production-ready serving frameworks have emerged:

• vLLM: the gold standard for open-source serving. PagedAttention, continuous batching, speculative decoding, and support for dozens of model architectures. The default choice for most self-hosted deployments.
• TGI (Text Generation Inference by HuggingFace): production-ready serving with a focus on ease of use. Great Docker-based deployment. Used by HuggingFace's Inference Endpoints.
• TensorRT-LLM (NVIDIA): maximum performance on NVIDIA hardware through deep optimization. More complex to set up but achieves the highest throughput on supported GPUs.
• SGLang: a newer framework focused on structured generation (JSON output, constrained decoding). Growing in popularity for applications that need guaranteed output formats.

For API-based models (OpenAI, Anthropic, Google), the provider handles all of this for you. But you still need to manage concurrency, rate limiting, fallback strategies, and cost. Understanding how serving works helps you make better decisions even when using managed APIs.

The lesson from vLLM: production AI is as much about systems engineering as it is about model architecture. A better serving system can be more impactful than a better model. The best model in the world is useless if you cannot serve it efficiently.

What This Means for GenAI Engineers

When you deploy GenAI applications, serving infrastructure matters as much as model selection. A pipeline that makes 3 sequential LLM calls has 3x the latency of one with a single call. Adding a semantic cache eliminates LLM calls entirely for repeated queries, reducing both latency (cached responses are instant) and cost (no inference compute needed).

The key trade-offs you will face in production:

• Model size vs. latency: larger models give better answers but take longer to respond. Is the quality improvement worth the latency increase for your use case?
• Batching vs. responsiveness: larger batches improve throughput but increase individual request latency. Interactive applications need smaller batches.
• Caching vs. freshness: cached responses are instant and free, but they might be stale. How dynamic is your data?
• Self-hosted vs. API: self-hosting gives you control and can be cheaper at scale, but requires infrastructure expertise. APIs are simpler but more expensive per query.

Try It in the Lab

Acts 4 and 5 cover production concerns. When the latency gauge turns red, think about what vLLM teaches: the solution is often architectural (caching, batching, parallelization) rather than switching to a faster model. The lab's scoring system rewards these production-aware design decisions.

Further Reading

• Kwon et al. (2023): 'Efficient Memory Management for Large Language Model Serving with PagedAttention'
• vLLM documentation and benchmarks (vllm.ai)
• Anyscale blog on continuous batching
• NVIDIA TensorRT-LLM documentation
• Leviathan et al. (2023): 'Fast Inference from Transformers via Speculative Decoding'