How AI System Design Differs From Classic System Design

If you have worked through the system design fundamentals, you already know how to scale a service: load balancers, caches, queues, replicas, shards. Those skills do not expire. But adding an AI component changes the kind of problem you are solving in ways that surprise experienced engineers. This chapter names the differences, so the chapters that follow have something to hang on.

A bridge is engineered to a specification. You calculate loads, pour concrete, inspect it, and it behaves the way the calculations said. A garden is different. You plant, water, and tend it, and what grows depends on soil, weather, and pests you cannot fully control. You judge a garden by looking at it regularly, and you adjust. A classic service is a bridge. An AI system is a garden: its behavior is probabilistic, shaped by the data you feed it, and it changes over time.

• -The output is probabilistic. The same input can produce different outputs, and "correct" is often a matter of degree. You test by sampling and statistics, not by one assertion.
• -Quality is a distribution, not a status code. A wrong answer arrives fast, confident, and well formatted, with a 200 OK. Failures are silent unless you measure quality on purpose.
• -Data is a dependency. The training set, the retrieval corpus, and the labels are part of the system, and they must be versioned, validated, and monitored like code.
• -Cost per request varies. A classic API call costs about the same each time. An LLM call costs by the token, and prompts and answers vary widely in length.
• -The latency profile differs. There is a time to first token and a time per token, GPU queues, and long tails, as the inference module showed.
• -Feedback loops appear. A recommender decides what users see, so it shapes the data it learns from next.
• -The world drifts. User behavior changes, and a model provider may update a model under you, so a system that passed its tests last month may degrade without a code change.

What does not change is as important. Capacity planning, caching, queues, retries, security, privacy, and data modeling all still apply, and often matter more. An AI feature is a service with an unusual, expensive, unreliable dependency, and designing around that dependency is most of the work.

Consider cost first, because it is the difference that shows up on the invoice. The code simulates 20,000 LLM requests in which most prompts are short and a few are very long, priced with illustrative per-token rates, and compares them with an ordinary API call. Press Run, and read the distribution from the median to the most costly request.

The median LLM request costs $0.0071, about 18 times an ordinary API call at $0.0004. The 99th percentile is $0.0292, four times the median, and the most costly request is $0.0912, about 13 times the median. The costliest 10% of requests account for 25% of the bill. Nothing here is dramatic, but it changes habits. You set a maximum for output tokens and prompt length, give each user a budget, watch the distribution of cost and not only the average, and treat one unusually long conversation as a capacity event, not a curiosity.

In a classic design, tests come after the architecture. In an AI design, the evaluation plan shapes the architecture: what you can measure decides what you can improve. Before the diagram, ask what a good answer is, how you will know, and what a bad one costs. The evals module covers the mechanics. In an interview, saying this early marks you as someone who has run these systems.

• -That you clarify the goal, scale, and constraints before drawing anything.
• -That you choose a metric and a baseline, and can say how you would know the system works.
• -That you think about failure: wrong answers, drift, abuse, outages, and cost blowups.
• -That you reason about trade-offs with numbers, not slogans.
• -That you keep the design as simple as the problem allows.