How to Approach Any GenAI System Design

Every worked example in this closing tier, and every real GenAI system design interview you will ever face, is built from the exact same set of pieces this course already covered: tokens and context in Tier 7, retrieval in Tier 8, agents and tools in Tier 9, and production concerns in Tier 10. This chapter is the framework that turns that pile of pieces into a repeatable process for approaching a system you have never seen before.

A GenAI system design question is almost always deliberately underspecified, the same way a general system design question is. Resist the urge to start drawing boxes. Ask first.

• -Scale: how many users, how many requests per day, how much data. A 50-document RAG system and a 2-million-document one are not the same design, even though they use identical components.
• -Latency and freshness: does an answer need to reflect something that happened five minutes ago, or is a day of staleness fine? This alone decides whether caching, from Tier 10, is even usable.
• -Budget: is this a cost-sensitive, high-volume consumer product, or a lower-volume, quality-first enterprise tool? This decides model tier and how aggressively to route to cheaper models.
• -Safety criticality: what happens if this system is confidently wrong? A trivia bot and a medical or financial system have wildly different tolerances, and that tolerance should shape every guardrail decision that follows.

Almost every GenAI system need maps onto one or more of four core capabilities, and naming which ones apply immediately narrows the design space enormously.

Once the core capabilities are named, the pipeline is largely an assembly problem, not an invention problem. Need retrieval? Document ingestion, chunking, embedding, vector storage, hybrid search, reranking, all from Tier 8, in that order. Need agency? Tool schemas, a ReAct or plan-and-execute loop, memory, from Tier 9. This is precisely why the framework matters: it turns "design a system" into "identify which of the patterns I already know apply here, and in what combination."

This is the step that separates a system that would actually survive production from a diagram that only looks complete. Every one of these applies regardless of which core capabilities are involved, and every one of them was covered in depth in Tier 10.

• -Safety: guardrails scaled to the stakes identified in step 1, not a single default sensitivity for every system.
• -Cost: model tiering, caching, and budget guardrails, especially for anything running at real user scale.
• -Resilience: failover between providers, graceful degradation, never a single point of failure for anything user-facing.
• -Observability: traces, metrics, and specifically a quality signal, the one dimension traditional monitoring cannot see on its own.
• -Evaluation: how you would actually know, with a number and not a vibe, whether this system is working.

Every remaining chapter in this tier applies exactly this framework to a real GenAI Lab level, twice per chapter. None of them introduce a new component or a new mechanic. Every single piece used from here forward was already covered, in depth, verified, and built by hand somewhere earlier in this course. What is new is only the composition.