You draw the retriever, the model, the cache and the queue. Then the interviewer asks two questions that sound simple. How do you know it works? And what stops it from saying something harmful? The usual answer is "we would add some tests and a content filter." It is true, and it earns almost nothing, because it names no mechanism, no number and no owner.
This post gives you the mechanism: three layers and one loop, a checklist you can fill in for any AI system, and a 90-second answer you can say out loud.
Why this layer gets skipped
It is not a box in the architecture diagram, so it feels like a job for QA and not for design. For systems built on language models that instinct is wrong. A normal service returns the right value or it does not, and a failing test says so. A language model returns fluent text whether it is right or not, and nothing fails to compile when it is wrong. The evaluation and safety layer is what tells you whether you can ship, and whether you can change a prompt, a model or a retriever next month without breaking things.
Three layers, one loop
Think of a restaurant kitchen. Tasting a dish before it joins the menu is offline evals. A rule that no dish with nuts leaves the pass without a flag is a guardrail. Reading the customers' comment cards is online monitoring. And when a card describes a problem, it becomes a new dish to taste before the next menu change.
- Offline evals run before you ship, and again on every change.
- Guardrails run on every request, in the live path.
- Online monitoring runs after you ship and watches real traffic.
- The loop: a failure found online becomes a new test case offline, so the same mistake is caught before the next release.
Layer 1: offline evals
- A golden set. A fixed collection of realistic questions, each with the expected outcome: a reference answer, the right source document, the right tool call, or the right refusal. Start with 20 to 50 cases that real users actually send. Grow it to a few hundred, and include the hard and adversarial cases, not only the easy ones.
- Cheap, exact checks first. Does the output match the schema? Does it cite a source? Did the SQL run? Did it call the right tool with valid arguments? These are fast, free and never disagree with themselves.
- A model as judge for the fuzzy parts. Is the answer faithful to the retrieved text? Is it relevant and polite? An LLM judge is practical, but it has known biases: it tends to favor longer answers, answers shown first, and its own style, as the MT-Bench study reports. Calibrate it against a sample of human labels before you trust it.
- People on a sample. A small, regular human review keeps the judge honest and finds failures nobody wrote a test for.
- A regression gate. Run the set on every change to a prompt, model, retriever or tool. Block the release if the success rate drops by more than an agreed amount, or if any critical case fails.
What you measure depends on the system: task success, faithfulness to sources, retrieval recall and precision for RAG, tool-call accuracy for agents, whether it refused when it should, and latency and cost per request. Evaluating RAG shows how four separate scores point at the failing stage.
Layer 2: guardrails in the live path
| Where | What it checks | Example | Typical cost |
|---|---|---|---|
| Input | Who is asking and what: authentication, rate limits, personal data, injection and jailbreak screening, scope | Redact card numbers before they reach the model. Refuse questions outside the product's scope | A rule takes microseconds, a classifier tens of milliseconds |
| Retrieval | What the caller is allowed to see | Filter documents by the caller's permissions at query time, never by asking the model to hide them | A metadata filter, almost free |
| Model and tools | What the model is allowed to do | A tool allowlist, validated arguments, read-only by default, human approval for refunds or emails, step and spend caps | Approval costs minutes, but only for the rare risky action |
| Output | What leaves the system | A schema check, a citation and grounding check, a policy classifier, a personal-data check, a safe "I don't know" | A classifier tens of milliseconds, a model-based check several hundred |
The costs are typical orders of magnitude, not a benchmark. Measure your own. Four principles carry most of the weight:
- Deterministic first, model-based second. A schema check or a pattern is predictable. Use a model-based check only for what rules cannot express.
- Enforce rules outside the prompt. A sentence in the system prompt asking the model to behave is a request, not a control. Permissions, caps and approvals belong in code.
- Fail closed for high-risk actions, and fail open, with a warning, for low-risk ones.
- Budget the latency. Guardrails add time, so run independent checks in parallel with generation, or only on the risky paths.
OWASP keeps a list of the top risks for LLM applications, with prompt injection at the head of it, which makes a handy checklist for the input and output rows. Guardrails & Content Safety Basics and Production-Grade Guardrails cover the building blocks.
Layer 3: online monitoring
- Traces. One record per request: the input, the retrieved sources, the tool calls, the output, tokens, latency and cost. Redact personal data before you store it.
- Quality signals. Thumbs up and down, regenerate and escalate-to-human rates, refusal rate, and a small percentage of traffic sampled for review.
- Operational signals. Latency at the 95th percentile, error rate, cost per request and cache hit rate.
- Drift. The questions users ask change over time, and a provider may change a model underneath you. Watch the mix of topics, and re-run the golden set on a schedule.
- Safe rollout. Send prompt and model changes to a small slice of traffic first, a canary, or run them in shadow mode where answers are scored but not shown.
- Close the loop. Every thumbs-down worth reading becomes a case in the golden set.
Learn covers this in Observability for LLM Systems and Offline & Online Evaluation.
A worked example: a refund-support assistant
A support assistant can look up orders and, with approval, issue refunds. Here is the three-layer answer, with illustrative numbers.
- Offline. A golden set of 120 cases: 80 ordinary questions, 25 adversarial ones (injection attempts, requests for other customers' data) and 15 refund-policy edge cases. The release gate: at least 90% task success overall, and 100% on the 10 cases marked critical, such as never refunding above the policy limit.
- Guardrails. Card numbers redacted on the way in. Retrieval filtered to the caller's own orders. The refund tool is the only write tool, and anything above $50 needs a person to approve it. Answers must cite a policy passage or say that they cannot find one.
- Online. Every request traced. 2% of conversations sampled daily for review. An alert when the thumbs-down rate passes 8% over an hour, or when refunds per hour leave their usual range. Each reviewed failure is added to the golden set.
Notice the pattern. Every number is something you can measure, and every control has an owner.
What to say in 90 seconds
I would add three layers around the model. Before shipping, offline evals: a golden set of real questions with expected outcomes, exact checks first, an LLM judge calibrated against human labels, and a gate that blocks a release if success drops or any critical case fails. In the live path, guardrails: redact sensitive input, filter retrieval by permission, restrict tools and require approval for risky actions, and validate and ground the output, with the rules enforced in code and not only in the prompt. After shipping, monitoring: traces, sampled review, thumbs-down and cost alerts, and a canary for every prompt or model change. Failures we find online become new test cases, so the system gets safer every month. The cost is some latency and a team that owns the set, and I would budget for both.
Say it slowly, and stop to give a number when the interviewer pulls on a thread. The builder below drafts the same answer from the controls your own design has.
Build your own checklist
Choose the kind of system you are designing and tick what it already has. The builder shows your coverage by layer, what to add first, and a draft of what to say.
A checklist builder. Weights are our judgment: they differ by system type, because a document Q&A tool has no write actions to approve.
Check your instincts
Six failures, three layers. Which layer should catch each one first?
Pick a layer for each failure. The explanation names the best first line of defense.
Common mistakes
- Grading with another model and no people. The judge drifts and shares the blind spots of the system it grades.
- A golden set of easy cases. A set the system passes on day one measures nothing.
- Guardrails that live only in the prompt. The prompt is a request. Permissions, caps and approvals have to be enforced in code.
- A report with no gate. If a falling score does not block a release, nobody reads it.
- No owner. Someone has to add cases, read samples and answer the alerts.
- Counting a refusal as a success. Measure whether it refused when it should, and whether it answered when it should have.
Where to practice
In the GenAI Lab, build The RAG Evaluator, The Agent Evaluator, The Guard Tower, The Observability Stack and The A/B Tester. In the HLD Lab, add an evaluation and safety layer to your design for Level 50, the Enterprise RAG Pipeline and Level 52, the Enterprise AI Gateway.
The companion post Design an Agentic RAG System covers the architecture that this layer wraps, and System Design Interviews in 2026 puts both in the context of what companies say about their interviews.