A model that works in a demo is not a feature that works at a customer. The gap between the two is mostly not about the model: it is about who may see which document, where data is allowed to go, how anyone knows the answers are good, what each answer costs, and who is paged at night. This guide is a checklist of five gates, in the order a customer's security, legal and operations teams tend to ask about them. Last reviewed in October 2026.
Gate 1: Access
If your feature searches the customer's documents, it must only answer from documents the person asking could already open. A company wiki with an HR folder and a finance folder becomes a leak the day a search box can quote both. The fix is to carry the user's permissions into retrieval, not to filter the answer afterward. Then test it: ask as a user without access and confirm nothing from the restricted folder appears, even paraphrased.
Also decide what happens when someone leaves or loses access. The answer should stop including those documents the same day, which means permissions are read at question time, not copied once into an index.
Gate 2: Data
Write down, with the customer, what data goes to the model provider and what must not. Names, account numbers, health or payment details are common candidates for masking before anything leaves their network. Settle where the model runs (region, provider, or inside their own cloud), how long prompts and outputs are kept, and whether the provider may train on them. These are not engineering questions, but you are the person who has to ask them, and an answer in writing saves a stalled launch. For the basics, see Privacy and Data Protection Basics.
Gate 3: Quality
"It looks good" is not a quality bar. Build a small set of real questions with the people who know the right answers, agree on what counts as a pass, and run it every time something changes. A customer who helped write the test believes the result. Two habits make answers safer: show the source for every claim, and let the system say it does not know instead of guessing.
- Start with 30 to 50 real questions, including some where the right answer is "it is not in the documents."
- Score faithfulness (does the answer match the source) separately from relevance (did it answer the question).
- Keep a list of failures the customer cares about most, such as a wrong policy or a wrong price, and test those every time.
The Learn course goes deeper: Why AI Needs Evals, Building Eval Datasets and Evaluating RAG Faithfulness and Relevance. The post LLM Evals and Guardrails covers the same ground for design interviews.
Gate 4: Cost and speed
Agree on two numbers before launch: what a request may cost and how long an answer may take. Price depends on tokens, and output tokens cost more than input tokens. As an illustration, a request that reads 1,800 tokens of documents and writes 400 tokens of answer, at illustrative prices of $3 and $15 per million tokens, costs about $0.012 once retrieval and other overhead are added. At 20,000 requests a day that is $240 a day, or $7,200 a month.
Set a monthly cap and a plan for what happens at the cap: slow down, answer from a cheaper model, or stop and tell the user. See Cost Controls and Budget Guardrails for the options.
Gate 5: Operations
Before launch, answer four questions in writing. Who is paged when answers get worse or the bill spikes? How do you turn the feature off without a deploy? Where do you see what users asked and what the system answered, within the privacy rules you agreed in gate 2? How does the customer tell you an answer was wrong, and who reads that? Observability for LLM Systems and Production-Grade Guardrails are good next reads.
Check your own launch
The checklist below turns the gates into ten checks. Items marked "must" block a launch, and the calculator shows the monthly bill from the numbers above. Change the requests or the price and watch the figure move.
A readiness check. Which items block a launch is our judgment, a starting point to adjust with your customer.
A quick way to use the check: run it in the room with the customer's security lead, not alone. Every unticked box is a conversation you would rather have before launch than after. If a "must" item cannot be ticked yet, do not argue it away. Write down who will tick it and by when, and let the launch date follow that answer.
After launch
- Start with a small group, then widen. Watch the failure list, not just the average score.
- Add every wrong answer a user reports to the eval set. That is how the set stays honest.
- Re-run the evals whenever the model, the prompt, the documents or the retrieval settings change.
- Tell the product team what you learned. A reusable check or connector is worth more than one more custom fix.
What we know and what we do not
The five gates are our own framework, built from common practice in deploying retrieval-based features; they are not a published method of any company. The prices and volumes are illustrative, and a real provider's pricing, a customer's rules and regulated industries (health, finance) will add requirements this list does not cover. Treat it as the start of a conversation with the customer's security and legal teams, never as a replacement for it. Last reviewed in October 2026.
