Capstone: Scoping an AI Design Question

The first five minutes decide the rest of the answer. A good scope turns a vague request into numbers and constraints that you can design against. This capstone practices that on a realistic prompt: "Design an assistant that answers employees' questions using our internal documents." The same moves work for most AI design questions.

An architect does not start drawing when a client says "I want a house." They ask how many people will live there, what the budget is, what the land is like, and what matters most. The answers fix the shape of everything that follows. Scoping an AI system is the same conversation, with different questions.

• -Who are the users and how many? This sets throughput and cost, and whether to optimize for latency or quality.
• -How big and how varied is the content, and how fast does it change? This sets the index size, ingestion, and refresh design.
• -Who is allowed to see what? Access control in retrieval can be the hardest requirement of all.
• -What does a wrong answer cost, and what is acceptable? This sets how much to invest in citations, evals, and human fallback.
• -What are the latency and availability expectations, and the budget? These bound the model size and the architecture.
• -Where does it live: a web page, a chat tool, an API? This sets the integration and the streaming needs.
• -What exists today, and how will we know it is better? This gives the baseline and the metric.

Suppose the interviewer answers: 2 million documents averaging 1,500 words, 10,000 employees asking about 6 questions each per working day, answers that cite their sources, and access limited to what each person may read. The code converts those facts into a corpus size, an index size, a request rate, and a monthly model cost, using illustrative prices. Press Run, then change a number and see which estimate moves.

The corpus is about 4 billion tokens, or 10 million chunks of 400 tokens. At 1,024 dimensions in 32-bit floats the vector index is about 41 GB, which fits in the memory of one large machine, so sharding is not the first problem. The system sees 60,000 questions a day, a peak of about 6 per second, which is modest. Model cost is about $16,830 a month. What jumps out is that the 3,000 tokens of retrieved context per question drive most of the cost, so a reranker that sends fewer, better chunks would save real money. None of this took a diagram, and it already shows where to look.

• -Assumption: each question needs about 3,000 tokens of context. If your retrieval is weak, it will need more, and cost grows.
• -Assumption: the 10 million chunks are mostly static. Heavy daily change would make ingestion a bigger part of the design.
• -Risk: access control. A chunk a person cannot see must never reach the model, so permissions are filtered during retrieval, not afterward.
• -Risk: wrong answers with fake confidence. Citations, an eval set, and a way to say "I do not know" are part of version one.
• -Risk: stale documents. Plan a refresh path and show document dates.

Before building more, collect 50 real questions with their correct source documents, and measure retrieval hit rate at 5, then judge answer faithfulness. That small eval set, from the evals module, tells you whether to invest in better chunking, a reranker, or a different model. Say this in the interview. It turns a design into a plan.