Critiquing an AI System Design

Interviewers often hand you a design and ask what is wrong with it. Teams do the same in design reviews. Critiquing a design is a different skill from creating one, and it is one you can practice: look through a fixed set of lenses, ask pointed questions, and rank what you find. This chapter gives you the lenses and a small tool to practice with.

An inspector does not rebuild the house. They walk through with a checklist: wiring, plumbing, structure, exits. They find the problems a builder, focused on finishing, would miss, and they rank them, because a missing handrail and a cracked foundation are not the same. A design review is an inspection.

• -Requirements fit: does the design solve the stated problem at the stated scale, or a different one? What is missing from the goal and the metric?
• -Data and knowledge: where does it come from, how fresh is it, who may see it, and what happens when it is wrong or missing?
• -Evaluation: how will they know it works, and that a change did not make it worse? Is there a baseline, an eval set, and an online experiment?
• -Failure and safety: what happens on a wrong answer, an outage, abuse, prompt injection, or private data in a prompt?
• -Cost and operations: what does it cost per request and at peak, what are the limits and fallbacks, and what is monitored, including quality?

The code represents a proposed customer support chatbot as data, and a small function that checks it against a list of common gaps. It is a toy, but it makes the point that a review can be systematic. Press Run and read the findings, then think about which you would fix first.

Nine findings, and each one maps to a lens. The first two, no baseline and no eval set, belong to the evaluation lens, and they come first, because without them you cannot tell whether any other change helps. The retrieval gaps (vector-only, no reranker) are quality fixes. The prompt injection and personal data findings are safety and privacy risks that could cause real harm, so they are high priority even though they would not show up in a demo. The fallback, cost controls, and quality monitoring are the operational gaps that decide whether the system survives its first bad day.

• -Fix what can cause harm or loss first: privacy, safety, security, and anything irreversible.
• -Then what blocks learning: no evals, no baseline, no monitoring of quality.
• -Then what hurts quality: retrieval and prompt improvements.
• -Then what hurts cost and polish: caching, routing, and fine-tuning.

• -Ask clarifying questions, to learn whether the point is about requirements or about the design.
• -Acknowledge a good point plainly, and say what you would change. Do not defend a gap.
• -If you disagree, give your reasoning and the trade-off, and say what evidence would change your mind.
• -Never claim a problem does not exist because you have not seen it. Say how you would measure it.