Decision Cheat Sheets & Where to Go Next

This is the last chapter of the module, and the one meant to be reopened before an interview or a design review. It compresses the decisions from the design, inference, and evals chapters into short lists. None of it is new. Each line points back to a chapter where the reasoning lives, so use it to jog your memory, not to replace the reasoning.

A pilot knows how to fly, and still carries a card with key speeds and procedures, because under pressure the right number should be one glance away. These lists are that card. If a line surprises you, go back and read the chapter.

• -Rare positive class, like fraud: precision and recall at a chosen threshold or review capacity, with a cost-weighted view.
• -Ranked results, like search and recommendation: NDCG at k, recall at k for retrieval, and a click or satisfaction metric online.
• -A number used as a probability, like click rate: log loss and calibration, not only AUC.
• -Estimates with asymmetric cost, like ETA: a quantile objective and error by slice.
• -Free-form LLM output: task-specific checks, and a validated judge with true positive and true negative rates.

• -Words differ between question and document: query rewriting, hybrid search.
• -A chunk loses meaning alone: contextual retrieval or parent-child chunks.
• -Right chunk retrieved but ranked low: a reranker.
• -Nothing relevant retrieved at all: look at ingestion, chunking, and embeddings, since reranking cannot help.
• -Results ignore obvious constraints: metadata filters.

• -Unclear instructions or format: prompting and examples.
• -Missing or changing knowledge, or a need for citations: RAG.
• -Consistent behavior, style, or format gap that prompts cannot close: fine-tuning, after an eval shows it.
• -Too slow or too costly but accurate: a smaller model, routing, caching.

• -Checkable by rule, such as a format, a number, or a database state: code.
• -Depends on meaning and many wordings are valid: a model judge, validated against human labels.
• -High stakes, still discovering what good means, or checking a judge: people.

• -Want to see behavior with no risk: shadow mode.
• -Want to test on real users carefully: a canary, then widen.
• -Want an instant switch back: blue-green.
• -Want to know whether it improves a user metric: an A/B test, with a sample ratio check and guardrails.

• -Slow time to first token: shorten or cache the prompt prefix, use chunked prefill, check queueing.
• -Slow tokens after the first: reduce bytes per token with quantization, batch more, try speculative decoding.
• -Running out of memory: grouped-query attention, a paged cache, a quantized cache, a shorter context.
• -Too costly: caching, routing easy requests to a smaller model, shorter outputs, autoscaling.
• -Model too large for one GPU: tensor parallelism inside a server.

• -You can write the steps down: a workflow, with checks.
• -The path cannot be predicted: an agent, with a step limit, a budget, narrow permissions, and human approval for risky actions.

Rather than list them, the code computes them from the formulas in the earlier chapters, so you can check your memory and see where each comes from. All hardware figures are peaks, and prices are not included, since they change. Press Run and read each line with its chapter in mind.

Each line stands for a chapter. The decode ceiling is the memory-bandwidth argument from the prefill and decode chapter. The two cache sizes come from the memory math chapter and explain why long contexts limit how many users fit on a GPU. Little's law sizes concurrency. The chain multiplication is why agent steps need checks. The Elo line is the arena chapter. The margin of error is the noise chapter. The last line is why downsampled ad models need a correction. If any number is unfamiliar, that is your reading list.

• -Practice out loud: pick a question from each family, set a timer, and run the eight-step framework. Then record yourself and read the transcript for red flags.
• -Build a small eval for something you own: read 50 real outputs, name the failure modes, and write one code check and one validated judge.
• -Revisit the quizzes with spaced repetition, so the numbers and trade-offs stay fresh.
• -Read the primary sources behind this phase: Google's Rules of Machine Learning, Chip Huyen's books on ML systems and AI engineering, Eugene Yan's essays on production LLM systems and recommender design, Hamel Husain's essays on evals, and the engineering posts from Anthropic on agents, evals, and context.
• -Build and measure: stand up a retrieval pipeline over your own documents, evaluate it with hit rate and MRR, and improve it one change at a time.