Prompting, RAG, or Fine-Tuning: Choosing the Lever

Your assistant gives wrong answers. There are three standard fixes: write better instructions, give the model the right information at request time, or train the model on examples. They solve different problems, cost different amounts, and fail differently. Picking the wrong one wastes months. This chapter gives you a way to choose.

Imagine three ways to prepare a student. You can give them clear instructions about how to answer (prompting). You can let them bring the textbook and look things up (retrieval). Or you can have them study for weeks until they have it memorized (fine-tuning). An open-book exam suits facts that change or are too many to memorize. Memorizing suits a way of writing, or a rule applied the same way every time. Clear instructions are the cheapest and are always worth trying first.

Start from error analysis, as the evals module taught, and name the kind of failure.

• -The model misunderstands the task or the format: the instructions are unclear. Fix with prompting, examples, and a clearer output schema.
• -The model lacks facts it was never trained on, or the facts change: the knowledge is missing. Fix with retrieval (RAG).
• -The model knows the facts but behaves wrongly in a consistent way, such as the wrong tone, a custom format, or a domain rule: the behavior is off. Fine-tuning is the candidate.
• -The model is right but too slow or expensive: the size is wrong. A smaller model, possibly fine-tuned to match the larger one on your task, may help.

Hamel Husain's essay on evals puts the division of labor simply: fine-tuning is best for learning syntax, style, and rules, while retrieval supplies context and up-to-date facts. A model cannot reliably memorize a changing catalog through fine-tuning, and retrieval cannot teach a model a house style.

• -Prompting first. It is free to try, fast to change, and tells you what the model can already do.
• -RAG next, when the problem is missing or changing knowledge. Retrieval also gives you citations and respects each user's access rights.
• -Fine-tuning last, once an evaluation set shows a consistent behavior gap that prompts and retrieval cannot close, and you have enough good examples.
• -Combine them. A common design is a fine-tuned smaller model, called with retrieved context and a short prompt.

• -RAG: fresh knowledge on every request, citations you can show, per-user permissions applied at retrieval time, and a document you can delete to forget it. It costs extra input tokens on every query, plus an index to run.
• -Fine-tuning: consistent behavior and format, shorter prompts, and sometimes a smaller, cheaper, faster model. It costs labeled examples, training runs, evaluation, and retraining whenever the target changes. Knowledge baked into weights cannot be cited, and cannot be selectively removed.
• -Prompting: instant and reversible, but limited by context length and by how reliably the model follows instructions.

RAG pays a little on every query, because retrieved text makes prompts longer. Fine-tuning pays a fixed amount up front and often less per query. So the cheaper choice depends on volume, and on how often the knowledge changes. The code compares the two. The prices are invented for illustration, and so are the build and hosting figures. Press Run and find, in each column, the row where the cheaper option flips.

A query costs $0.0135 with RAG on the large model and $0.00051 on the small tuned model, a gap of 26 times. At 50,000 queries a month, RAG is cheaper in total, $1,175 against $2,359 for a tuned model with stable facts, because the tuned model's fixed costs have not been earned back. At 200,000, RAG costs $3,200 while the tuned model costs $2,435 if facts are stable, but $4,435 if it must be retrained monthly. At a million queries, even the monthly-retraining tuned model is cheaper, $4,843 against $14,000. At 10 million the gap is enormous. Cost is not the only thing, since the tuned model may be less accurate on facts, which is why you check quality on your own evaluation set before switching.