Offline & Online Evaluation for LLM Systems
You'll learn to
- -Distinguish offline eval sets from online production signals
- -Design a continuous evaluation loop
- -Compute a genuine quality-cost Pareto frontier across model candidates
A team evaluates which LLM to use by running ten queries and eyeballing the results, then makes a decision clients will live with for months. That is not an evaluation framework, it is a guess with extra steps. Tier 9 covered evaluating agent trajectories specifically. This chapter covers the broader discipline: evaluating LLM systems generally, both before and after they reach production.
Offline Evaluation: Before Anything Ships
Offline evaluation runs a fixed, curated test set through every candidate, model or prompt, before either ever reaches a real user. This is where LLM-as-judge, first introduced for RAG in Tier 8, generalizes: a powerful model scores each candidate's outputs against the same criteria, at a scale no team of human reviewers could sustain across thousands of test cases.
No Single Metric Tells the Whole Story
Comparing models purely on accuracy misses that a model can be highly accurate and prohibitively expensive, or cheap and just barely good enough. A genuine comparison needs quality and cost plotted against each other, and the models actually worth considering are the ones on the Pareto frontier: no other candidate beats them on both dimensions simultaneously.
legacy-fine-tune is the interesting result here. It is not the worst model on the list, but open-source-70b beats it on both quality and cost at once, which is exactly what "dominated" means. A model does not need to be bad in isolation to be the wrong choice. It just needs a strictly better alternative to exist.
Online Evaluation: After It Ships
Offline evaluation answers "which candidate looked best on our test set." Online evaluation answers the question that actually matters: how is the system performing against real production traffic, right now. This means continuously sampling live requests and responses, scoring a percentage of them with the same LLM-as-judge approach, and tracking that score over time, exactly the regression-detection pattern from Tier 9's agent evaluation chapter, now applied to the LLM system as a whole rather than one agent's trajectories.
Interview Signal is part of Pro
See a real weak answer next to a real strong one for this exact topic.
Quiz is part of Pro
Test what you just read with a short quiz, and bank the XP.
Build the Fine-Tuner and Evaluator levels in the GenAI Lab.