Offline & Online Evaluation for LLM Systems
You'll learn to
- -Separate offline evaluation (a fixed test set before launch) from online evaluation (live traffic after launch)
- -Compare candidate models on both quality and cost with a Pareto frontier
- -Detect a quality regression after launch with a simple moving average
A team picks an LLM by running ten queries and eyeballing the results, then makes a decision clients will live with for months. That is not an evaluation framework, it is a guess with extra steps. The earlier tier covered evaluating agent trajectories, and the Evals modules just before this tier go deep on building a test set and trusting a judge. This chapter takes the production view of the broader discipline: comparing candidates, and watching a system both before and after it reaches production.
An Analogy: The Kitchen Taste Test and the Customer Reviews
A restaurant tests a new dish in the kitchen before putting it on the menu: that is offline evaluation, with a fixed set of testers and criteria. After it opens, the restaurant also reads real customers' reviews and watches what comes back uneaten: that is online evaluation. A dish can pass the kitchen test and still disappoint real diners, and a supplier can change the recipe quietly later.
Offline Evaluation: Before Anything Ships
Offline evaluation runs a fixed, curated test set through every candidate, model or prompt, before any of them reaches a real user. This is where LLM-as-judge, introduced for RAG in an earlier tier, generalizes. A strong model scores each candidate's outputs against the same criteria, at a scale no team of human reviewers could keep up across thousands of test cases.
No Single Metric Tells the Whole Story
Comparing models on accuracy alone misses that a model can be highly accurate and prohibitively expensive, or cheap and barely good enough. A real comparison plots quality against cost. The models worth considering are on the Pareto frontier: no other candidate beats them on both dimensions at once. A model is dominated when some other candidate is at least as good and cheaper, or at least as cheap and better.
The legacy fine-tune is the interesting result. It is not the worst model on the list, but open-source-70b beats it on both quality and cost together, which is exactly what "dominated" means. A model does not need to be bad in isolation to be the wrong choice. It only needs a strictly better alternative to exist.
Online Evaluation: After It Ships
Offline evaluation answers "which candidate looked best on our test set?". Online evaluation answers the question that matters: how is the system doing on real production traffic right now? That means continuously sampling live requests and responses, scoring a percentage of them with the same LLM-as-judge approach, and tracking the score over time. It is the regression-detection pattern from the agent evaluation chapter, applied to the whole LLM system. The demo shows why it matters. A provider changes something at week 7, and the weekly quality score slowly slides.
Interview Signal is part of Pro
See a real weak answer next to a real strong one for this exact topic.
Quiz is part of Pro
Test what you just read with a short quiz, and bank the XP.
Build the Fine-Tuner and Evaluator levels in the GenAI Lab.
Discussion0
Join the Discussion
Sign in to leave comments, reply to others, or like insights.
No comments yet. Be the first to start the thread!