Skip to content
GenAI Learn/Fine-Tuning & Evaluation at Scale
Browsing as a guest. Sign in to save your progress and earn XP as you complete chapters.

Offline & Online Evaluation for LLM Systems

4 min read

You'll learn to

  • -Separate offline evaluation (a fixed test set before launch) from online evaluation (live traffic after launch)
  • -Compare candidate models on both quality and cost with a Pareto frontier
  • -Detect a quality regression after launch with a simple moving average

A team picks an LLM by running ten queries and eyeballing the results, then makes a decision clients will live with for months. That is not an evaluation framework, it is a guess with extra steps. The earlier tier covered evaluating agent trajectories, and the Evals modules just before this tier go deep on building a test set and trusting a judge. This chapter takes the production view of the broader discipline: comparing candidates, and watching a system both before and after it reaches production.

An Analogy: The Kitchen Taste Test and the Customer Reviews

A restaurant tests a new dish in the kitchen before putting it on the menu: that is offline evaluation, with a fixed set of testers and criteria. After it opens, the restaurant also reads real customers' reviews and watches what comes back uneaten: that is online evaluation. A dish can pass the kitchen test and still disappoint real diners, and a supplier can change the recipe quietly later.

Offline Evaluation: Before Anything Ships

Offline evaluation runs a fixed, curated test set through every candidate, model or prompt, before any of them reaches a real user. This is where LLM-as-judge, introduced for RAG in an earlier tier, generalizes. A strong model scores each candidate's outputs against the same criteria, at a scale no team of human reviewers could keep up across thousands of test cases.

No Single Metric Tells the Whole Story

Comparing models on accuracy alone misses that a model can be highly accurate and prohibitively expensive, or cheap and barely good enough. A real comparison plots quality against cost. The models worth considering are on the Pareto frontier: no other candidate beats them on both dimensions at once. A model is dominated when some other candidate is at least as good and cheaper, or at least as cheap and better.

Computing a real quality-cost Pareto frontier across model candidates

The legacy fine-tune is the interesting result. It is not the worst model on the list, but open-source-70b beats it on both quality and cost together, which is exactly what "dominated" means. A model does not need to be bad in isolation to be the wrong choice. It only needs a strictly better alternative to exist.

Online Evaluation: After It Ships

Offline evaluation answers "which candidate looked best on our test set?". Online evaluation answers the question that matters: how is the system doing on real production traffic right now? That means continuously sampling live requests and responses, scoring a percentage of them with the same LLM-as-judge approach, and tracking the score over time. It is the regression-detection pattern from the agent evaluation chapter, applied to the whole LLM system. The demo shows why it matters. A provider changes something at week 7, and the weekly quality score slowly slides.

Press Run: the moving average raises an alert at week 8, long before anyone would notice by eye

Interview Signal is part of Pro

See a real weak answer next to a real strong one for this exact topic.

Quiz is part of Pro

Test what you just read with a short quiz, and bank the XP.

Ready to Build This?

Build the Fine-Tuner and Evaluator levels in the GenAI Lab.

Discussion0

Join the Discussion

Sign in to leave comments, reply to others, or like insights.

Sign In to ScaleDojo

No comments yet. Be the first to start the thread!