GenAI Learn/Serving, Evaluation & Monitoring

Browsing as a guest. Sign in to save your progress and earn XP as you complete chapters.

Offline Evaluation: Metrics That Fit the Task

5 min read

You'll learn to

-Choose an offline metric that matches what the system is used for
-Compute precision at k and NDCG for a ranker, and explain why NDCG rewards order
-Explain calibration, and show why a good AUC does not guarantee it
-List reasons offline metrics and online results disagree

A model's offline score is the number you have before you have any users. It decides which model gets built, which change ships, and which idea is dropped. If the metric does not fit the task, those decisions are wrong, however carefully they are made.

An Analogy: Choosing the Right Instrument

A bathroom scale measures weight, not height. A tape measure does the opposite. Neither is wrong, but using the wrong one gives you a precise number that answers a question you did not ask. Metrics are instruments. The task decides which to use.

Match the Metric to the Task

-Classification (spam or not, fraud or not): precision, recall, and the precision-recall curve. Use them with a chosen threshold. The classification metrics chapter earlier in this course covers these in depth.
-Ranking and recommendation: precision at k, recall at k, mean reciprocal rank, mean average precision, and NDCG. These care about the order of results, because a user sees the top few.
-Regression (ETA, price, demand): mean absolute error, root mean squared error, and quantile loss when over and under predictions cost different amounts.
-Probability estimates (click rate, conversion): log loss and calibration, because the number itself is used, for example to price an ad.

Ranking Metrics

Precision at k asks: of the top k results, how many are relevant? It ignores the order inside the top k. NDCG, normalized discounted cumulative gain, fixes that. It gives each result a gain based on its relevance grade, discounts the gain by how far down the list it sits, and divides by the best possible score so that 1.0 is a perfect ordering.

Discounted cumulative gain at position k

DCG = Σ (2^relevance − 1) ÷ log₂(position + 1) and NDCG = DCG ÷ DCG of the best possible order

A relevant result at position 1 counts fully, at position 3 about half, and a perfect result buried at position 10 barely counts.

Calibration

A calibrated model's numbers mean what they say: of all the cases it scores at 10%, about 10% really happen. This matters whenever the score is used as a probability. An ad auction multiplies the predicted click chance by the bid, and a model that is twice as confident as it should be makes every ad look twice as valuable.

AUC measures ranking: does the model put positives above negatives? It does not care about the scale of the scores. Double every score and the ranking, and the AUC, stay exactly the same, but the probabilities are now wrong. The code below has two parts. The first scores two rankers on three queries with graded relevance. The second builds two models that rank customers identically, where model B simply doubles model A's scores. Press Run and compare the AUC and the average predicted rate in the lower table.

Press Run: NDCG sees the better ordering, and AUC cannot see the doubled scores

import math
import random

def dcg(grades):
    return sum((2 ** g - 1) / math.log2(i + 2) for i, g in enumerate(grades))

def ndcg(grades):
    ideal = dcg(sorted(grades, reverse=True))
    return dcg(grades) / ideal if ideal else 0.0

def precision_at(k, grades):
    return sum(g > 0 for g in grades[:k]) / k

# Relevance grades (0 = bad ... 3 = perfect) of the top 5 results each ranker returned, per query.
ranker_a = [[3, 2, 0, 1, 0], [0, 3, 1, 0, 2], [2, 0, 0, 3, 1]]
ranker_b = [[2, 3, 1, 0, 0], [3, 1, 2, 0, 0], [3, 2, 1, 0, 0]]
for name, results in (("ranker A", ranker_a), ("ranker B", ranker_b)):
    p3 = sum(precision_at(3, g) for g in results) / len(results)
    n5 = sum(ndcg(g) for g in results) / len(results)
    print(f"{name}: precision@3 {p3:.2f}   NDCG@5 {n5:.3f}")

# Calibration: two models rank customers identically, but model B is overconfident.
random.seed(6)
N = 5000
p_true = [random.betavariate(1, 12) for _ in range(N)]                 # true chance of the event
outcome = [random.random() < p for p in p_true]
model_a = p_true[:]                                                    # well calibrated
model_b = [min(0.99, 2 * p) for p in p_true]                           # same order, doubled scores

def auc(scores):
    order = sorted(range(N), key=lambda i: scores[i])
    rank_sum = sum(r + 1 for r, i in enumerate(order) if outcome[i])
    n_pos = sum(outcome)
    return (rank_sum - n_pos * (n_pos + 1) / 2) / (n_pos * (N - n_pos))

def log_loss(scores):
    return -sum(math.log(s if o else 1 - s) for s, o in zip(scores, outcome)) / N

observed = sum(outcome) / N
print()
print(f"{'':<10}{'AUC':>8}{'log loss':>10}{'avg predicted':>15}{'actual rate':>13}")
for name, scores in (("model A", model_a), ("model B", model_b)):
    avg = sum(scores) / N
    print(f"{name:<10}{auc(scores):>8.3f}{log_loss(scores):>10.4f}{avg:>15.3%}{observed:>13.3%}")

import math
import random

def dcg(grades):
    return sum((2 ** g - 1) / math.log2(i + 2) for i, g in enumerate(grades))

def ndcg(grades):
    ideal = dcg(sorted(grades, reverse=True))
    return dcg(grades) / ideal if ideal else 0.0

def precision_at(k, grades):
    return sum(g > 0 for g in grades[:k]) / k

# Relevance grades (0 = bad ... 3 = perfect) of the top 5 results each ranker returned, per query.
ranker_a = [[3, 2, 0, 1, 0], [0, 3, 1, 0, 2], [2, 0, 0, 3, 1]]
ranker_b = [[2, 3, 1, 0, 0], [3, 1, 2, 0, 0], [3, 2, 1, 0, 0]]
for name, results in (("ranker A", ranker_a), ("ranker B", ranker_b)):
    p3 = sum(precision_at(3, g) for g in results) / len(results)
    n5 = sum(ndcg(g) for g in results) / len(results)
    print(f"{name}: precision@3 {p3:.2f}   NDCG@5 {n5:.3f}")

# Calibration: two models rank customers identically, but model B is overconfident.
random.seed(6)
N = 5000
p_true = [random.betavariate(1, 12) for _ in range(N)]                 # true chance of the event
outcome = [random.random() < p for p in p_true]
model_a = p_true[:]                                                    # well calibrated
model_b = [min(0.99, 2 * p) for p in p_true]                           # same order, doubled scores

def auc(scores):
    order = sorted(range(N), key=lambda i: scores[i])
    rank_sum = sum(r + 1 for r, i in enumerate(order) if outcome[i])
    n_pos = sum(outcome)
    return (rank_sum - n_pos * (n_pos + 1) / 2) / (n_pos * (N - n_pos))

def log_loss(scores):
    return -sum(math.log(s if o else 1 - s) for s, o in zip(scores, outcome)) / N

observed = sum(outcome) / N
print()
print(f"{'':<10}{'AUC':>8}{'log loss':>10}{'avg predicted':>15}{'actual rate':>13}")
for name, scores in (("model A", model_a), ("model B", model_b)):
    avg = sum(scores) / N
    print(f"{name:<10}{auc(scores):>8.3f}{log_loss(scores):>10.4f}{avg:>15.3%}{observed:>13.3%}")

Ranker B beats ranker A on both ranking metrics: precision at 3 is 1.00 against 0.56, and NDCG at 5 is 0.938 against 0.774. Ranker A puts a zero-relevance result in the top three on every query, which is what drags it down. In the lower table, models A and B have the same AUC, 0.781, since B only rescales A. But the average predicted rate for B is 15.5% while the true rate is 7.6%, and its log loss is worse, 0.2659 against 0.2325. Log loss and calibration see what AUC cannot.

More Ways Offline Metrics Mislead

-Averages hide slices. A model can be good overall and bad for new users, one country, or one device. Report the metric by segment.
-Random splits leak time. When data has a time order, train on the past and test on the future.
-Feedback loops and position bias. Logged clicks reflect what the old system chose to show, so a better new ranker can look worse on old logs.
-Metric mismatch. The offline metric is a proxy. A better NDCG does not guarantee more engagement, which is why launches need an online experiment.

Choose your offline metrics before you see results, write down the minimum improvement that would justify a launch, and keep a metric on each segment you care about. Then treat the offline number as a filter for what to test online, not as the verdict.

Interview Signal is part of Pro

See a real weak answer next to a real strong one for this exact topic.

Quiz is part of Pro

Test what you just read with a short quiz, and bank the XP.

Model Serving Basics: Latency & Rollouts Monitoring, Drift & Retraining

Discussion0

Join the Discussion

No comments yet. Be the first to start the thread!