GenAI Learn/Classic ML Design Questions

Browsing as a guest. Sign in to save your progress and earn XP as you complete chapters.

Design Ad Click Prediction

5 min read

You'll learn to

-Explain why calibration matters more for ad prediction than for most models
-Describe how negative downsampling inflates predictions and how to correct it
-Use normalized entropy as a metric and read it
-List the features, data challenges, and serving constraints of a CTR model

Online ads run on a prediction: how likely is this person to click this ad right now? Every ad impression involves a model call, and the answer feeds an auction worth billions of dollars. Click prediction is also an unusually good teacher, because it forces you to deal with extreme imbalance, delayed feedback, and probabilities that must be right and not merely well ordered.

An Analogy: Pricing by Odds

An auction for an ad slot works a little like betting. The platform multiplies what an advertiser will pay per click by the chance of a click, and the ad with the highest result wins the slot. If your click estimate is twice what it should be, every ad looks twice as valuable, the wrong ads win, and the prices are wrong. That is why the number itself must be right.

The Task

For every (user, ad, context) combination, predict the probability of a click, and sometimes of a later conversion. Typical click rates are one or two percent or less, so the data is extremely imbalanced. The model is scored by an auction on hundreds of candidate ads per request, in a few milliseconds each.

Data and Features

-User features: demographics, interests, history of clicks and purchases.
-Ad features: the advertiser, creative, category, text and image embeddings, past performance.
-Context: device, time, page, position of the slot.
-Cross features: the combination of a user's interests with the ad category. These are often where the signal is.
-High-cardinality IDs: millions of user, ad, and advertiser IDs. Use hashing or learned embeddings.

A 2014 paper from Facebook's ads team, "Practical Lessons from Predicting Clicks on Ads at Facebook", is a classic. It combines gradient boosted trees, which build useful feature combinations, with logistic regression, and reports that the combination beats either alone by more than 3%. It also stresses that the most important thing is having the right features, in particular historical information about the user and the ad, ahead of fine-tuning the model. The paper evaluates with normalized entropy and calibration, two ideas this chapter covers.

Negative Downsampling

Training on billions of impressions where 98% are non-clicks is wasteful. A common trick is to keep all clicks and only a fraction of non-clicks, say 10%. Training gets cheaper, but the model now sees clicks as ten times more common than they are, so its probabilities come out too high. The correction is a short formula: with keep rate w, the true probability is the model's probability p divided by p + (1 − p) ÷ w. The code applies it, and then computes normalized entropy. Press Run, and compare the three average click chances in the middle.

Press Run: downsampling inflates the average click chance from 0.031 to 0.214 until the correction fixes it

import math

KEEP = 0.1                      # we keep only 10% of the non-clicks in the training data

def undo_downsampling(p_trained, keep=KEEP):
    """Turn a score from a model trained on downsampled data back into a real click chance."""
    return p_trained / (p_trained + (1 - p_trained) / keep)

print("score from the downsampled model -> corrected click chance")
for score in (0.05, 0.10, 0.20, 0.50):
    print(f"  {score:.2f} -> {undo_downsampling(score):.4f}")

# Five ads, each with a true click chance. A model trained on downsampled data inflates it.
true_p = [0.005, 0.01, 0.02, 0.04, 0.08]
inflated = [p / (p + (1 - p) * KEEP) for p in true_p]       # what the downsampled model learns
fixed = [undo_downsampling(p) for p in inflated]

print()
print(f"{'true':>8}{'model says':>12}{'after correction':>18}")
for t, s, f in zip(true_p, inflated, fixed):
    print(f"{t:>8.3f}{s:>12.3f}{f:>18.3f}")
avg = (sum(true_p) / 5, sum(inflated) / 5, sum(fixed) / 5)
print(f"average click chance: true {avg[0]:.3f}, uncorrected {avg[1]:.3f}, corrected {avg[2]:.3f}")

# Normalized entropy: log loss divided by the log loss of always predicting the average click rate.
def log_loss(preds, outcomes):
    total = sum(math.log(p) if y else math.log(1 - p) for p, y in zip(preds, outcomes))
    return -total / len(outcomes)

background = 0.02
outcomes = [True] * 2 + [False] * 98                       # 100 impressions at a 2% click rate
model_preds = [0.30, 0.20] + [0.008] * 98
baseline_loss = log_loss([background] * 100, outcomes)
model_loss = log_loss(model_preds, outcomes)
print()
ne = model_loss / baseline_loss
print(f"normalized entropy of the model: {ne:.2f}  (below 1.0 beats always guessing 2%)")

import math

KEEP = 0.1                      # we keep only 10% of the non-clicks in the training data

def undo_downsampling(p_trained, keep=KEEP):
    """Turn a score from a model trained on downsampled data back into a real click chance."""
    return p_trained / (p_trained + (1 - p_trained) / keep)

print("score from the downsampled model -> corrected click chance")
for score in (0.05, 0.10, 0.20, 0.50):
    print(f"  {score:.2f} -> {undo_downsampling(score):.4f}")

# Five ads, each with a true click chance. A model trained on downsampled data inflates it.
true_p = [0.005, 0.01, 0.02, 0.04, 0.08]
inflated = [p / (p + (1 - p) * KEEP) for p in true_p]       # what the downsampled model learns
fixed = [undo_downsampling(p) for p in inflated]

print()
print(f"{'true':>8}{'model says':>12}{'after correction':>18}")
for t, s, f in zip(true_p, inflated, fixed):
    print(f"{t:>8.3f}{s:>12.3f}{f:>18.3f}")
avg = (sum(true_p) / 5, sum(inflated) / 5, sum(fixed) / 5)
print(f"average click chance: true {avg[0]:.3f}, uncorrected {avg[1]:.3f}, corrected {avg[2]:.3f}")

# Normalized entropy: log loss divided by the log loss of always predicting the average click rate.
def log_loss(preds, outcomes):
    total = sum(math.log(p) if y else math.log(1 - p) for p, y in zip(preds, outcomes))
    return -total / len(outcomes)

background = 0.02
outcomes = [True] * 2 + [False] * 98                       # 100 impressions at a 2% click rate
model_preds = [0.30, 0.20] + [0.008] * 98
baseline_loss = log_loss([background] * 100, outcomes)
model_loss = log_loss(model_preds, outcomes)
print()
ne = model_loss / baseline_loss
print(f"normalized entropy of the model: {ne:.2f}  (below 1.0 beats always guessing 2%)")

The model trained on downsampled data says an ad with a true click chance of 0.005 will be clicked with probability 0.048, nearly ten times too high. Averaged over the five ads, the true click chance is 0.031, the uncorrected model says 0.214, and after the correction it is back to 0.031. Without the correction, auction prices would be off by a large factor, even though the ranking of ads would be unchanged. Normalized entropy divides the model's log loss by the log loss of always predicting the average rate, so below 1.0 means the model beats that baseline. Here it is 0.37.

Use the demo below to see the inflation and the fix. Choose how much of the non-clicks you keep and an ad's true click chance, and compare what the model learns with the corrected score.

Downsampling Non-Clicks Inflates the Click Chance

Keep only a share of the non-clicks when training and the model learns that clicks are far more common than they are. Choose how much you keep and an ad's true click chance, then see what the model learns and what the correction restores.

Share of non-clicks kept for training10%

True click chance of this ad2.0%

True click chance

2.0%

What the model learns

16.9%

8.5x too high

After the correction

2.0%

matches the truth

Learned score16.9%

Corrected score2.0%

Keeping 10% of the non-clicks makes a 2.0% ad look like 16.9%, 8.5 times too high. An auction that multiplies bid by this score would overpay by that factor, which is why the score must be corrected before it is used.

Other Challenges

-Delayed feedback: a click arrives in seconds, but a purchase can take days. Use windows, wait for labels to mature, or model the delay.
-Freshness: new ads and trends appear constantly, so models retrain often, sometimes hourly or online.
-Position bias: top slots get more clicks. Train with position as a feature, then set it to a fixed value when scoring.
-Exploration: new ads have no history. Reserve some impressions so they can be learned.
-Latency: hundreds of candidate ads, a few milliseconds each. Keep features precomputed and the model compact.

Evaluation

-Offline: log loss or normalized entropy, calibration overall and by segment, and AUC for ranking.
-Online: A/B test on revenue and advertiser value, with guardrails on user experience, such as hides and complaints.

Interview Signal is part of Pro

See a real weak answer next to a real strong one for this exact topic.

Quiz is part of Pro

Test what you just read with a short quiz, and bank the XP.

Design Search Ranking Design Fraud Detection

Discussion0

Join the Discussion

No comments yet. Be the first to start the thread!