Design Search Ranking

Search is the oldest ML design problem and still one of the most asked. A user types a few words, and within about 200 milliseconds you must return the ten best results out of millions. Every stage has a standard solution. The interesting part is the training data, because the clicks you log are not an honest measure of what is relevant.

A good reference librarian first works out what you actually mean ("jaguar", the car or the cat?), then pulls a stack of candidate sources, then orders them by how well they answer you. A search system does the same in three steps: understand the query, retrieve candidates, rank them.

• -Query understanding: spelling correction, tokenization, recognizing intent (a product, a place, a question), and expanding with synonyms.
• -Retrieval: lexical matching (an inverted index with a scoring function like BM25) finds documents with the right words. Embedding search finds documents with the right meaning. Many systems run both and merge, as in the hybrid search chapter.
• -First-pass ranking: a cheap model scores a few thousand candidates.
• -Learning to rank: a stronger model scores the top few hundred with rich features, such as text match quality, freshness, popularity, and the user's context. Gradient boosted trees are common, and a neural model that reads the query and document together may rescore the final handful.
• -Final rules: filters, safety, and diversity.

• -Human relevance judgments: trained raters grade query and result pairs, for example from 0 (irrelevant) to 4 (perfect), following written guidelines. Accurate and costly, so they form the evaluation set and part of the training data.
• -Click logs: abundant and free, but they measure what users clicked in the positions you showed, not what is relevant.
• -Other behavior: dwell time, "long clicks", whether the user reformulated the query or came back to the results.

People look at the first result far more than the fifth. So a mediocre result at position 1 gets many clicks and a great result at position 5 gets few. A model trained on raw clicks learns to rank whatever was already on top, and confirms the old ranker's mistakes. Joachims, Swaminathan, and Schnabel studied this in a 2017 paper on unbiased learning to rank. The fix is to weight each click by one divided by the chance that position was examined, called a propensity weight.

The simulation below shows five documents, always shown in the same order by an old ranker. People look at the positions with chances of 100%, 70%, 50%, 35%, and 25%, and click a document they have looked at with that document's true relevance. Press Run, and compare the raw click rate, the corrected one, and the true relevance.

Document D has the highest true relevance, 0.80, but it sat at position 5, so its raw click rate is only 0.202, and ranking by raw clicks puts it fourth. Document B, with relevance 0.45 at position 2, wins the raw ranking. After weighting each click by the inverse of the chance it was seen, the corrected values come out close to the truth (0.807 for D, 0.599 for C), and the corrected ranking matches the true one exactly. Real systems have to estimate the look chances, for example by randomly shuffling a small share of results, and the weights can be noisy, but the principle holds.

• -Offline: NDCG at 10 on graded human labels, by query type, including head queries (popular) and tail queries (rare).
• -Online: success rate (did the user find something), click-through on top results, reformulation rate, and abandonment. Interleaving compares two rankers quickly.
• -Latency: about 200 ms end to end, so retrieval and each ranking stage have budgets of tens of milliseconds, and heavier models run only on a short list.

• -Tail queries have little data, so rely on text and embedding signals.
• -Spam and manipulation: people tune pages to win. Add quality and trust signals.
• -Ambiguous queries: diversify results across plausible meanings.