Ranking Algorithms: How Search Results Are Ordered

How Google Improved Search by 0.5% and Made $500 Million

In 2019, Google deployed BERT (a language understanding model) to search ranking. The change improved search quality by just 0.5% as measured by NDCG. That tiny improvement translated to hundreds of millions in revenue because Google processes 8.5 billion searches daily - even microscopic ranking improvements affect billions of user sessions. This is why ranking is the most valuable ML problem in tech. Google, Amazon, Netflix, Spotify, LinkedIn, and TikTok all have dedicated ranking teams, because putting the RIGHT item at position 1 instead of position 5 changes user behavior dramatically. Studies show that 75% of users never scroll past the first page. Position 1 gets 30% of clicks, position 2 gets 15%, position 5 gets 5%. Ranking is not optional - it IS the product experience.

Two-Stage Search Architecture

Search Pipeline:

  User query: 'best laptop under $1000'
       |
  Stage 1: RETRIEVAL (fast, recall-focused)
  +--------------------------------------+
  | Inverted index + BM25                 |
  | Find ALL documents mentioning laptop  |
  | Result: 5,000 candidates in <10ms     |
  |                                       |
  | BM25 knows: term frequency, rarity,   |
  | field length                           |
  | BM25 does NOT know: user intent,       |
  | that '$1000' means price filter,       |
  | that 'best' means high-rated           |
  +--------------------------------------+
       |
  Stage 2: RANKING (slow, precision-focused)
  +--------------------------------------+
  | ML model scores 5,000 candidates      |
  | using 200+ features per document       |
  |                                       |
  | Output: ranked list, top 20 shown      |
  | Latency budget: <100ms for all 5,000   |
  +--------------------------------------+
       |
  Stage 3: POST-PROCESSING
  +--------------------------------------+
  | Remove duplicates and near-duplicates  |
  | Apply business rules (boost promoted)  |
  | Ensure diversity (not all same brand)  |
  | A/B test allocation (model A vs B)     |
  +--------------------------------------+

Ranking Signal Categories

Ranking Features (200+ in production systems):

  Category             Example Signals            Weight
  ---------------      -----------------------    ------
  Query-Document       BM25 score                 High
                       Semantic similarity (BERT)  High
                       Exact phrase match          High
                       Title match vs body match   Medium
                       Query term coverage         Medium

  Document Quality     PageRank / authority score  High
                       Content freshness (days)    Medium
                       Word count / depth          Low
                       Spelling/grammar quality    Low
                       Mobile-friendly             Medium

  User Behavior        Historical CTR for query    High
                       Dwell time (time on page)   High
                       Bounce rate                 Medium
                       Past clicks on this domain  Medium

  Personalization      User location               Medium
                       User search history         Medium
                       User language preference    High
                       Device type                 Low

  Context              Time of day                 Low
                       Session query sequence      Medium
                       Is query navigational?      High
                       Is query transactional?     High

Learning to Rank

Three LTR Approaches:

  1. POINTWISE: Score each document independently
     Input: (query, document, features) -> predicted_relevance: 0.85
     Model: regression or classification
     Pros: simple to implement
     Cons: ignores relative ordering between documents

  2. PAIRWISE: Predict which of two documents is better
     Input: (query, doc_A, doc_B) -> A is better than B? Yes/No
     Model: RankNet, LambdaRank, LambdaMART
     Pros: directly optimizes ordering
     Cons: O(N^2) pairs for N documents

  3. LISTWISE: Optimize entire ranked list at once
     Input: (query, [doc_1, doc_2, ..., doc_N]) -> optimal ordering
     Model: ListNet, optimize NDCG directly
     Pros: most theoretically sound
     Cons: complex to train

  In Practice:
  LambdaMART (gradient-boosted trees) remains the workhorse.
  Google uses neural models (BERT) for semantic understanding.
  Most companies start with LambdaMART, add neural features later.

  Evaluation Metrics:
  Metric  What It Measures             Range
  ------  --------------------------   -----
  NDCG    Quality of ranking order      0-1
  MRR     Position of first relevant    0-1
  MAP     Average precision at cutoffs  0-1
  CTR     % of users who click          0-100%
  Dwell   Time spent after clicking     seconds

Interview Tip

When discussing search ranking, say: 'I would use a two-stage pipeline. Stage 1 is retrieval using BM25 on an inverted index - fast, finds all potential candidates. Stage 2 is ranking using a LambdaMART model (gradient-boosted trees) with features from four categories: query-document relevance (BM25, semantic similarity), document quality (PageRank, freshness), user behavior (historical CTR, dwell time), and personalization (location, device). I would evaluate offline using NDCG on historical data, then run online A/B tests comparing click-through rate and user satisfaction. Key insight: a 0.5% NDCG improvement at Google's scale translates to hundreds of millions in value.'

<

/blockquote>

Key Takeaway

Ranking transforms a bag of retrieved documents into a precisely ordered list using 200+ signals: text relevance, document quality, user behavior, and personalization. Learning to Rank (LambdaMART) trains ML models to optimize ranking metrics like NDCG directly. Evaluate offline with historical click data and online with A/B tests. Even tiny ranking improvements at scale generate enormous business value.