Full-Text Search with Elasticsearch: From Queries to Millions of Results in Milliseconds

How Wikipedia Searches 60 Million Articles in 50ms

Wikipedia receives 1.3 billion unique device visits per month, and its search handles tens of thousands of queries per second across 60 million articles in 300+ languages. If you search for 'distributed systems,' Wikipedia does not scan 60 million articles looking for your words. Instead, it uses Elasticsearch (and its predecessor Lucene) with an inverted index - a data structure that maps every word to the list of documents containing it. The lookup takes microseconds regardless of how many documents exist. Google, Amazon, Airbnb, Uber, GitHub, and Slack all use Elasticsearch for their search infrastructure. Understanding how inverted indexes work is fundamental to designing any system with search.

The Inverted Index

Forward Index (what your database has):

  Doc 1: 'system design is important for interviews'
  Doc 2: 'distributed systems handle scale'
  Doc 3: 'design patterns for system architecture'

  To find docs containing 'system':
  Scan ALL documents -> check each one -> O(N) per query
  At 60M documents, this takes seconds.

Inverted Index (what Elasticsearch builds):

  Term           -> Document IDs
  --------       ---------------
  'system'       -> [1, 3]
  'design'       -> [1, 3]
  'important'    -> [1]
  'interview'    -> [1]
  'distributed'  -> [2]
  'scale'        -> [2]
  'pattern'      -> [3]
  'architecture' -> [3]

  To find docs containing 'system':
  Lookup 'system' in hash map -> [1, 3] -> O(1) per query!

  To find docs containing 'system' AND 'design':
  Lookup 'system' -> [1, 3]
  Lookup 'design' -> [1, 3]
  Intersect -> [1, 3]    (done in microseconds)

Text Analysis Pipeline

Before Indexing, Text is Transformed:

  Raw: 'The System Design Lab runs specialized interviews'

  Step 1 - Tokenize:   ['The', 'System', 'Design', 'Lab', 'runs', 'specialized', 'interviews']
  Step 2 - Lowercase:  ['the', 'system', 'design', 'lab', 'runs', 'specialized', 'interviews']
  Step 3 - Stop words: ['system', 'design', 'lab', 'runs', 'specialized', 'interviews']
                        (removed 'the' - too common to be useful)
  Step 4 - Stemming:   ['system', 'design', 'lab', 'run', 'special', 'interview']
                        ('runs' -> 'run', 'interviews' -> 'interview')
  Step 5 - Synonyms:   ['system', 'design', 'lab', 'run', 'special', 'interview']
                        (could map: 'system' -> 'system, platform')

  Result: searching 'running' matches 'runs' (both stem to 'run').
  Result: searching 'interviews' matches 'interview'.

  Analyzer Types:
  Analyzer        Use Case           Example
  ----------      ---------------    ------------------
  standard        General English    Default, works well
  english         English-specific   Better stemming
  icu_analyzer    Multi-language     Unicode support
  custom          Domain-specific    Medical, legal terms
  keyword         Exact match        IDs, codes, emails

BM25 Relevance Scoring

How Elasticsearch Ranks Results:

  BM25 Score = TF * IDF * field_length_norm

  1. Term Frequency (TF):
     How often does the term appear in this document?
     Doc A: mentions 'system' 5 times -> higher score
     Doc B: mentions 'system' 1 time  -> lower score
     (Saturates: 10 mentions is not 10x better than 1)

  2. Inverse Document Frequency (IDF):
     How rare is this term across ALL documents?
     'the' appears in 95% of docs -> low IDF (worthless)
     'kubernetes' appears in 0.1% of docs -> high IDF (discriminating)

  3. Field Length Normalization:
     'system design' in a 5-word title  -> very relevant
     'system design' in a 10,000-word essay -> less relevant

  Practical Example:
  Query: 'distributed system design'

  Doc A: Title: 'System Design Guide'  Body: 'Learn distributed systems...'
  Doc B: Title: 'Introduction to CS'   Body: '...also covers system design briefly...'

  Doc A wins because:
  - Title match (short field, high relevance)
  - Term frequency higher
  - 'distributed' has high IDF (rare across all docs)

Elasticsearch Cluster Architecture

Scaling Elasticsearch:

  [Index: products]  100M documents
      |
  Split into 5 primary shards + 1 replica each:

  Node 1          Node 2          Node 3
  +----------+    +----------+    +----------+
  |Shard 0(P)|    |Shard 1(P)|    |Shard 2(P)|
  |Shard 3(R)|    |Shard 4(R)|    |Shard 0(R)|
  +----------+    +----------+    +----------+
                  Node 4          Node 5
                  +----------+    +----------+
                  |Shard 3(P)|    |Shard 4(P)|
                  |Shard 1(R)|    |Shard 2(R)|
                  +----------+    +----------+

  P = primary shard (writes + reads)
  R = replica shard (reads + failover)

  Query execution (scatter-gather):
  1. Query hits coordinating node
  2. Coordinator sends query to one copy of each shard (5 shards)
  3. Each shard searches its local inverted index (parallel)
  4. Results merged, sorted by BM25 score, top N returned

  Shard sizing rule of thumb:
  - Target 10-50 GB per shard
  - 100M docs x 1KB avg = 100 GB -> 5 shards of 20 GB

Interview Tip

When designing search in system design interviews, say: 'I would use Elasticsearch backed by an inverted index. Documents are analyzed through a text pipeline (tokenize, lowercase, stem) and indexed so any term lookup returns matching document IDs in O(1). BM25 scoring ranks results by term frequency, document rarity, and field length. For scale, I would shard the index across nodes with replicas for availability and use the scatter-gather pattern for distributed queries. The key design decisions are the analyzer choice (affects what matches), shard count (affects parallelism), and the mapping (define field types upfront before indexing). I would sync data from the primary database to Elasticsearch asynchronously via CDC or a queue.'

<

/blockquote>

Key Takeaway

Elasticsearch uses inverted indexes to convert O(N) document scans into O(1) term lookups. Text analysis pipelines normalize text so searches for 'running' match 'runs.' BM25 scoring produces relevance rankings using term frequency, inverse document frequency, and field length normalization. At scale, shard horizontally with replicas, and always define your mapping before indexing production data.