How Wikipedia Searches 60 Million Articles in 50ms
Wikipedia receives 1.3 billion unique device visits per month, and its full-text search handles tens of thousands of queries per second across 60 million articles in 300+ languages. If you search for 'distributed systems,' Wikipedia does not scan 60 million articles looking for your words. Instead, it uses Elasticsearch (and its predecessor Lucene) with an inverted index, a data structure that maps every word to the list of documents containing it. The lookup takes microseconds regardless of how many documents exist. Google, Amazon, Airbnb, Uber, GitHub, and Slack all use Elasticsearch for their search infrastructure, and understanding how inverted indexes work is fundamental to designing any system with search.
The Inverted Index
Forward Index (what your database has):
Doc 1: 'system design is important for interviews'
Doc 2: 'distributed systems handle scale'
Doc 3: 'design patterns for system architecture'
To find docs containing 'system':
Scan ALL documents -> check each one -> O(N) per query
At 60M documents, this takes seconds.
Inverted Index (what Elasticsearch builds):
Term -> Document IDs
-------- ---------------
'system' -> [1, 3]
'design' -> [1, 3]
'important' -> [1]
'interview' -> [1]
'distributed' -> [2]
'scale' -> [2]
'pattern' -> [3]
'architecture' -> [3]
To find docs containing 'system':
Lookup 'system' in hash map -> [1, 3] -> O(1) per query!
To find docs containing 'system' AND 'design':
Lookup 'system' -> [1, 3]
Lookup 'design' -> [1, 3]
Intersect -> [1, 3] (done in microseconds)Text Analysis Pipeline
Before any of that lookup magic can happen, Elasticsearch has to break raw text down into a normalized set of terms. This is the analyzer's job, and getting it right (or wrong) determines what your users can and can't find.
Before Indexing, Text is Transformed:
Raw: 'The System Design Lab runs specialized interviews'
Step 1 - Tokenize: ['The', 'System', 'Design', 'Lab', 'runs', 'specialized', 'interviews']
Step 2 - Lowercase: ['the', 'system', 'design', 'lab', 'runs', 'specialized', 'interviews']
Step 3 - Stop words: ['system', 'design', 'lab', 'runs', 'specialized', 'interviews']
(removed 'the' - too common to be useful)
Step 4 - Stemming: ['system', 'design', 'lab', 'run', 'special', 'interview']
('runs' -> 'run', 'interviews' -> 'interview')
Step 5 - Synonyms: ['system', 'design', 'lab', 'run', 'special', 'interview']
(could map: 'system' -> 'system, platform')
Result: searching 'running' matches 'runs'.
Result: searching 'interviews' matches 'interview'.
Analyzer Types:
Analyzer Use Case Example
---------- --------------- ------------------
standard General English Default, works well
english English-specific Better stemming
icu_analyzer Multi-language Unicode support
custom Domain-specific Medical, legal terms
keyword Exact match IDs, codes, emailsBM25 Relevance Scoring
Once the index exists, Elasticsearch still needs to decide which matching documents to show first. That's where BM25 comes in, it's the default ranking algorithm, and it combines three signals into a single relevance score.
How Elasticsearch Ranks Results:
BM25 Score = TF * IDF * field_length_norm
1. Term Frequency (TF):
How often does the term appear in this document?
Doc A: mentions 'system' 5 times -> higher score
Doc B: mentions 'system' 1 time -> lower score
(Saturates: 10 mentions is not 10x better than 1)
2. Inverse Document Frequency (IDF):
How rare is this term across ALL documents?
'the' appears in 95% of docs -> low IDF (worthless)
'kubernetes' appears in 0.1% of docs -> high IDF (discriminating)
3. Field Length Normalization:
'system design' in a 5-word title -> very relevant
'system design' in a 10,000-word essay -> less relevant
Practical Example:
Query: 'distributed system design'
Doc A: Title: 'System Design Guide' Body: 'Learn distributed systems...'
Doc B: Title: 'Introduction to CS' Body: '...also covers system design briefly...'
Doc A wins because:
- Title match (short field, high relevance)
- Term frequency higher
- 'distributed' has high IDF (rare across all docs)Elasticsearch Cluster Architecture
A single node can only hold so much index before lookups start to slow down and hardware runs out of room. Elasticsearch handles this the same way most distributed systems do: split the data into shards, spread the shards across nodes, and replicate for safety. If you want a broader refresher on how these tradeoffs show up across distributed systems in general, the low-level design fundamentals post is a good companion read.
Scaling Elasticsearch:
[Index: products] 100M documents
|
Split into 5 primary shards + 1 replica each:
Node 1 Node 2 Node 3
+----------+ +----------+ +----------+
|Shard 0(P)| |Shard 1(P)| |Shard 2(P)|
|Shard 3(R)| |Shard 4(R)| |Shard 0(R)|
+----------+ +----------+ +----------+
Node 4 Node 5
+----------+ +----------+
|Shard 3(P)| |Shard 4(P)|
|Shard 1(R)| |Shard 2(R)|
+----------+ +----------+
P = primary shard (writes + reads)
R = replica shard (reads + failover)
Query execution (scatter-gather):
1. Query hits coordinating node
2. Coordinator sends query to one copy of each shard (5 shards)
3. Each shard searches its local inverted index (parallel)
4. Results merged, sorted by BM25 score, top N returned
Shard sizing rule of thumb:
- Target 10-50 GB per shard
- 100M docs x 1KB avg = 100 GB -> 5 shards of 20 GBKeeping Elasticsearch in sync with a primary database is its own design problem. Most teams solve it asynchronously, either through change data capture or by pushing updates through a queue, so search never blocks a write path. If you're weighing how that async plumbing fits into the rest of your architecture, it's worth pairing this with the piece on Kafka and event sourcing or the transactional outbox pattern, since both cover keeping downstream systems consistent without coupling them tightly to the source of truth.
Interview Tip
When designing search in system design interviews, say: 'I would use Elasticsearch backed by an inverted index. Documents are analyzed through a text pipeline (tokenize, lowercase, stem) and indexed so any term lookup returns matching document IDs in O(1). BM25 scoring ranks results by term frequency, document rarity, and field length. For scale, I would shard the index across nodes with replicas for availability and use the scatter-gather pattern for distributed queries. The key design decisions are the analyzer choice (affects what matches), shard count (affects parallelism), and the mapping (define field types upfront before indexing). I would sync data from the primary database to Elasticsearch asynchronously via CDC or a queue.'
Architecture overview
Key Takeaway
Elasticsearch uses inverted indexes to convert O(N) document scans into O(1) term lookups. Text analysis pipelines normalize text so searches for 'running' match 'runs.' BM25 scoring produces relevance rankings using term frequency, inverse document frequency, and field length normalization. At scale, shard horizontally with replicas, and always define your mapping before indexing production data.