LSM-Trees and Elasticsearch: The Storage Engines Behind Write-Heavy Systems
When Write Performance Matters More Than Read Performance
Traditional database indexes commonly use B-trees: sorted tree structures designed for fast point lookups and range queries. To really understand why LSM-trees and Elasticsearch exist as alternatives, it helps to first look at where B-trees start to struggle.
B-trees are excellent for read-heavy workloads, but writes can become expensive at high throughput. An insert or update may require modifying existing index pages, causing random page writes, page splits, and additional write amplification. Databases mitigate this with memory caching and write-ahead logging, but random I/O can still be expensive, especially on spinning disks.
In 1996, Patrick O'Neil and colleagues introduced the Log-Structured Merge-Tree (LSM-tree), a design that takes a different approach.
Instead of repeatedly modifying existing disk structures, an LSM-based storage engine buffers writes in memory and flushes them to disk as large, sorted, immutable files. Those files are then merged in the background through a process called compaction.
The core trade-off is simple:
B-trees favor efficient reads and in-place updates. LSM-trees favor high sequential write throughput by deferring and batching disk work.
This trade-off becomes especially valuable for workloads such as:
Log ingestion
IoT sensor data
Metrics collection
Event tracking
Time-series data
High-volume key-value workloads
How LSM-Trees Work
A simplified LSM write path looks like this:
Client
↓
Write-Ahead Log (WAL)
↓
MemTable
↓
Immutable MemTable
↓
Flush
↓
SSTable
↓
Background Compaction
↓
Larger SSTables
Want to practice scalable system architecture hands-on? Explore our HLD Architecture Lab.
1. Write-Ahead Log
Before the write is considered durable, many LSM-based databases first append the operation to a Write-Ahead Log (WAL).
The WAL protects against data loss if the machine crashes before the in-memory data has been flushed to disk.
Write
↓
WAL → Durable
↓
MemTable → Fast
The WAL and MemTable serve different purposes:
WAL: durability
MemTable: fast in-memory organization
2. MemTable
The actual data is then written to an in-memory structure called a MemTable.
The MemTable maintains data in sorted-key order, making it efficient to later flush the data as a sorted file.
Because the write happens primarily in memory and the WAL is append-oriented, the storage engine can handle a large number of writes without immediately performing random disk updates.
3. SSTables
When the MemTable reaches a configured threshold, it becomes immutable and is flushed to disk as an SSTable (Sorted String Table).
An SSTable is immutable: once written, it isn't modified in place.
A simplified SSTable might look like:
apple → 42
banana → 17
car → 91
dog → 23
Because the data is sorted, the storage engine can efficiently locate relevant blocks using indexes and other metadata.
4. Compaction
Over time, the database accumulates many SSTables.
Instead of modifying them individually, the storage engine periodically merges them together.
SSTable A ─┐
SSTable B ─┼──→ Compaction ──→ Larger SSTable
SSTable C ─┘
During compaction, the engine can:
Merge sorted data
Remove obsolete versions
Remove deleted records
Reduce the number of files that reads must examine
Compaction is one of the most important trade-offs in an LSM-based database.
Writes become efficient, but the database must perform additional background work later.
How Reads Work
The downside of immutable SSTables is that the latest version of a key might not exist in only one place.
A simplified read path is:
Read(key)
↓
MemTable
↓
Bloom Filter
↓
Block Cache
↓
SSTable
The storage engine generally checks the newest data first and then older structures as necessary.
Bloom Filters
A Bloom filter is a probabilistic data structure that quickly determines whether a key might exist in an SSTable.
Its most important property:
A Bloom filter can have false positives, but not false negatives.
In other words:
Bloom Filter
"NO" → The key definitely isn't here.
"YES" → The key might be here.
This prevents many unnecessary disk reads.
Block Cache
Frequently accessed portions of SSTables can also be kept in memory through a block cache.
This creates several layers of read optimization:
MemTable — newest data is already in memory
Bloom filter — eliminates SSTables that definitely don't contain the key
Block cache — keeps frequently accessed SSTable blocks in memory
Disk — accessed only when necessary
This is how LSM-based databases reduce the read amplification created by their write-optimized design.
The Real Trade-Off: Amplification
LSM-trees don't eliminate work. They move the work.
Write Amplification
Write amplification measures how much data the storage engine ultimately writes compared with the amount of data written by the application.
For example:
Application writes: 100 GB
WAL + flushing + compaction
↓
Physical disk writes: 300 GB
Write amplification = 3×
Higher write amplification means more disk I/O, more SSD wear, and potentially lower throughput.
Read Amplification
Read amplification refers to how many storage structures or disk reads may need to be examined to satisfy one logical read.
Compaction, Bloom filters, indexes, caching, and leveled structures help control this cost.
Space Amplification
Space amplification occurs when multiple versions of data temporarily coexist.
During compaction, old SSTables cannot always be deleted until their replacements have been safely created.
So an LSM design must balance:
Write throughput
↕
Read performance
↕
Disk space
↕
Compaction cost
LSM Compaction Strategies: Size-Tiered vs Leveled
Compaction strategy has a major impact on the performance characteristics of an LSM-based database.
Size-Tiered Compaction
Size-tiered compaction groups similarly sized SSTables together and merges them when enough files accumulate.
SSTable SSTable SSTable SSTable
\ | | /
Compaction
↓
Larger SSTable
It generally favors write throughput because compactions can be performed less frequently and in larger batches.
The trade-off is potentially higher read and space amplification.
Characteristics:
High write throughput
Larger compaction operations
Higher read amplification
Higher temporary space usage
Cassandra supports size-tiered compaction and other strategies depending on workload and configuration.
Leveled Compaction
Leveled compaction organizes SSTables into levels.
A simplified model looks like:
L0
↓
L1
↓
L2
↓
L3
Each level is progressively larger, and compaction moves data between levels.
The goal is to keep the amount of overlapping data relatively small, making reads more predictable.
The trade-off is increased write amplification because data can be rewritten multiple times as it moves through levels.
Characteristics:
Lower read amplification
More predictable reads
Higher write amplification
More continuous background compaction
RocksDB and LevelDB support leveled compaction, along with other compaction approaches.
Time-Window Compaction
For time-series workloads, Time-Window Compaction Strategy (TWCS) groups SSTables into time windows and primarily compacts data within those windows.
This can be particularly effective for append-heavy workloads where old data becomes immutable and eventually expires.
10:00–11:00 → SSTables
11:00–12:00 → SSTables
12:00–13:00 → SSTables
↓
Compaction within time windows
The important lesson is that there is no universally best compaction strategy.
Choose the strategy based on the workload.
Where LSM-Trees Are Used
LSM-based storage engines are widely used in systems where high write throughput is important.
Examples include:
Cassandra
RocksDB
LevelDB
HBase
CockroachDB
InfluxDB
This is why LSM-trees frequently appear in system-design interviews.
If you're designing a system that continuously ingests millions of events, metrics, logs, or sensor readings, the storage engine becomes an important architectural decision.
Elasticsearch and the Inverted Index
LSM-trees solve one problem:
How can we efficiently store large volumes of writes?
But what if the problem is:
How can we search billions of documents by their content?
That's where Elasticsearch comes in.
Elasticsearch is a distributed search and analytics engine built on Apache Lucene.
Its core strength comes from Lucene's inverted index.
Shay Banon originally developed Elasticsearch after working on Compass, a search library built on Lucene. Elasticsearch exposed Lucene's search capabilities through a distributed system and a developer-friendly REST API.
How an Inverted Index Works
A normal document-oriented view looks like:
Document 1 → "distributed systems"
Document 2 → "distributed databases"
Document 3 → "database indexing"
An inverted index reverses this relationship:
distributed → [Document 1, Document 2]
systems → [Document 1]
databases → [Document 2]
database → [Document 3]
indexing → [Document 3]
Now consider a search for:
"distributed database"
Instead of scanning every document, the search engine can retrieve the document lists associated with the relevant terms and combine them.
The inverted index can also store information such as term positions, enabling phrase queries and other text-search features.
This is why full-text search can be dramatically faster than scanning every document.
BM25 and Relevance
Finding matching documents isn't enough.
A search engine also needs to determine:
Which matching documents should appear first?
Elasticsearch uses Lucene's relevance-scoring mechanisms, including BM25, to rank matching documents.
Factors such as term frequency and how common a term is across the collection influence relevance.
So a search engine effectively performs two jobs:
1. Find matching documents
2. Rank the matches
Elasticsearch's Key Characteristics
Inverted Index
Maps terms to documents and associated information such as positions.
Near-Real-Time Search
Documents are not necessarily searchable immediately after an indexing request. Lucene segments become visible to search after a refresh.
With the commonly used default refresh behavior, newly indexed documents are often searchable within roughly a second, although this can be configured.
Sharding and Replication
Elasticsearch distributes indexes across shards and can replicate those shards across nodes.
This allows the system to scale storage and search capacity horizontally.
Aggregations
Elasticsearch isn't limited to keyword search.
It can also perform analytics such as:
Grouping
Histograms
Top-N queries
Percentiles
Statistical aggregations
This combination made Elasticsearch popular for log analytics and observability workloads.
Dynamic Mapping
Elasticsearch supports dynamic mapping, meaning it can automatically infer field types.
However, that does not mean Elasticsearch has no schema.
For production systems, explicit mappings are often preferable when you need predictable field types and protection against uncontrolled field creation.
Elasticsearch and Lucene: A Similar Storage Idea
There is an important connection between LSM-trees and Elasticsearch that system-design candidates should understand.
Elasticsearch is not simply an LSM-tree.
However, Lucene uses a conceptually similar strategy:
LSM-tree:
MemTable
↓
SSTable
↓
Compaction
↓
Merged SSTables
Lucene:
Documents
↓
Immutable Segments
↓
Segment Merges
↓
Larger Segments
Both systems rely heavily on:
Immutable structures
Sequential/background work
Periodic merging
Avoiding constant in-place modification
The implementation details are different, but the underlying engineering principle is similar:
Write new immutable structures and merge them in the background instead of constantly modifying large structures in place.
This is one of the most useful connections between storage engines and search systems.
When to Use Elasticsearch vs a Database
Elasticsearch and a relational database solve different problems.
Use a Database When You Need
Transactions
Strong consistency
Relational queries
Constraints
Structured data
Source-of-truth storage
General-purpose CRUD operations
Examples:
Orders
Payments
Inventory
User accounts
Financial records
Use Elasticsearch When You Need
Full-text search
Fuzzy search
Autocomplete
Faceted search
Log search
Large-scale search analytics
Relevance ranking
Examples:
Product search
Document search
Log analytics
Search suggestions
Observability platforms
A Common Production Architecture
For many applications, Elasticsearch should be treated as a search index rather than the system of record.
A common architecture is:
┌───────────────┐
│ Application │
└───────┬───────┘
│
┌─────────┴─────────┐
↓ ↓
PostgreSQL Search Pipeline
Source of Truth ↓
Elasticsearch
The search index can be populated using mechanisms such as:
Change Data Capture (CDC)
Event streams
Application-level indexing
If Elasticsearch becomes unavailable, the primary database can still contain the authoritative business data.
This separation is especially valuable when transactional correctness matters.
Elasticsearch does support document-level atomicity and concurrency controls, but it does not provide general-purpose multi-document ACID transactions like a relational database.
Elasticsearch Production Pitfalls
Elasticsearch is easy to start with and significantly harder to operate well at scale.
Several failure patterns appear repeatedly.
1. Deep Pagination
A request such as:
from = 100000
size = 20
can become expensive because Elasticsearch may need to collect and sort many candidate documents across shards before returning the requested page.
For large result sets, use search_after with an appropriate deterministic sort and tiebreaker instead of relying on deep from/size pagination.
2. Mapping Explosion
Dynamic mapping can become dangerous when applications accept arbitrary user-defined fields.
For example:
{
"custom_field_1": "...",
"custom_field_2": "...",
"custom_field_3": "..."
}
If untrusted input continually introduces new field names, the mapping can grow dramatically and put pressure on cluster resources.
For controlled schemas, consider explicit mappings and use dynamic: strict or dynamic: false where appropriate.
3. Hot Shards
Time-series applications frequently write most new data into the current time-based index.
This can create uneven workload distribution if the active index or its shards receive substantially more traffic than older indices.
Common techniques include:
Data streams
Index rollover
Appropriate shard sizing
Time-based indices
Careful routing design
The goal is to prevent a small subset of shards from becoming bottlenecks.
4. Heap Pressure
Elasticsearch depends heavily on both JVM memory and the operating system's filesystem cache.
A common operational guideline is to allocate roughly half of system memory to the JVM heap while leaving substantial memory available for the OS page cache, subject to Elasticsearch's current recommendations and workload.
Simply increasing heap indefinitely isn't a solution.
Large heaps can increase garbage-collection pressure and reduce memory available for filesystem caching.
5. Refresh Interval
A refresh makes recently indexed documents visible to search.
A short refresh interval provides lower search latency for newly indexed documents but increases background work.
During heavy bulk ingestion, increasing the refresh interval—or temporarily controlling refresh behavior—can improve indexing throughput.
The right value depends on how quickly the application actually needs newly indexed data to become searchable.
What Interviewers Test About Storage Engines and Search
Storage-engine questions usually appear as part of a larger system-design problem.
The interviewer isn't simply testing whether you can define an LSM-tree.
They want to know whether you can connect workload → storage engine → performance characteristics → architecture.
Want to practice database and storage design hands-on? Explore our Database Lab.
Question: You have a write-heavy workload. Why consider an LSM-based database?
A strong answer:
An LSM-based engine can buffer writes in memory and flush them as sequential immutable files, avoiding many random in-place updates. The trade-off is compaction, write amplification, and potentially higher read amplification.
Question: How do LSM databases reduce read amplification?
Think in layers:
MemTable
↓
Bloom Filter
↓
Block Cache
↓
SSTable
Each layer tries to prevent unnecessary disk work.
Question: Why isn't Cassandra ideal for arbitrary read queries?
Cassandra is designed around known access patterns and partition-key-based queries.
It is not a general-purpose relational database where you can freely introduce arbitrary query patterns and expect efficient execution.
Data modeling must be designed around the queries the application needs.
Question: When would you choose Elasticsearch?
If the system needs:
Full-text search
Relevance ranking
Fuzzy matching
Autocomplete
Faceted search
Search-oriented analytics
Elasticsearch becomes a strong candidate.
If the system needs:
Transactions
Strong relational guarantees
Constraints
General-purpose CRUD
A relational database is generally a better fit.
Question: Is Elasticsearch immediately consistent?
Not necessarily.
Elasticsearch is near-real-time for search visibility. A successfully indexed document may not be visible to searches until a refresh occurs.
That means applications must account for a small delay between indexing and search visibility.
The Bigger System-Design Lesson
The most important lesson isn't simply:
"Use LSM-trees for writes."
It's this:
The storage engine is an architectural decision.
Consider a high-volume event platform:
Events
↓
Load Balancer
↓
Kafka
↓
Storage Workers
↓
LSM-based Database
↓
Compaction
The LSM-based storage engine helps absorb a large write workload.
But users may also need to search those events:
Events
↓
Kafka / CDC
↓
Elasticsearch
↓
Search & Analytics
Now the architecture has two different storage technologies because the system has two different access patterns:
LSM-based database
↓
High-throughput writes / durable data
Elasticsearch
↓
Full-text search / analytics
This is the key architectural idea:
Don't choose a database based only on what data you have. Choose the storage technology based on how the system needs to read and write that data.
Final Takeaway
B-trees and LSM-trees represent two different approaches to the same fundamental problem: how to organize data efficiently on storage.
B-trees:
Fast reads
Excellent range queries
In-place page updates
Potential random I/O
LSM-trees:
High write throughput
Sequential writes
Immutable SSTables
Background compaction
Potential read/write amplification
And Elasticsearch takes a different approach to a different problem:
Documents
↓
Inverted Index
↓
Fast Full-Text Search
↓
Relevance Ranking
↓
Distributed Search & Analytics
The deeper lesson for system design is that storage engines shape the architecture above them.
If writes dominate, you may favor an LSM-based storage engine.
If full-text search dominates, you may introduce Elasticsearch.
If transactional correctness dominates, a relational database may remain the source of truth.
The best architecture doesn't ask:
"Which database is the best?"
It asks:
"What access pattern am I optimizing for, and what storage engine is designed for it?"
LSM Compaction Strategies: Size-Tiered vs Leveled
As SSTables accumulate, reads must check more and more files for a given key. Compaction merges SSTables to reduce this overhead and reclaim space from deleted keys. The compaction strategy fundamentally shapes the performance profile of the database. Size-tiered compaction (Cassandra default) groups similarly-sized SSTables together and merges them when a size threshold is reached. It is write-optimized: few compaction operations, high write throughput. The trade-off: reads become slower over time as tiers accumulate, and space amplification is high (the new merged file coexists with the old files until the merge completes, temporarily doubling disk usage). Leveled compaction (RocksDB/LevelDB default, optional in Cassandra) maintains sorted levels where each level is 10x larger than the previous. New data enters L0, then gets compacted into L1, then L2 progressively. Reads are much faster (each level is a sorted run with minimal overlap), but write amplification is higher.
Size-tiered: fewer, larger compaction operations. High write throughput. High read amplification and space amplification.
Leveled: continuous smaller compactions into sorted levels. Low read amplification. Higher write amplification.
TWCS (Time-Window Compaction Strategy): Cassandra strategy for time-series. Groups SSTables by time window, compacts within windows. Low write and read amplification for append-heavy workloads.
Write amplification: the ratio of data written to disk vs data written by the application. Critical for SSD wear and throughput.
Read amplification: number of disk reads per logical read. Bloom filters and level structure minimize this.
Elasticsearch Production Pitfalls
Elasticsearch is easy to get started with and surprisingly difficult to operate at scale. The most common production failures follow predictable patterns. Deep pagination (requesting page 10,000 of search results) triggers a distributed merge sort across all shards and can OOM your cluster. Use search-after pagination with a sort key instead. Mapping explosion happens when dynamic mapping encounters fields with high cardinality (like user-submitted JSON keys): Elasticsearch creates a mapping entry for every unique field name, potentially creating millions of mappings that exhaust heap. Use explicit mappings and disable dynamic mapping for untrusted data.
Deep pagination: use search_after with a tiebreaker sort field instead of from/size. Avoids the distributed merge-sort penalty.
Mapping explosion: set dynamic: 'strict' or dynamic: false in your index mapping for fields derived from user input.
Hot shard problem: if your shard key is a timestamp and you are writing time-series data, all writes go to the current shard. Use index aliases with time-based rollover.
Heap pressure: Elasticsearch uses half of available memory for the JVM heap. The other half is for OS file cache. Exceeding 32GB heap disables compressed object pointers and dramatically increases GC pressure.
Refresh interval: default 1-second refresh makes new documents searchable quickly but is expensive under heavy write load. Increase to 30 seconds or more during bulk indexing.
What Interviewers Test About Storage Engines and Search
Storage engine questions usually come up in the context of database selection or system design for high-throughput write scenarios. The interviewer wants to see that you can trace system behavior back to storage layer decisions.
Know: why Cassandra is not a good fit for read-heavy workloads with arbitrary query patterns
Know: the three levels of read amplification mitigation in LSM: MemTable (free), bloom filters (probabilistic), block cache (hot SSTable blocks in memory)
Know: when to use Elasticsearch vs a database for search (Elasticsearch for full-text and analytics, database for transactional and structured queries)
Know: Elasticsearch's eventual consistency model (near-real-time indexing, 1-second default refresh) and its implications
Know: why Elasticsearch is not a primary datastore (no transactions, possible data loss on poor configuration, operational complexity)
Related reading
LSM Compaction Strategies: Size-Tiered vs Leveled
As SSTables accumulate, reads must check more and more files for a given key. Compaction merges SSTables to reduce this overhead and reclaim space from deleted keys. The compaction strategy fundamentally shapes the performance profile of the database. Size-tiered compaction (Cassandra default) groups similarly-sized SSTables together and merges them when a size threshold is reached. It is write-optimized: few compaction operations, high write throughput. The trade-off: reads become slower over time as tiers accumulate, and space amplification is high (the new merged file coexists with the old files until the merge completes, temporarily doubling disk usage). Leveled compaction (RocksDB/LevelDB default, optional in Cassandra) maintains sorted levels where each level is 10x larger than the previous. New data enters L0, then gets compacted into L1, then L2 progressively. Reads are much faster (each level is a sorted run with minimal overlap), but write amplification is higher.
Size-tiered: fewer, larger compaction operations. High write throughput. High read amplification and space amplification.
Leveled: continuous smaller compactions into sorted levels. Low read amplification. Higher write amplification.
TWCS (Time-Window Compaction Strategy): Cassandra strategy for time-series. Groups SSTables by time window, compacts within windows. Low write and read amplification for append-heavy workloads.
Write amplification: the ratio of data written to disk vs data written by the application. Critical for SSD wear and throughput.
Read amplification: number of disk reads per logical read. Bloom filters and level structure minimize this.
Elasticsearch Production Pitfalls
Elasticsearch is easy to get started with and surprisingly difficult to operate at scale. The most common production failures follow predictable patterns. Deep pagination (requesting page 10,000 of search results) triggers a distributed merge sort across all shards and can OOM your cluster. Use search-after pagination with a sort key instead. Mapping explosion happens when dynamic mapping encounters fields with high cardinality (like user-submitted JSON keys): Elasticsearch creates a mapping entry for every unique field name, potentially creating millions of mappings that exhaust heap. Use explicit mappings and disable dynamic mapping for untrusted data.
Deep pagination: use search_after with a tiebreaker sort field instead of from/size. Avoids the distributed merge-sort penalty.
Mapping explosion: set dynamic: 'strict' or dynamic: false in your index mapping for fields derived from user input.
Hot shard problem: if your shard key is a timestamp and you are writing time-series data, all writes go to the current shard. Use index aliases with time-based rollover.
Heap pressure: Elasticsearch uses half of available memory for the JVM heap. The other half is for OS file cache. Exceeding 32GB heap disables compressed object pointers and dramatically increases GC pressure.
Refresh interval: default 1-second refresh makes new documents searchable quickly but is expensive under heavy write load. Increase to 30 seconds or more during bulk indexing.
What Interviewers Test About Storage Engines and Search
Storage engine questions usually come up in the context of database selection or system design for high-throughput write scenarios. The interviewer wants to see that you can trace system behavior back to storage layer decisions.
Know: why Cassandra is not a good fit for read-heavy workloads with arbitrary query patterns
Know: the three levels of read amplification mitigation in LSM: MemTable (free), bloom filters (probabilistic), block cache (hot SSTable blocks in memory)
Know: when to use Elasticsearch vs a database for search (Elasticsearch for full-text and analytics, database for transactional and structured queries)
Know: Elasticsearch's eventual consistency model (near-real-time indexing, 1-second default refresh) and its implications
Know: why Elasticsearch is not a primary datastore (no transactions, possible data loss on poor configuration, operational complexity)
LSM Compaction Strategies: Size-Tiered vs Leveled
As SSTables accumulate, reads must check more and more files for a given key. Compaction merges SSTables to reduce this overhead and reclaim space from deleted keys. The compaction strategy fundamentally shapes the performance profile of the database. Size-tiered compaction (Cassandra default) groups similarly-sized SSTables together and merges them when a size threshold is reached. It is write-optimized: few compaction operations, high write throughput. The trade-off: reads become slower over time as tiers accumulate, and space amplification is high (the new merged file coexists with the old files until the merge completes, temporarily doubling disk usage). Leveled compaction (RocksDB/LevelDB default, optional in Cassandra) maintains sorted levels where each level is 10x larger than the previous. New data enters L0, then gets compacted into L1, then L2 progressively. Reads are much faster (each level is a sorted run with minimal overlap), but write amplification is higher.
- Size-tiered: fewer, larger compaction operations. High write throughput. High read amplification and space amplification.
- Leveled: continuous smaller compactions into sorted levels. Low read amplification. Higher write amplification.
- TWCS (Time-Window Compaction Strategy): Cassandra strategy for time-series. Groups SSTables by time window, compacts within windows. Low write and read amplification for append-heavy workloads.
- Write amplification: the ratio of data written to disk vs data written by the application. Critical for SSD wear and throughput.
- Read amplification: number of disk reads per logical read. Bloom filters and level structure minimize this.
Elasticsearch Production Pitfalls
Elasticsearch is easy to get started with and surprisingly difficult to operate at scale. The most common production failures follow predictable patterns. Deep pagination (requesting page 10,000 of search results) triggers a distributed merge sort across all shards and can OOM your cluster. Use search-after pagination with a sort key instead. Mapping explosion happens when dynamic mapping encounters fields with high cardinality (like user-submitted JSON keys): Elasticsearch creates a mapping entry for every unique field name, potentially creating millions of mappings that exhaust heap. Use explicit mappings and disable dynamic mapping for untrusted data.
- Deep pagination: use search_after with a tiebreaker sort field instead of from/size. Avoids the distributed merge-sort penalty.
- Mapping explosion: set dynamic: 'strict' or dynamic: false in your index mapping for fields derived from user input.
- Hot shard problem: if your shard key is a timestamp and you are writing time-series data, all writes go to the current shard. Use index aliases with time-based rollover.
- Heap pressure: Elasticsearch uses half of available memory for the JVM heap. The other half is for OS file cache. Exceeding 32GB heap disables compressed object pointers and dramatically increases GC pressure.
- Refresh interval: default 1-second refresh makes new documents searchable quickly but is expensive under heavy write load. Increase to 30 seconds or more during bulk indexing.
What Interviewers Test About Storage Engines and Search
Storage engine questions usually come up in the context of database selection or system design for high-throughput write scenarios. The interviewer wants to see that you can trace system behavior back to storage layer decisions.
- Know: why Cassandra is not a good fit for read-heavy workloads with arbitrary query patterns
- Know: the three levels of read amplification mitigation in LSM: MemTable (free), bloom filters (probabilistic), block cache (hot SSTable blocks in memory)
- Know: when to use Elasticsearch vs a database for search (Elasticsearch for full-text and analytics, database for transactional and structured queries)
- Know: Elasticsearch's eventual consistency model (near-real-time indexing, 1-second default refresh) and its implications
- Know: why Elasticsearch is not a primary datastore (no transactions, possible data loss on poor configuration, operational complexity)