The NoSQL Landscape

A relational database makes strong promises, and those promises have a price: every row in a table must fit one fixed shape, and keeping its guarantees across many machines is hard. As a product grows you may need something the relational model is not best at: data whose shape varies from record to record, an enormous volume of simple reads or writes, or relationships that run many steps deep. NoSQL (often read as not only SQL) is a family of databases that each relax something the relational model insists on, in exchange for doing one kind of job very well. There is no single NoSQL model, so the real skill is knowing the four main shapes and what each is for.

Picture four different places to keep your things. A coat check hands you a numbered ticket: give the ticket, get your coat in an instant, but nobody can ask the attendant for every red coat. A drawer of personal folders holds one folder per person, and each folder contains whatever papers that person happens to have, so no two folders need the same contents. A huge ledger sorted by account number lets every row grow its own extra columns, and the bookkeeper is built to add new lines at tremendous speed. And a corkboard with pins and string shows who knows whom, so finding a friend of a friend means following two strings.

• -The coat check is a key-value store (Redis, DynamoDB): the fastest possible lookup by key, but you can only look up by key, not by what is inside the value.
• -The drawer of folders is a document store (MongoDB, Firestore): one self-describing JSON-like record per entity, so records can differ in shape.
• -The sorted ledger is a wide-column store (Cassandra, HBase): rows found by key, each with its own sparse set of columns, built for a very high rate of writes.
• -The corkboard is a graph database (Neo4j, Neptune): the connections are stored as first-class data, so following them is cheap.

The easiest contrast with SQL is a user profile whose shape varies. In a relational design, addresses, phone numbers and preferences would each be spread across their own tables and joined back together. A document store keeps the whole entity as one record, which can be read or written in a single step:

The trade-off is the mirror image of SQL's: a flexible shape and one-step reads, but limited joins across documents, and keeping several documents consistent with each other is largely your job.

The graph model is the one whose reason to exist can be shown with arithmetic. Suppose each person has 200 friends and you want everyone within three steps. In SQL that means joining the friendship table to itself, and each extra step multiplies the work: 200 friends, then about 200 times 200 = 40,000 friends of friends, then about 200 times 200 times 200 = 8,000,000 candidates at three steps (before removing duplicates). Both approaches have to visit the people in range. The graph database's advantage is that each hop follows a stored link instead of searching a table of millions of rows, so its cost depends on the size of the neighborhood rather than on the size of the whole data set.

• -You get horizontal scale: data is divided by key and designed to spread across many machines.
• -You get a flexible schema: records can change shape without migrating a whole table.
• -You get speed on the one access pattern the model was built for.
• -You give up ad hoc queries: you design for your access patterns up front, and asking a new kind of question can mean restructuring the data.
• -You usually give up joins, and often transactions that span several records (this varies by product).

Many NoSQL products have narrowed this gap over the years, so check what a specific product guarantees rather than assuming. The honest summary is that NoSQL is not faster than SQL in general: it is faster at one pattern because it gave something up. For a side-by-side comparison with concrete scenarios, the card below sets the two against each other:

Imagine taking notes in a meeting. Jotting each point at the bottom of a scratch pad is instant, whereas tidying every point into its proper place in a bound notebook as it arrives would slow you down. So you scribble fast, and when the pad is full you copy it out in one sweep, neatly sorted, as a new sealed page. That is an LSM-tree: the scratch pad is the memtable, and each sealed, sorted page is an SSTable.

The B-tree index from the last chapter is read-optimized: it keeps data sorted on disk at all times, so every write has to find the right spot and update it in place, which means random disk I/O. Wide-column stores like Cassandra and HBase, along with embedded engines like RocksDB and LevelDB, use a different structure entirely to get their write throughput: the Log-Structured Merge-Tree, or LSM-tree.

An LSM-tree never updates data in place. Every write is first appended to an in-memory, sorted structure called a memtable, an operation that costs no disk I/O at all. Once the memtable fills up, it is flushed to disk as a new, immutable, sorted file called an SSTable (Sorted String Table), and a fresh, empty memtable takes over for new writes. Because a flush writes one large sequential block instead of many scattered updates, it is dramatically cheaper than the random writes a B-tree needs to keep itself sorted in place.

Run the simulation below and watch writes pile up in the memtable, then flush to disk as one sorted file, without anything ever being updated in place.

The cost shows up on reads instead. A single key might exist in the memtable, or in any of several SSTables on disk (whichever one was current when it was last written), so a lookup may have to check multiple places, newest first, until it finds the key. A background compaction process periodically merges older SSTables together, both to reclaim space taken by overwritten or deleted keys and to bound how many separate files a read ever has to check.