Database Sharding & Partitioning
You'll learn to
- -Compare hash-based, range-based, and directory-based sharding
Consistent hashing is the general technique; sharding is what you call it when you apply that idea specifically to splitting a database across multiple independent instances, each owning a slice of the total data.
Sharding is usually the last resort, not the first move: it's a one-way door that adds real complexity (cross-shard queries, rebalancing, no more simple JOINs). Vertical scaling and read replicas solve a large fraction of real-world scale problems without it.
A shard router directs each request to the database instance that owns that piece of data.
Split your database into multiple independent chunks (shards), each owning a subset of the data, deployed on separate hardware.
A library with 10 million books splits into 26 branches - one per letter of the alphabet. Books A-D go to Branch 1, E-H to Branch 2, and so on. Any librarian can immediately tell you which branch has your book. Each branch can be upgraded independently.
When a single database server cannot handle your data volume or write throughput, you partition (shard) the data across multiple database instances. Each shard is an independent database that owns a non-overlapping slice of the full dataset.
Key-based (Hash) Sharding: `shard = hash(user_id) % num_shards`. Simple and even distribution. Problem: resharding is painful - adding a new shard requires remapping and moving a huge portion of data. Consistent hashing solves this but adds complexity.
Range-based Sharding: Shard by value ranges (user_ids 0-1M on Shard 1, 1M-2M on Shard 2). Simple to understand and enables range queries within a shard. Problem: hotspots - if all new users are being created right now, they all land on the same shard.
Directory-based Sharding: A lookup table maps each key to a specific shard. Maximum flexibility (any key can be moved to any shard). Problem: the lookup table becomes a bottleneck and single point of failure.
The hardest problems with sharding: cross-shard joins (expensive, often impossible), distributed transactions (require complex 2-phase commit), and rebalancing (moving data when shard sizes skew). Many teams avoid sharding and instead choose Vitess, CockroachDB, or Spanner which handle sharding transparently.
- -Sharding enables horizontal scaling of writes (not just reads).
- -Shard key choice is critical and nearly irreversible after launch.
- -Cross-shard joins and transactions are extremely complex - avoid designing for them.
- -Consider Vitess (YouTube's MySQL sharding layer), CockroachDB, or Aurora for managed sharding.
- -MongoDB, Cassandra, and DynamoDB shard automatically and transparently.
You shard users by hash(user_id). A product manager now wants "show me all users in the EU" for a compliance report. What broke?
"Nothing: you just query every shard and combine the results."
"That works, but notice what changed: a query that used to be one lookup is now a scatter-gather across every shard, which is slower and more complex than it looks (partial failures, merging results, pagination across shards). Hash-based sharding optimizes for even distribution and point lookups by key; it actively works against range or attribute-based queries. If that access pattern is common, it's a sign this data might be a better fit for range-based or directory-based sharding instead."