Database Sharding (Partitioning)
DatabasesSplit your database into multiple independent chunks (shards), each owning a subset of the data, deployed on separate hardware.
A library with 10 million books splits into 26 branches - one per letter of the alphabet. Books A-D go to Branch 1, E-H to Branch 2, and so on. Any librarian can immediately tell you which branch has your book. Each branch can be upgraded independently.
When a single database server cannot handle your data volume or write throughput, you partition (shard) the data across multiple database instances. Each shard is an independent database that owns a non-overlapping slice of the full dataset.
Key-based (Hash) Sharding: `shard = hash(user_id) % num_shards`. Simple and even distribution. Problem: resharding is painful - adding a new shard requires remapping and moving a huge portion of data. Consistent hashing solves this but adds complexity.
Range-based Sharding: Shard by value ranges (user_ids 0-1M on Shard 1, 1M-2M on Shard 2). Simple to understand and enables range queries within a shard. Problem: hotspots - if all new users are being created right now, they all land on the same shard.
Directory-based Sharding: A lookup table maps each key to a specific shard. Maximum flexibility (any key can be moved to any shard). Problem: the lookup table becomes a bottleneck and single point of failure.
The hardest problems with sharding: cross-shard joins (expensive, often impossible), distributed transactions (require complex 2-phase commit), and rebalancing (moving data when shard sizes skew). Many teams avoid sharding and instead choose Vitess, CockroachDB, or Spanner which handle sharding transparently.
- Sharding enables horizontal scaling of writes (not just reads).
- Shard key choice is critical and nearly irreversible after launch.
- Cross-shard joins and transactions are extremely complex - avoid designing for them.
- Consider Vitess (YouTube's MySQL sharding layer), CockroachDB, or Aurora for managed sharding.
- MongoDB, Cassandra, and DynamoDB shard automatically and transparently.
Curated Curation & Deep Insights
ScaleDojo CertifiedOur system architects are vetting high-quality, authorized video guides for Database Sharding (Partitioning) with zero third-party platform links.
We are preparing premium, zero-competitor deep-dives for Database Sharding (Partitioning). Authorized reading references will appear automatically.