Database Sharding (Partitioning)

Databases
07 / 21

Split your database into multiple independent chunks (shards), each owning a subset of the data, deployed on separate hardware.

Hash Routerhash(key) % 3SHARD 1IDs A - HSHARD 2IDs I - PSHARD 3IDs Q - Z
SD blueprint: Database Sharded Cluster
The Analogy (Read This First)

A library with 10 million books splits into 26 branches - one per letter of the alphabet. Books A-D go to Branch 1, E-H to Branch 2, and so on. Any librarian can immediately tell you which branch has your book. Each branch can be upgraded independently.

Deep Dive Analysis

When a single database server cannot handle your data volume or write throughput, you partition (shard) the data across multiple database instances. Each shard is an independent database that owns a non-overlapping slice of the full dataset.

Key-based (Hash) Sharding: `shard = hash(user_id) % num_shards`. Simple and even distribution. Problem: resharding is painful - adding a new shard requires remapping and moving a huge portion of data. Consistent hashing solves this but adds complexity.

Range-based Sharding: Shard by value ranges (user_ids 0-1M on Shard 1, 1M-2M on Shard 2). Simple to understand and enables range queries within a shard. Problem: hotspots - if all new users are being created right now, they all land on the same shard.

Directory-based Sharding: A lookup table maps each key to a specific shard. Maximum flexibility (any key can be moved to any shard). Problem: the lookup table becomes a bottleneck and single point of failure.

The hardest problems with sharding: cross-shard joins (expensive, often impossible), distributed transactions (require complex 2-phase commit), and rebalancing (moving data when shard sizes skew). Many teams avoid sharding and instead choose Vitess, CockroachDB, or Spanner which handle sharding transparently.

Key Bullets
  • Sharding enables horizontal scaling of writes (not just reads).
  • Shard key choice is critical and nearly irreversible after launch.
  • Cross-shard joins and transactions are extremely complex - avoid designing for them.
  • Consider Vitess (YouTube's MySQL sharding layer), CockroachDB, or Aurora for managed sharding.
  • MongoDB, Cassandra, and DynamoDB shard automatically and transparently.
Trade-offs
✅ Unlimited write throughput✅ Data partitioned for regulatory compliance (user data by region)❌ Cross-shard queries are slow/impossible❌ Schema changes must be applied to all shards❌ Rebalancing is complex and risky
Real-World Examples
Instagram: sharded Postgres using user_id for shard keyDiscord: migrated to ScyllaDB (automatic sharding)Shopify: Vitess-based MySQL sharding by shop_id

Curated Curation & Deep Insights

ScaleDojo Certified
Video Tutorial Pending

Our system architects are vetting high-quality, authorized video guides for Database Sharding (Partitioning) with zero third-party platform links.

Verification in Progress
Reading Material Pending

We are preparing premium, zero-competitor deep-dives for Database Sharding (Partitioning). Authorized reading references will appear automatically.

Verification in Progress
ScaleDojo Logo
Initializing ScaleDojo