Why Caching Works: Locality & Hit Rates
You'll learn to
- -Explain the locality principle behind caching
- -Define cache hit ratio
The data tier was flagged back in Module 2 as the hardest tier to scale, and caching is the single highest-leverage fix available, because it lets you serve most requests without touching the database at all.
Locality: Some Data Is Hotter Than Other Data
Real traffic is almost never evenly distributed. A handful of trending products, viral posts, or celebrity profiles account for a disproportionate share of all reads; the same 80/20 pattern shows up everywhere. That means you don't need to cache everything to get most of the benefit; you need to cache the hot fraction.
Cache Hit Ratio
Hit ratio = hits / (hits + misses). If 1,000 requests hit the cache and 100 miss through to the database, that's a 90.9% hit ratio, meaning the database only ever sees roughly 1 in 11 requests. This is why a relatively small, cheap cache in front of an expensive database can absorb a huge share of total traffic: it only needs to hold the hot fraction of your data, not all of it.
Most reads are absorbed by the cache; only a cache miss ever reaches the database.
Know the order-of-magnitude cost of every operation - from L1 cache to cross-continental packets.
If L1 cache access = 1 second (reading a note on your desk), then RAM access = 2 minutes (walking to a filing cabinet), SSD access = 3 days, spinning disk = 2 months, and a round-trip packet from New York to London = 2 days of flights but you're also flying at jet speed. This relative scale governs every performance decision.
These numbers are approximations (2020s hardware). The exact numbers matter less than knowing the RATIOS between them. The fact that disk is 1000x slower than RAM is architecturally meaningful.
L1 cache reference: ~0.5 ns. Nearly instantaneous.
Branch misprediction: ~5 ns.
L2 cache reference: ~7 ns. 14x slower than L1.
Mutex lock/unlock: ~25 ns.
- -RAM is 1,000x faster than disk. Put hot data in memory (Redis).
- -SSD is 10-100x faster than spinning disk. Always use NVMe SSDs in production.
- -Cross-datacenter round trips are ~1-5ms. Design your system to minimize them.
- -Network within the same AZ is ~0.5ms. Cross-region is ~30-150ms.
- -Memory bandwidth (RAM reads) is extremely fast - sequential reads are your friend.
Your cache has a 99% hit ratio. Is that automatically "good enough" and safe to stop optimizing?
"Yes, 99% is a great hit ratio."
"Only relative to the actual request volume. At 1,000 requests/sec, a 1% miss rate is 10 requests/sec hitting the database (trivial). At 1,000,000 requests/sec, that same 1% is 10,000 requests/sec, potentially still enough to overwhelm the database. A hit ratio is only meaningful next to the traffic number it's a percentage of."