CDNs and Edge Caching

Your servers can be fast and your code perfect, and a user on the other side of the world will still wait. The reason is not computing; it is distance. Information cannot travel faster than light, and in practice, through undersea fiber and a chain of routers, it travels a good deal slower. A Content Delivery Network, or CDN, attacks that problem directly: instead of making every user travel to your one server, it puts copies of your content near the users.

Take a server in Virginia, on the US east coast, and a user in Mumbai. The cable route between them is on the order of 17,000 kilometers. Light in fiber covers roughly 200,000 kilometers per second, so even in a perfect world one trip takes about 85 ms, and a round trip about 170 ms. Real round trips on that route are typically 200 to 250 ms once routers and detours are counted.

That would be tolerable if a page load needed one trip, but it needs many. Remember the last two chapters: a first visit to an HTTPS site costs a round trip for TCP, one for TLS and one for the request, so roughly three round trips before the first byte arrives. At 250 ms each that is 750 ms of pure waiting before your server has done anything, and then a real page fetches dozens of images, scripts and stylesheets. No amount of server tuning can remove a delay that the speed of light imposes.

Suppose the only copy of every book is in one national library in Virginia. Everyone, everywhere, has to travel there to borrow anything. That is a single origin server. Now the library opens a branch in every city. Each branch keeps copies of the books people in that city actually ask for. When you ask for a popular book, the branch hands it over in minutes. When you ask for a rare one it does not have, the branch orders it from the national library, gives it to you, and keeps a copy on its shelf for the next reader.

• -The national library is the origin: your real server, the one source of truth.
• -A branch is an edge server, in a location called a point of presence (PoP). Large CDNs run hundreds of them around the world.
• -A book on the branch shelf is a cache hit: served from nearby, fast, and the origin never hears about it.
• -A book the branch has to order is a cache miss: the edge fetches it from the origin, serves it, and keeps a copy, so only the first reader in each city pays the long trip.
• -Sending you to the nearest branch is done with DNS or routing tricks: the CDN's answer depends on where you are, which is the geo-routing idea from the DNS chapter.

The two pictures below show the same user, without and then with a branch library nearby.

Two numbers describe a CDN's value. The first is latency. If the edge is in Mumbai, the round trip drops from about 250 ms to about 20 ms, so those three round trips cost 60 ms instead of 750 ms. The second is the hit ratio, the share of requests served from the edge without bothering the origin. At a 95% hit ratio, an origin that would have faced 2,000 requests per second now sees only 100. The CDN has become a shield: your servers are protected from the traffic, and a spike such as a big launch or a viral post is mostly absorbed by the edge.

A CDN can only safely cache something that is the same for everyone who asks. That makes the split fairly clear:

• -Safe: images, videos, fonts, stylesheets, JavaScript bundles, downloads, and public pages or API responses that are identical for every visitor.
• -Unsafe: anything personal or private, such as a logged-in user's account page, their basket, or any response that contains their data. If the edge cached it, the next visitor could be served someone else's page.

Your server tells the CDN what it may do through a Cache-Control header on each response. In plain words: `max-age=3600` means anyone may reuse this for an hour; `public` means a shared cache such as a CDN may store it; and `no-store` means never keep a copy at all. The header is the contract between your origin and every cache between you and the user.

Back to the library. You publish a corrected edition of a book, but every branch still has the old edition on its shelf and will keep lending it out until told otherwise. This is cache invalidation, famously one of the hard problems in computing. For a CDN it means that a file you updated can keep being served in its old version, from hundreds of locations, until the cached copy expires. There are two standard fixes, and they work in opposite ways:

• -Purge: tell the CDN to throw the old copy away. Most providers offer an API call for this, and it is the right tool for something that must change right now, like a pricing page. The cost is that it is an operation you have to remember to run, and it takes from a moment to about a minute to reach every location.
• -Version the filename: never change a file, create a new one. A script is deployed as app.3f9c1a.js, and the next release becomes app.b72e40.js, where the part in the middle is a fingerprint of the contents. The page now points at a name no cache has ever seen, so nothing stale can be served, and the old file can safely be cached for a year. This is how most large sites ship their scripts and images.

A rule of thumb: version anything that changes rarely and is cached for a long time, like bundles and images; purge, or use a short max-age, for things that change in place, like a homepage or a price.