Availability Patterns: Failover & Redundancy

Everything fails eventually: disks, servers, racks, whole data centers. Availability is the discipline of making sure that a failure somewhere does not become an outage for the user. It starts with a way to measure it, then rests on one idea, redundancy, and on a choice between two ways of arranging the spare capacity. This chapter covers all three.

A car carries one spare tire in the boot. If a tire bursts you stop, swap it, and carry on: a short outage, and the spare sat unused until then. A twin-engine plane flies on both engines at once. If one fails, the other keeps it in the air with no pause at all, although it must be strong enough to carry the whole plane alone.

• -The spare tire is active-passive: a standby waits idle and takes over after a failure, with a short gap while it does.
• -The twin engines are active-active: both are working all the time, so losing one costs capacity, not continuity.
• -The plane having to fly on one engine is the capacity trap: each remaining part must be able to carry the whole load.

Availability is the share of time a service works, usually quoted as a number of nines. It looks like a tiny difference between 99% and 99.99%, but in downtime it is enormous:

Each extra nine cuts the allowed downtime by ten, and costs much more than the one before it, so the right target depends on what an outage costs the business.

When components are chained, so that a request needs every one of them, their availabilities multiply, and the result is worse than any single part. A web tier that is 99.9% available in front of a database that is 99.9% available gives 0.999 x 0.999 = 99.8% for the pair. When components are redundant, so that the service works if either one does, the failures have to coincide to cause an outage. Two independent servers that are each 99% available are both down at once only 0.01 x 0.01 = 0.0001 of the time, which is 99.99% available. Redundancy turns two modest parts into one very reliable service.

The word independent is doing a lot of work. Two servers in the same rack, on the same power feed, running the same buggy release, will fail together, and the nice arithmetic no longer applies. Real redundancy means spreading across racks, zones and regions, and avoiding shared dependencies.

One node or region serves all traffic while a standby waits, ready to take over if the primary fails. It is simple to reason about, but the standby's capacity sits idle most of the time, and the switch takes a while: the failure has to be detected, the standby promoted, and traffic redirected, often tens of seconds to a few minutes. If the data is copied asynchronously, the last few writes can also be lost in the switch.

Several nodes or regions serve production traffic at the same time, so no capacity is idle, and losing one just means the others absorb its share. The costs are real. If two regions can both accept writes to the same data, something has to resolve conflicts when they disagree. And there is the capacity trap: if each of two regions runs at 70% of its capacity, then when one fails the survivor must handle 140% and is overwhelmed. Active-active needs each part sized to carry the whole load, or kept below 50% busy.

The diagram shows the shape: both regions serve live traffic and replicate to each other.

• -Health checks: something must notice quickly and reliably that a node has stopped working.
• -Automatic switching: waiting for a person to wake up and act adds minutes of downtime.
• -Replicated data: a standby is useless if it has no recent copy of the data.
• -Regular rehearsal: a failover path that is never exercised usually does not work when needed, so practice it on purpose, a habit a later module on chaos engineering develops.