Circuit Breakers, Retries & Bulkheads

In a system of many services, the most dangerous failure is not a crash. A crashed service is detected quickly and routed around. The dangerous one is a service that is merely slow, because callers keep waiting for it, and the waiting spreads. This chapter explains how a failure spreads, and the three standard defenses: circuit breakers, retries done properly, and bulkheads.

Service A calls Service B, which calls Service C. Say B has 100 threads, normally a call to C takes 50 milliseconds, and requests reach B at 200 per second. Now C becomes slow, and each call hangs for 10 seconds until it times out. Each request now ties up a B thread for 10 seconds, so B would need 200 x 10 = 2,000 threads to keep up and has only 100. At 200 requests a second the 100 threads are all taken within 100 / 200 = half a second. B can now answer nothing, even requests that do not touch C, and A, waiting on B, runs out of threads next. One slow dependency took down the whole chain within seconds.

A house has a circuit breaker in its fuse box. If something draws too much current, the breaker trips and cuts that circuit, so the wiring does not overheat and burn the house down. After the fault is fixed, you flip it back. The house sacrifices one circuit to save the whole.

• -The overloaded appliance is the struggling dependency.
• -The breaker tripping is the circuit breaker opening: calls fail fast instead of piling up.
• -Flipping the breaker back on after the fix is the breaker closing again.

The diagram shows the breaker between a caller and a struggling dependency:

A circuit breaker watches calls to a dependency and has three states. Closed is normal: calls pass through, and failures are counted. Open means failures crossed a threshold, say half of the last 20 calls failed, so the breaker stops sending calls and answers at once with a fallback such as a cached value, a default, or a clear error. Half-open comes after a cooldown, say 30 seconds: the breaker lets one trial request through. If it succeeds, the breaker closes and traffic resumes; if it fails, the breaker opens again for another cooldown. In the example above, an open breaker would have answered in about a millisecond instead of ten seconds, and B would never have run out of threads. The card below describes the component as it appears in the lab:

A retry can rescue a request that failed for a momentary reason, but done carelessly it makes things worse. If a service is struggling and every client retries immediately, the service receives its normal load plus a wave of retries, and may never recover. This is a retry storm. The remedy is exponential backoff with jitter: wait longer after each failure (for example 1 second, then 2, then 4), and add a random amount to each wait so that thousands of clients do not all come back at the same instant.

• -Retry only what is safe to repeat, such as reads and idempotent operations, and keep a cap on attempts, commonly three.
• -Always set a timeout. Without one, a hung call never fails, so it is never retried and never frees its thread.
• -Do not retry errors that cannot change, such as a 400 Bad Request. Retry timeouts and 503 responses.
• -Combine retries with a breaker: when the breaker is open, do not retry at all.

A ship is divided into watertight compartments, so a hole floods one compartment and not the whole hull. The same idea in software: give each dependency its own pool of threads or connections. Say a service has 100 threads, split into 40 for payments, 40 for the catalog and 20 for recommendations. If recommendations become slow, they can use up only their own 20 threads. Payments and the catalog carry on unharmed, and the page shows without recommendations instead of failing.