Circuit Breakers, Retries & Bulkheads
You'll learn to
- -Explain how a circuit breaker prevents cascading failure
Cascading Failure
Service A calls Service B, which calls Service C. If C becomes slow, requests pile up waiting on B's side; B's own capacity fills up, and now A is waiting on a slow B too. One slow dependency, left unchecked, can take down an entire chain of otherwise-healthy services within minutes.
Think of a circuit breaker in your house's electrical panel. When too much electricity flows through a wire (a short circuit), the breaker TRIPS and cuts the power. Preventing a fire. A software circuit breaker does the exact same thing for service-to-service calls. Suppose your Order Service calls the Payment Service, and the Payment Service starts timing out (it's overloaded). Without a circuit breaker, the Order Service sends request after request, each one waiting 30 seconds before timing out. Soon, the Order Service itself is stuck with hundreds of pending requests, its thread pool is exhausted, and it crashes too. That's a cascade failure. One sick service takes down the entire system like dominoes. A circuit breaker monitors the failure rate. Once it exceeds a threshold (say, 50% of calls fail), the breaker 'OPENS' and immediately returns an error (or a fallback response like 'Payment currently unavailable, please try again in 60 seconds') WITHOUT even calling the Payment Service. This protects both the caller (Order Service doesn't waste resources) and the failing service (Payment Service gets breathing room to recover). After a cooldown period, the breaker allows a few test requests through ('half-open state'). If those succeed, the breaker closes and normal traffic resumes.
Examples: Resilience4j (Java/Kotlin, the modern standard), Hystrix (Netflix, pioneered the pattern, now in maintenance mode), Polly (.NET), Sentinel (Alibaba), Envoy proxy (built-in outlier detection acts as a circuit breaker), AWS App Mesh
A circuit breaker sits between a caller and a struggling dependency, failing fast instead of letting requests pile up.
A circuit breaker watches calls to a dependency, and once failures cross a threshold, it "opens": failing fast with a fallback instead of letting new requests pile up waiting on a service that's clearly struggling. After a cooldown, it lets a trial request through; if that succeeds, it closes again and traffic resumes normally.
Retries (With Backoff)
Naively retrying a failed request immediately just adds more load to an already-struggling service: a "retry storm." Exponential backoff (wait longer between each retry) combined with jitter (randomizing the exact wait) spreads retries out instead of having every client hammer the recovering service at the same instant.
Bulkheads
Named after a ship's watertight compartments: isolate the thread pool or connection pool used for each dependency, so a slow or failing dependency can only exhaust the resources allocated to it, not the resources every other dependency also needs.
A circuit breaker just opened for a failing payment provider. What should the caller do instead of just failing the user's request?
"Show the user a generic error message."
"A fallback is usually better than a bare failure, though it depends on the operation: queue the payment for retry once the breaker closes and tell the user 'order received, payment processing' rather than losing the order entirely, or fail over to a backup payment provider if one exists. The circuit breaker's job is to fail fast and stop the pile-up; the caller's job is deciding what 'failed fast' should actually look like to the end user."