Circuit Breakers & Fallbacks
You'll learn to
- -Trace a circuit breaker through closed, open and half-open, and say what each state protects
- -Return a useful fallback while the circuit is open, instead of failing the whole page
- -Explain what a bulkhead isolates that a circuit breaker does not, and use both
When a downstream dependency starts failing, continuing to call it on every request, and waiting for each call to time out, wastes resources and adds delay for every caller. This often turns a bad situation into a worse one. A circuit breaker stops calling a failing dependency for a while. It fails fast instead of failing slowly.
An Analogy: The Fuse in Your House
A fuse cuts the power when a circuit draws too much, instead of letting the wires overheat. It protects the house. You check the fault, then reset it, and normal service resumes. A circuit breaker in software is the same, with one improvement: it can reset itself by testing carefully whether the fault has gone.
The Three States
- -Closed: normal operation. Requests pass through to the dependency, and failures are counted.
- -Open: too many recent failures. Requests fail at once, without trying the dependency, giving it room to recover instead of continuing to hit it.
- -Half-open: after a recovery timeout, a limited number of requests are let through as a test. If they succeed, the circuit closes. If they fail, it opens again and waits again.
The half-open state is what makes the breaker self-healing. Without it you would have to reset the circuit by hand, or you would flip back to closed after a timer and hammer a dependency that has not actually recovered. The demo runs one through the states with a fake clock, a recommendations service that is down, and a fallback of an empty list.
Fallbacks: What to Return When the Circuit Is Open
Failing fast is better than failing slowly, but failing is often not the best available response. A product-recommendation service being down does not have to break checkout. Returning a generic, cached or empty set of recommendations, as in the demo, lets the rest of the page work. A well-designed fallback gives a degraded but working experience for the failing dependency's part of the system, instead of treating every failure as equally catastrophic.
The threshold and the recovery timeout are tuning parameters for your workload, not universal constants. Too sensitive a threshold trips the breaker on normal brief blips. Too lenient a threshold keeps hammering a struggling dependency for too long before giving it room to recover.
Bulkhead: Isolating One Failure So It Doesn't Sink Everything Else
A circuit breaker decides whether to keep calling a failing dependency. It does not address a different problem: one slow dependency quietly using every worker thread or connection in a shared pool, so requests to all the other healthy dependencies that share that pool starve. A bulkhead, named after the watertight compartments in a ship's hull, caps how many concurrent calls any one dependency may use. The last lines of the demo show a shared pool being emptied by one slow service, and separate pools containing the damage.
# recommendations-service is slow today, but it's just ONE of five
# downstream calls this handler makes - without isolation, every
# thread in the shared pool can end up blocked waiting on it,
# leaving zero threads free to serve calls to the other four
# (healthy) dependencies.
pool = ThreadPoolExecutor(max_workers=50) # shared by all downstream calls
# A bulkhead gives each dependency its own bounded slice instead:
pools = {
"recommendations": ThreadPoolExecutor(max_workers=10),
"inventory": ThreadPoolExecutor(max_workers=10),
"pricing": ThreadPoolExecutor(max_workers=10),
}
# recommendations-service being slow now only exhausts its own
# 10 workers - inventory and pricing keep serving normally.A bulkhead and a circuit breaker solve adjacent problems, and production systems normally use both. The bulkhead limits the blast radius of one dependency's slowness on shared resources. The breaker decides when to stop calling that dependency altogether once it is clearly failing and not merely slow.
Interview Signal is part of Pro
See a real weak answer next to a real strong one for this exact topic.
Quiz is part of Pro
Test what you just read with a short quiz, and bank the XP.
Design Circuit Breaker API in the API Design Lab's Full System Design act.
Discussion0
Join the Discussion
Sign in to leave comments, reply to others, or like insights.
No comments yet. Be the first to start the thread!