Circuit Breaker and Bulkhead: Stopping Cascading Failures

The 2017 S3 Outage That Broke the Internet

On February 28, 2017, a single typo in an S3 maintenance command took down a huge chunk of the internet. Slack, Trello, IFTTT, and thousands of apps crashed because they depended on S3 and had no protection against its failure. Services without circuit breakers exhausted their thread pools waiting for S3 responses that would never come. One failure cascaded into total system collapse.

Circuit Breaker: The Three States

Circuit Breaker State Machine:

         success
          +--+
          |  v
  +-------+-------+        failure threshold
  |    CLOSED     | -------(50% errors in 60s)------+
  | (normal flow) |                                  |
  +---------------+                                  v
         ^                                  +--------+--------+
         |                                  |      OPEN       |
         |          timeout (30s)            |  (fail fast -   |
         +-------<---------+--------<-------+   no calls)     |
         |                 |                +--------+--------+
         |        +--------+--------+
         |        |    HALF-OPEN    |
         +---<----| (probe: 1 call) |
        success   +-----------------+
                     if fails: back to OPEN

  Without circuit breaker:
    1000 requests * 30s timeout = 30,000 thread-seconds wasted
    All threads blocked -> service dies

  With circuit breaker (OPEN state):
    1000 requests * instant fail = ~0 thread-seconds
    Failure response in < 1ms
    Service stays healthy

Bulkhead: Containing the Blast Radius

Bulkhead Pattern (like ship compartments):

  Without Bulkhead:
  [Shared Thread Pool: 100 threads]
  Service A calls: B, C, D
  If B is slow -> 90 threads stuck waiting for B
  -> only 10 left for C and D
  -> C and D calls also start failing
  -> EVERYTHING fails because of B

  With Bulkhead:
  [Pool for B: 30 threads]  [Pool for C: 30 threads]  [Pool for D: 30 threads]
  If B is slow -> 30 threads stuck (B's pool only)
  C and D pools: untouched, working perfectly
  Failure CONTAINED to one compartment

  Ship analogy:
  +------+------+------+------+
  |      | FLOOD|      |      |  <- water in one compartment
  | Dry  | ~~~~ | Dry  | Dry  |     does not sink the ship
  +------+------+------+------+

Combining Both Patterns

Production Setup:

  [Service A]
      |
  +---+--- calls to Service B ---+
  |   [Bulkhead: 20 threads max] |
  |   [Circuit Breaker:          |
  |     threshold: 50% errors    |
  |     window: 60 seconds       |
  |     timeout: 30 seconds      |
  |     half-open probes: 3      |
  |   ]                          |
  |   [Fallback: return cached   |
  |    data or default response] |
  +------------------------------+

  Flow when B is down:
  1. First 10 requests fail (50% threshold not yet hit)
  2. Circuit OPENS after 50% failure rate
  3. All subsequent requests get instant fallback
  4. After 30s, circuit goes HALF-OPEN
  5. Probes B with 3 requests
  6. If B is back: circuit CLOSES
  7. If B still down: circuit stays OPEN

Implementation Libraries

Language     Library          Notes
-----------  ---------------  -------------------------
Java         Resilience4j     Modern, lightweight
Java         Hystrix          Netflix (deprecated)
.NET         Polly            Mature, widely used
Python       pybreaker        Simple circuit breaker
Go           gobreaker        Sony's implementation
Node.js      opossum          Red Hat maintained
K8s/Istio    Envoy proxy      Infrastructure-level

Infrastructure-level approach (service mesh):
  Envoy sidecar handles circuit breaking
  for ALL services without code changes.
  Configured via Istio DestinationRules.

Interview Tip

When discussing microservice resilience, say: 'I would protect every outbound service call with a circuit breaker. If a dependency's error rate exceeds 50% in a 60-second window, the circuit opens and all calls fail fast with a fallback response - cached data or a graceful degradation. I would pair this with bulkheads - separate thread pools per dependency so a slow Service B cannot exhaust threads needed for Service C. Together, these patterns contain failures to the affected dependency instead of letting them cascade through the entire system.'

<

/blockquote>

Key Takeaway

Circuit breakers detect failing dependencies and fail fast instead of waiting, preventing thread exhaustion. Bulkheads isolate failures to a bounded pool so one bad dependency cannot consume all resources. Together they are the foundation of resilient microservice architectures.