The Cascading Failure That Took Down Amazon
During a 2017 AWS incident, a single overloaded dependency service started returning errors slowly (30-second timeouts instead of instant failures). Services calling it tied up all their worker threads waiting for responses. Those services then appeared slow to their callers, who also tied up threads. Within minutes, hundreds of microservices were frozen - a cascading failure that started from one slow dependency. Circuit breakers would have stopped it in seconds by fast-failing instead of waiting.
This pattern repeats across every large distributed system. Netflix discovered it early and built Hystrix, the circuit breaker library that became the industry standard. The lesson is universal: in a microservices architecture, one unhealthy service can poison every service that depends on it, and every service that depends on those, until your entire platform is frozen. Health checks and circuit breakers are the two mechanisms that prevent this domino effect.
Health Check Architecture
Health checks answer a simple question: is this server ready to receive traffic? But the way you answer that question matters enormously. A shallow health check (is the process alive?) runs in under a millisecond and is used by load balancers to route traffic. A deep health check (can this service actually do its job?) verifies database connectivity, cache availability, and disk space, and is used by orchestration tools like Kubernetes to decide if a pod should receive requests.
Health Check Design:
GET /health (shallow - for load balancers)
Response: {"status": "healthy"}
Checks: app process is running
Speed: <1ms
Use: LB decides if this instance gets traffic
GET /health/ready (deep - for orchestration)
Response: {
"status": "healthy",
"checks": {
"database": {"status": "ok", "latency_ms": 3},
"redis": {"status": "ok", "latency_ms": 1},
"disk": {"status": "ok", "free_gb": 45}
}
}
Checks: critical dependencies reachable
Speed: <100ms (with timeouts per dependency)
Use: K8s readiness probe, monitoring dashboards
Anti-pattern: checking ALL downstream services
If ServiceC depends on ServiceD which depends on ServiceE,
ServiceC's health check should NOT check ServiceE.
Otherwise one unhealthy leaf makes every service 'unhealthy'.
Rule: only check YOUR direct dependencies.Liveness vs Readiness in Kubernetes
Kubernetes makes a critical distinction: a liveness probe tells K8s whether to restart a pod (is the process stuck?), while a readiness probe tells K8s whether to send traffic to it (can it handle requests?). A common mistake is making the liveness probe check database connectivity. If the database goes down, K8s restarts all your pods, which then all try to reconnect simultaneously, overwhelming the database when it recovers. The liveness probe should only check if the process itself is healthy. The readiness probe should check external dependencies.
Circuit Breaker State Machine
A circuit breaker wraps every outbound call to a dependency. It works like an electrical circuit breaker: when too many failures occur, it trips open and stops current (requests) from flowing. This protects both the caller (no more wasted threads waiting for timeouts) and the callee (no more traffic when it is struggling). After a cooldown period, it allows a few test requests through to see if the dependency has recovered.
Circuit Breaker States:
CLOSED (normal operation)
All requests pass through.
Monitor: count failures in sliding window.
|
5 failures in 10 seconds?
|
v
OPEN (tripped - fail fast)
All requests immediately return error.
No call to downstream service.
Threads are freed instantly.
Wait: 30 seconds cooldown.
|
Cooldown expired?
|
v
HALF-OPEN (testing recovery)
Allow 1 test request through.
|
+----+----+
| |
Success Failure
| |
v v
CLOSED OPEN
(reset) (restart cooldown)
Example config (Resilience4j/Hystrix):
failure_threshold: 5 errors in 10 sec window
open_duration: 30 seconds
half_open_requests: 3 (need 3 consecutive successes)
timeout: 2 seconds (max wait per call)Fallback Strategies When the Circuit Is Open
When a circuit breaker trips open, you need to decide what to return to the caller. There are several strategies: return a cached response (stale but functional), return a default value (an empty list, a generic recommendation), degrade gracefully (show a simpler version of the feature), or fail fast with a clear error. Netflix uses all four depending on the context - their recommendation service falls back to a generic 'trending' list, while their payment service fails fast because showing wrong prices is worse than showing an error.
Combining Health Checks and Circuit Breakers
Health checks and circuit breakers complement each other but work at different levels. Health checks are passive and server-centric: the load balancer periodically asks each server 'are you healthy?' and removes unhealthy ones. Circuit breakers are active and call-centric: each client monitors its own calls to a dependency and stops calling if failures exceed a threshold. In a well-designed system, both are present. A server might pass its health check (the process is running, the database is reachable) but still be slow enough to trigger circuit breakers in its callers.
Interview Tip
Health checks and circuit breakers are complementary: health checks detect failed SERVERS (load balancer removes them), circuit breakers detect failed SERVICES (callers stop sending requests). When discussing microservices reliability, always mention both: 'Each service has a /health endpoint checked by the load balancer every 5 seconds, AND each outbound call is wrapped in a circuit breaker that trips after 5 failures in 10 seconds to prevent cascading failures.'
Key Takeaway
Health checks let the infrastructure remove failed servers from the pool. Circuit breakers let your code stop calling failed services before threads are exhausted. Together, they form the foundation of self-healing systems. Without them, a single slow dependency can cascade into a platform-wide outage in minutes.