Designing for Failure: Chaos Engineering

Most teams test whether their system works under normal conditions. Chaos engineering tests whether it fails gracefully. It does this by deliberately injecting real failure, such as killing a live instance, adding artificial delay, or cutting a network link between services, in a controlled way, with the team watching. It sounds reckless, and the reasoning behind it is the opposite: you will face these failures eventually, so the only choice is whether you meet them on your own schedule or unexpectedly at 3 am.

A building has fire exits, alarms and an evacuation plan. None of it is worth anything if the first time it is used is during a real fire, when someone finds the exit blocked or the alarm silent. So the building holds fire drills: at a planned time, with the wardens ready, to find the blocked door while it is harmless. Chaos engineering is the fire drill for your system.

• -The fire exits and alarms are your circuit breakers, retries, replicas and failover paths.
• -The drill is the injected failure, at a time you choose.
• -The wardens watching are the engineers observing the system, with a way to stop.
• -The blocked door you find is the weakness you fix before a real incident.

• -Define the steady state: what normal looks like in numbers, for example an error rate of 0.02% and a p99 latency of 300 ms.
• -Form a hypothesis: if this failure happens, the steady state will hold.
• -Limit the blast radius: start small, one instance, one percent of traffic, a test environment first.
• -Inject the failure, and watch the metrics, ready to stop at once if the damage is bigger than planned.
• -Compare with the hypothesis, and fix whatever it exposed. Then run it again.

Say the hypothesis is: if one of three Orders instances is killed, the error rate stays under 0.1%. At 11:00 the team terminates one instance. The error rate climbs to 0.4% for about 30 seconds, then falls back. The hypothesis was wrong, and the cause is easy to see: the load balancer checks health every 10 seconds and removes an instance after 3 failed checks, so for 10 x 3 = 30 seconds it kept sending requests to a dead server. The fix is to check more often and to let clients retry on another instance. After the change the team repeats the experiment and the error rate stays under 0.1%. They found a real weakness at a time of their choosing, with everyone watching.

• -Kill an instance or a whole zone: does failover work, and how fast?
• -Add latency to a dependency: do timeouts, retries and circuit breakers behave as designed?
• -Cut the network between two services: what do callers do?
• -Make a dependency return errors: is there a sensible fallback?
• -Fill a disk or exhaust memory: do alerts fire before users notice?

Netflix made the idea famous with Chaos Monkey, which randomly terminated production instances. Today there are tools for it, including Gremlin and AWS Fault Injection Service, and many teams run regular game days: scheduled sessions where a failure is injected and the whole team practices responding.

• -You can see the system: without metrics, logs and traces you cannot tell what the experiment did.
• -You have an agreed steady state and an SLO, so everyone can see whether the hypothesis held.
• -You can stop: there is an abort switch, and someone has the job of pressing it.
• -People know it is happening, and on-call is ready, ideally during working hours.

And one reason not to: if you already know the system cannot survive the failure, there is nothing to learn from breaking it. Fix the known weakness first, then use chaos to find the unknown ones.