Retry with Exponential Backoff: Handling Transient Failures Gracefully

The AWS DynamoDB Thundering Herd

In 2015, a brief DynamoDB capacity issue caused thousands of AWS Lambda functions to retry simultaneously. Without backoff, every function retried every second. The retry storm generated 10x the normal load, turning a 2-second blip into a 20-minute outage. The retries were the problem, not the original failure. This is why exponential backoff with jitter is not optional - it is essential.

Naive Retry vs Exponential Backoff vs Backoff + Jitter

1000 clients fail at T=0. All retry:

Naive retry (every 1 second):
  T=0:  1000 requests (original)
  T=1:  1000 retries  -> 2000 total load
  T=2:  1000 retries  -> 2000 total load
  T=3:  1000 retries  -> 2000 total load
  Server NEVER recovers. Constant 2x overload.

Exponential backoff (no jitter):
  T=0:  1000 requests
  T=1:  1000 retries  (all wait 1s)
  T=3:  1000 retries  (all wait 2s) <- synchronized spikes!
  T=7:  1000 retries  (all wait 4s)
  Spikes are smaller but still synchronized.

Exponential backoff WITH jitter:
  T=0:    1000 requests
  T=0-2:  ~330 retries  (random 0-2s)
  T=1-6:  ~330 retries  (random 1-6s)
  T=3-14: ~330 retries  (random 3-14s)
  Load is SPREAD EVENLY. Server can recover.

Implementation

# Python implementation:
import random
import time

def retry_with_backoff(func, max_retries=5, base_delay=1.0):
    for attempt in range(max_retries):
        try:
            return func()
        except RetryableError:
            if attempt == max_retries - 1:
                raise  # exhausted all retries

            # Exponential backoff
            delay = base_delay * (2 ** attempt)

            # Full jitter: random between 0 and delay
            jittered_delay = random.uniform(0, delay)

            # Cap at 60 seconds max
            jittered_delay = min(jittered_delay, 60.0)

            time.sleep(jittered_delay)

# Attempt delays (before jitter):
# Attempt 0: 1s  (1 * 2^0)
# Attempt 1: 2s  (1 * 2^1)
# Attempt 2: 4s  (1 * 2^2)
# Attempt 3: 8s  (1 * 2^3)
# Attempt 4: 16s (1 * 2^4) -> give up

What to Retry (and What NOT to Retry)

HTTP Status   Retry?  Why
-----------   ------  --------------------------------
200 OK        No      Success!
400 Bad Req   NO      Client error. Same request = same error.
401 Unauth    NO      Auth problem. Retry will not fix it.
403 Forbidden NO      Permission denied. Will not change.
404 Not Found NO      Resource does not exist.
408 Timeout   YES     Server did not respond in time.
429 Rate Lmt  YES     Use Retry-After header for delay!
500 Internal  MAYBE   Could be transient or permanent.
502 Bad GW    YES     Upstream server issue, likely transient.
503 Unavail   YES     Server overloaded, back off!
504 GW Tmout  YES     Upstream timeout, likely transient.

CRITICAL: Only retry IDEMPOTENT operations!
  GET /users/42           -> safe to retry
  DELETE /users/42        -> safe (deleting twice = same result)
  POST /payments          -> NOT safe (double charge!)
  POST /payments (with    -> SAFE (idempotency key
    Idempotency-Key: X)      deduplicates on server)

Retry Budget: Preventing Retry Storms

Advanced: Retry Budgets

  Instead of unlimited retries, set a budget:
  'Retries should be at most 10% of total requests'

  Normal traffic:  1000 req/s, 10 retries/s (1%) -> OK
  Degraded:        1000 req/s, 50 retries/s (5%) -> OK
  Overloaded:      1000 req/s, 150 retries/s      -> STOP
                   Budget exceeded! No more retries.

  Google's approach: each client tracks its retry ratio.
  If retries exceed 10% of successful requests,
  stop retrying until the ratio improves.
  This prevents retry storms by design.

Interview Tip

When discussing error handling in distributed systems, say: 'I would implement retry with exponential backoff and full jitter for all transient failures (5xx, timeouts, rate limits). The delay doubles each attempt (1s, 2s, 4s, 8s) with random jitter to desynchronize concurrent retries. I would cap at 5 retries with a 60-second maximum delay. Critically, I would only retry idempotent operations - for non-idempotent calls like payments, I would use idempotency keys so the server can safely deduplicate. For high-traffic systems, I would add a retry budget (max 10% retry ratio) to prevent retry storms.'

<

/blockquote>

Key Takeaway

Exponential backoff with jitter prevents retry storms while still recovering from transient failures. Only retry errors that are transient and operations that are idempotent. Set a maximum retry count (usually 3-5) to avoid infinite retry loops.