Distributed Locking with Redlock: Coordinating Across Nodes

The Double-Charge Nightmare

You have 5 instances of your payment service behind a load balancer. A webhook arrives from Stripe saying payment for order #123 completed. Due to a retry, the webhook hits two instances simultaneously. Both check the database - order #123 is unpaid. Both process the payment. Customer gets charged twice. Refund, apology, trust lost. A distributed lock prevents this.

Simple Redis Lock

Redis Single-Instance Lock:

  # Acquire lock
  SET lock:order:123 worker-7a NX EX 30
  #     key             value    only-if-    expires in
  #                              not-exists  30 seconds

  # If SET returns OK   -> you have the lock
  # If SET returns nil  -> someone else has it, back off

  # Do your work...
  process_payment(order_123)

  # Release lock (only if you still own it!)
  # Use Lua script for atomicity:
  if redis.call('GET', 'lock:order:123') == 'worker-7a' then
      return redis.call('DEL', 'lock:order:123')
  end

  Why check ownership before delete?
  - Your lock might have EXPIRED while you were working
  - Another worker acquired it
  - If you blindly DEL, you release THEIR lock
  - Now two workers are in the critical section

Redlock: Fault-Tolerant Distributed Lock

A single Redis instance is a single point of failure. Redlock uses N independent Redis instances (typically 5) for redundancy.

Redlock Algorithm (N=5 Redis instances):

  1. Record current time T1
  2. Try to acquire lock on all 5 instances sequentially
     (with short per-instance timeout, e.g., 5-10ms)
  3. Record current time T2

  Lock acquired if:
  [a] Got lock on majority (3+ of 5 instances)
  [b] Total time (T2-T1) < lock TTL

  Effective TTL = original TTL - (T2-T1)

  If lock NOT acquired:
  Release on ALL 5 instances (even successful ones)

  Example timeline:
  Instance 1: SET OK  (2ms)
  Instance 2: SET OK  (3ms)
  Instance 3: SET OK  (1ms)   -> 3/5 = majority!
  Instance 4: TIMEOUT (10ms)  -> skip
  Instance 5: SET OK  (2ms)

  Total: 18ms. TTL was 30s. Effective TTL: 29.982s
  Lock acquired! Can tolerate 2 instance failures.

The Kleppmann vs Antirez Debate

In 2016, Martin Kleppmann (author of Designing Data-Intensive Applications) published a famous critique of Redlock. The Redis creator (Antirez) responded. This is one of the most educational debates in distributed systems.

Kleppmann's concern - GC Pause Breaks the Lock:

  Time 0s:     Worker A acquires lock (TTL=30s)
  Time 0.001s: Worker A's JVM enters GC pause...
  Time 31s:    Lock expires! Worker A is still paused.
  Time 31s:    Worker B acquires lock legitimately.
  Time 35s:    Worker A wakes up, thinks it has lock.
  Time 35s:    BOTH workers in critical section!

  Kleppmann's fix: Use 'fencing tokens'
  - Lock service issues monotonically increasing token
  - Storage layer rejects writes with old tokens
  - Even if stale worker wakes up, its token is rejected

  The lesson: Distributed locks provide BEST-EFFORT
  mutual exclusion, not absolute safety.
  For true safety, use fencing tokens or consensus systems.

When to Use What

Need                     Solution             Notes
-----------------------  -------------------  ----------------------
Efficiency lock           Redis SET NX         Duplicate work is OK,
(avoid duplicate work)                         just wasteful
Correctness lock          ZooKeeper / etcd     Consensus-based, safe
(double-charge = bad)     with fencing tokens  against GC pauses
Simple leader election    etcd lease           Built for this
Distributed rate limit    Redis + Lua          Not a lock, but similar

Interview Tip

When discussing distributed locks, say: 'For efficiency locks - where duplicates are wasteful but not catastrophic - I would use Redis SET NX with an expiration. For correctness locks - like payment processing where duplicates cause real harm - I would use a consensus-based system like etcd or ZooKeeper with fencing tokens. The key insight from the Kleppmann-Antirez debate is that Redis locks can be broken by GC pauses or clock drift, so they provide best-effort, not absolute mutual exclusion.'

Key Takeaway

Distributed locks coordinate exclusive access across multiple service instances. Redis SET NX is simple. Redlock adds fault tolerance with multiple Redis nodes. For non-critical coordination, Redis locks work well. For critical financial operations, use a proper consensus system like etcd or ZooKeeper with fencing tokens.