Distributed Locking with Redlock: Coordinating Across Nodes

The Double-Charge Nightmare

You have 5 instances of your payment service behind a load balancer. A webhook arrives from Stripe saying payment for order #123 completed. Due to a retry, the webhook hits two instances simultaneously. Both check the database, order #123 is unpaid. Both process the payment. Customer gets charged twice. Refund, apology, trust lost. Distributed Locking with Redlock can prevent this by ensuring that only one instance acquires the lock and processes the payment at a time.

Simple Redis Lock

Redis Single-Instance Lock:

  # Acquire lock
  SET lock:order:123 worker-7a NX EX 30
  #     key             value    only-if-    expires in
  #                              not-exists  30 seconds

  # If SET returns OK   -> you have the lock
  # If SET returns nil  -> someone else has it, back off

  # Do your work...
  process_payment(order_123)

  # Release lock (only if you still own it!)
  # Use Lua script for atomicity:
  if redis.call('GET', 'lock:order:123') == 'worker-7a' then
      return redis.call('DEL', 'lock:order:123')
  end

  Why check ownership before delete?
  - Your lock might have EXPIRED while you were working
  - Another worker acquired it
  - If you blindly DEL, you release THEIR lock
  - Now two workers are in the critical section

Redlock: Fault-Tolerant Distributed Lock

A single Redis instance is a single point of failure. Redlock uses N independent Redis instances (typically 5) for redundancy.

Redlock Algorithm (N=5 Redis instances):

  1. Record current time T1
  2. Try to acquire lock on all 5 instances sequentially
     (with short per-instance timeout, e.g., 5-10ms)
  3. Record current time T2

  Lock acquired if:
  [a] Got lock on majority (3+ of 5 instances)
  [b] Total time (T2-T1) < lock TTL

  Effective TTL = original TTL - (T2-T1)

  If lock NOT acquired:
  Release on ALL 5 instances (even successful ones)

  Example timeline:
  Instance 1: SET OK  (2ms)
  Instance 2: SET OK  (3ms)
  Instance 3: SET OK  (1ms)   -> 3/5 = majority!
  Instance 4: TIMEOUT (10ms)  -> skip
  Instance 5: SET OK  (2ms)

  Total: 18ms. TTL was 30s. Effective TTL: 29.982s
  Lock acquired! Can tolerate 2 instance failures.

The Kleppmann vs Antirez Debate

In 2016, Martin Kleppmann (author of Designing Data-Intensive Applications) published a famous critique of Redlock. The Redis creator (Antirez) responded. This is one of the most educational debates in distributed systems.

Kleppmann's concern - GC Pause Breaks the Lock:

  Time 0s:     Worker A acquires lock (TTL=30s)
  Time 0.001s: Worker A's JVM enters GC pause...
  Time 31s:    Lock expires! Worker A is still paused.
  Time 31s:    Worker B acquires lock legitimately.
  Time 35s:    Worker A wakes up, thinks it has lock.
  Time 35s:    BOTH workers in critical section!

  Kleppmann's fix: Use 'fencing tokens'
  - Lock service issues monotonically increasing token
  - Storage layer rejects writes with old tokens
  - Even if stale worker wakes up, its token is rejected

  The lesson: Distributed locks provide BEST-EFFORT
  mutual exclusion, not absolute safety.
  For true safety, use fencing tokens or consensus systems.

When to Use What

Need                     Solution             Notes
-----------------------  -------------------  ----------------------
Efficiency lock           Redis SET NX         Duplicate work is OK,
(avoid duplicate work)                         just wasteful
Correctness lock          ZooKeeper / etcd     Consensus-based, safe
(double-charge = bad)     with fencing tokens  against GC pauses
Simple leader election    etcd lease           Built for this
Distributed rate limit    Redis + Lua          Not a lock, but similar

Interview Tip

When discussing distributed locks, say: 'For efficiency locks - where duplicates are wasteful but not catastrophic - I would use Redis SET NX with an expiration. For correctness locks - like payment processing where duplicates cause real harm - I would use a consensus-based system like etcd or ZooKeeper with fencing tokens. The key insight from the Kleppmann-Antirez debate is that Redis locks can be broken by GC pauses or clock drift, so they provide best-effort, not absolute mutual exclusion.'

Key Takeaway

Distributed locks coordinate exclusive access across multiple service instances. Redis SET NX is simple. Redlock adds fault tolerance with multiple Redis nodes. For non-critical coordination, Redis locks work well. For critical financial operations, use a proper consensus system like etcd or ZooKeeper with fencing tokens.