Idempotent Transaction Handling: Preventing Duplicate Charges at All Costs

The $7.2 Million Double-Charge

In 2019, a fintech startup double-charged 12,000 customers during a deployment that introduced a network timeout bug. Total overcharges: $7.2 million. Refunding took 3 weeks. Trust was destroyed. The fix? A single HTTP header that should have been there from day one: Idempotency-Key.

The Timeout Dilemma Visualized

The Problem: Network Timeout After Successful Charge

  [Your Server]              [Payment Processor]
       |                            |
  POST /charge $29.99 ------------>|
       |                           | charge succeeds
       |                           | balance deducted
       |     TIMEOUT (no response) |
       X <--- network drops -----  |
       |                            |
  Did it work?  UNKNOWN            Customer charged $29.99
       |
  Option A: Don't retry            Option B: Retry
  Risk: goods not delivered         Risk: DOUBLE CHARGE
  if charge succeeded               $59.98 instead of $29.99
       |                            |
  Both options are bad.             Both options are bad.

  Option C: Idempotency key
  Retry with same key -> processor returns original result
  Safe. Always.

Implementation: Server-Side Idempotency

Idempotency Key Flow:

  Request 1 (first attempt):
  POST /api/charges
  Idempotency-Key: 550e8400-e29b-41d4-a716-446655440000
  Body: {amount: 2999, customer: 'cus_abc'}
       |
  Server checks idempotency store:
  Key exists? NO
       |
  Process charge -> SUCCESS {id: 'ch_xyz', amount: 2999}
       |
  Store: {key: '550e...', status: 'complete', response: {...}}
       |
  Return 200 {id: 'ch_xyz', amount: 2999}


  Request 2 (retry after timeout):
  POST /api/charges
  Idempotency-Key: 550e8400-e29b-41d4-a716-446655440000  <- SAME KEY
  Body: {amount: 2999, customer: 'cus_abc'}
       |
  Server checks idempotency store:
  Key exists? YES, status: 'complete'
       |
  Return STORED response: 200 {id: 'ch_xyz', amount: 2999}
  NO reprocessing. No double charge.

Race Condition: Two Retries Arrive Simultaneously

Handling Concurrent Retries:

  Request A and Request B arrive with SAME idempotency key
  at the SAME TIME (before either finishes processing)

  Solution: database-level locking on the key

  -- Use INSERT ... ON CONFLICT for atomic check-and-set
  INSERT INTO idempotency_keys (key, status, locked_at)
  VALUES ('550e...', 'processing', NOW())
  ON CONFLICT (key) DO NOTHING
  RETURNING id;

  Request A: INSERT succeeds -> processes charge
  Request B: INSERT returns nothing (conflict) -> waits/retries

  When Request A completes:
  UPDATE idempotency_keys SET status='complete', response='{...}'
  WHERE key = '550e...'

  Request B retries -> finds status='complete' -> returns stored response

  Key details:
  - TTL on keys: 24 hours (prevent unbounded growth)
  - Store in Redis for speed, PostgreSQL for durability
  - Key format: client-generated UUID (never server-generated)

Interview Tip

When asked about handling retries in distributed systems, say: 'Every mutating API endpoint accepts an Idempotency-Key header - a client-generated UUID. The server atomically checks-and-locks the key before processing (using INSERT ON CONFLICT). If the key exists with a completed status, return the stored response without reprocessing. Keys expire after 24 hours to prevent unbounded storage. This makes all retries safe - the client can retry any failed network request without fear of duplicate side effects.'

<

/blockquote>

Key Takeaway

Idempotency keys make retries safe by ensuring an operation produces the same result no matter how many times it is attempted. Generate a UUID before the request, send it every time you retry, and let the server deduplicate. This simple pattern prevents an entire class of financial bugs.