The $7.2 Million Double-Charge
In 2019, a fintech startup double-charged 12,000 customers during a deployment that introduced a network timeout bug. Total overcharges: $7.2 million. Refunding took three weeks, and trust took even longer to rebuild. The root cause traced back to missing idempotent transaction handling, something that should have been baked into the payment flow from day one. The fix ended up being a single HTTP header: Idempotency-Key.
The Timeout Dilemma Visualized
The Problem: Network Timeout After Successful Charge
[Your Server] [Payment Processor]
| |
POST /charge $29.99 ------------>|
| | charge succeeds
| | balance deducted
| TIMEOUT (no response) |
X <--- network drops ----- |
| |
Did it work? UNKNOWN Customer charged $29.99
|
Option A: Don't retry Option B: Retry
Risk: goods not delivered Risk: DOUBLE CHARGE
if charge succeeded $59.98 instead of $29.99
| |
Both options are bad. Both options are bad.
Option C: Idempotency key
Retry with same key -> processor returns original result
Safe. Always.Implementation: Server-Side Idempotency
Idempotency Key Flow:
Request 1 (first attempt):
POST /api/charges
Idempotency-Key: 550e8400-e29b-41d4-a716-446655440000
Body: {amount: 2999, customer: 'cus_abc'}
|
Server checks idempotency store:
Key exists? NO
|
Process charge -> SUCCESS {id: 'ch_xyz', amount: 2999}
|
Store: {key: '550e...', status: 'complete', response: {...}}
|
Return 200 {id: 'ch_xyz', amount: 2999}
Request 2 (retry after timeout):
POST /api/charges
Idempotency-Key: 550e8400-e29b-41d4-a716-446655440000 <- SAME KEY
Body: {amount: 2999, customer: 'cus_abc'}
|
Server checks idempotency store:
Key exists? YES, status: 'complete'
|
Return STORED response: 200 {id: 'ch_xyz', amount: 2999}
NO reprocessing. No double charge.Race Condition: Two Retries Arrive Simultaneously
Handling Concurrent Retries:
Request A and Request B arrive with SAME idempotency key
at the SAME TIME (before either finishes processing)
Solution: database-level locking on the key
-- Use INSERT ... ON CONFLICT for atomic check-and-set
INSERT INTO idempotency_keys (key, status, locked_at)
VALUES ('550e...', 'processing', NOW())
ON CONFLICT (key) DO NOTHING
RETURNING id;
Request A: INSERT succeeds -> processes charge
Request B: INSERT returns nothing (conflict) -> waits/retries
When Request A completes:
UPDATE idempotency_keys SET status='complete', response='{...}'
WHERE key = '550e...'
Request B retries -> finds status='complete' -> returns stored response
Key details:
- TTL on keys: 24 hours (prevent unbounded growth)
- Store in Redis for speed, PostgreSQL for durability
- Key format: client-generated UUID (never server-generated)Interview Tip
When asked about handling retries in distributed systems, say: 'Every mutating API endpoint accepts an Idempotency-Key header - a client-generated UUID. The server atomically checks-and-locks the key before processing (using INSERT ON CONFLICT). If the key exists with a completed status, return the stored response without reprocessing. Keys expire after 24 hours to prevent unbounded storage. This makes all retries safe - the client can retry any failed network request without fear of duplicate side effects.
Key Takeaway
Idempotency keys make retries safe by ensuring an operation produces the same result no matter how many times it is attempted. Generate a UUID before the request, send it every time you retry, and let the server deduplicate. This simple pattern prevents an entire class of financial bugs.