Idempotency Keys: Making Retries Safe in Distributed Systems

The $150,000 Double-Charge That Built Stripe's Idempotency System

In Stripe's early days, a merchant's checkout page timed out during a $150,000 B2B payment. The buyer's browser retried automatically. Stripe processed the payment twice - $300,000 charged. The chargeback, customer support, and trust damage cost more than the duplicate charge itself. Stripe's engineering team built their idempotency key system the next week. Today, every Stripe API call that creates a resource or triggers a side effect requires an Idempotency-Key header. The same key submitted twice returns the cached result from the first call without re-executing. This single pattern prevents an entire class of distributed systems bugs.

How Idempotency Keys Work

Idempotency Key Flow:

  First Request:
  POST /v1/charges
  Idempotency-Key: idk_8f14e45f
  { "amount": 5000, "currency": "usd" }

  Server:
  1. Check idempotency store: idk_8f14e45f exists? NO
  2. Lock the key (prevent concurrent duplicates)
  3. Process: charge $50.00 to card
  4. Store: { key: idk_8f14e45f, response: {id: ch_abc, status: 200}, ttl: 24h }
  5. Return: { "id": "ch_abc", "amount": 5000, "status": "succeeded" }

  Retry (same key):
  POST /v1/charges
  Idempotency-Key: idk_8f14e45f
  { "amount": 5000, "currency": "usd" }

  Server:
  1. Check idempotency store: idk_8f14e45f exists? YES
  2. Return CACHED response: { "id": "ch_abc", "amount": 5000 }
  3. NO payment processed. Customer charged once. No duplicate.

  Concurrent Duplicate (race condition):
  Thread 1: POST (key=idk_8f14e45f) --> acquires lock --> processing...
  Thread 2: POST (key=idk_8f14e45f) --> lock taken --> WAIT
  Thread 1: done --> stores result --> releases lock
  Thread 2: lock acquired --> key exists --> return cached result

Implementation Architecture

Idempotency Store Options:

  Option          Pros                    Cons                Best For
  -----------     --------------------    -----------------   ---------------
  Redis           Fast, TTL built-in      Volatile (can lose) High-traffic APIs
  PostgreSQL      Durable, transactional  Slower              Financial systems
  DynamoDB        Scalable, durable       Cost                Serverless APIs

  Database Schema:
  CREATE TABLE idempotency_keys (
    key          VARCHAR(255) PRIMARY KEY,
    api_key_id   UUID NOT NULL,
    request_hash VARCHAR(64),     -- SHA-256 of request body
    response     JSONB,
    status_code  INTEGER,
    created_at   TIMESTAMP DEFAULT NOW(),
    expires_at   TIMESTAMP DEFAULT NOW() + INTERVAL '24 hours'
  );

  Request Hash Purpose:
  If client sends DIFFERENT body with SAME idempotency key:
  Key: idk_8f14e45f + body_hash: a1b2c3 (original: $50)
  Key: idk_8f14e45f + body_hash: d4e5f6 (retry: $100 <-- different!)
  Response: 422 'Idempotency key reused with different parameters'
  Prevents: using same key for different charges (likely a bug).

Which Operations Need Idempotency Keys

Idempotency Requirement by HTTP Method:

  Method    Naturally Idempotent?    Need Key?    Example
  ------    --------------------     ---------    ------------------
  GET       Yes (read-only)          No           GET /users/42
  PUT       Yes (full replace)       No           PUT /users/42
  DELETE    Yes (delete if exists)   No           DELETE /users/42
  PATCH     Maybe (depends)          Sometimes    PATCH /users/42
  POST      NO (creates/triggers)    YES!         POST /charges

  Critical POST Endpoints:
  - POST /payments       (double charge)
  - POST /orders         (duplicate order)
  - POST /transfers      (double transfer)
  - POST /notifications  (spam user)
  - POST /refunds        (double refund)

  Safe to Skip:
  - POST /search         (read-only, no side effects)
  - POST /login          (idempotent by nature)
  - POST /validate       (no state change)

Edge Cases and Pitfalls

• TTL too short: if key expires in 1 hour but client retries after 2 hours, the operation runs again. Use 24-72 hours.

• TTL too long: millions of keys accumulate. Use database TTL or periodic cleanup jobs.

• Concurrent requests: use distributed locks (Redis SETNX) to prevent two threads from processing the same key simultaneously.

• Different body, same key: always hash the request body. Reject mismatches with 422 error - it means the client has a bug.

• Server crash mid-processing: use database transactions so partial work is rolled back. The key should only be stored after successful completion.

Interview Tip

Idempotency keys are essential for any payment or order system design. Explain the full flow: client generates UUID, server checks store, processes or returns cached result. Mention three implementation details that show depth: (1) request body hashing to detect key reuse with different parameters, (2) distributed locks for concurrent duplicate requests, (3) the TTL trade-off (too short = duplicates on late retries, too long = storage bloat). Connect it to message delivery guarantees: at-least-once delivery + idempotent processing = effectively exactly-once. Stripe's Idempotency-Key header is the canonical reference.

Key Takeaway

Idempotency keys make retries safe by ensuring a request is processed at most once. The client generates a unique key (UUID), the server deduplicates by checking a store (Redis or database). Essential for any operation with real-world side effects: payments, orders, notifications, transfers. Hash the request body to detect key reuse with different parameters.