Webhook Design: Letting Your API Push Events to Clients

How Stripe Processes $1 Trillion in Webhooks Per Year

Stripe sends over 1 billion webhook events per year across millions of merchant endpoints. When a payment succeeds, fails, or gets disputed, Stripe's webhook system must reliably notify each merchant's server within seconds. Their system handles every possible failure: merchant servers that are down, slow, returning errors, or completely unreachable. Stripe retries failed webhooks for up to 3 days with exponential backoff, provides a dashboard showing every delivery attempt, and includes HMAC signatures so merchants can verify the events are genuine. This is the gold standard of webhook design - and every API platform should aspire to it.

Webhook Architecture

Webhook Delivery Pipeline:

  Event Occurs (e.g., payment.succeeded)
       |
  [Event Store]  <- persist first (never lose events)
       |
  [Fan-Out] --> Which webhooks are subscribed to this event type?
       |         Merchant A: payment.succeeded -> https://a.com/webhook
       |         Merchant B: payment.*         -> https://b.com/hooks
       |
  [Delivery Queue] (per-endpoint isolation)
       |
  [Worker Pool]
       |
  +----+---POST https://a.com/webhook---------+
  |  Headers:                                  |
  |    Content-Type: application/json          |
  |    X-Webhook-ID: evt_abc123                |
  |    X-Webhook-Timestamp: 1705312800         |
  |    X-Webhook-Signature: sha256=7d38cdd...  |
  |  Body:                                     |
  |    { "type": "payment.succeeded",          |
  |      "data": { "amount": 5000, ... } }     |
  +--------------------------------------------+
       |
  Response 200-299? -> Delivered!
  Response 5xx or timeout? -> Retry queue
  No response in 10s? -> Timeout -> Retry queue

Signature Verification

HMAC Signature Flow:

  Your Server (Sender):
  1. shared_secret = 'whsec_abc123...'  (known only to you and client)
  2. payload = JSON.stringify(body)
  3. timestamp = current_unix_time
  4. signed_content = f'{timestamp}.{payload}'
  5. signature = HMAC_SHA256(shared_secret, signed_content)
  6. Send header: X-Webhook-Signature: t=1705312800,v1=7d38cdd...

  Client's Server (Receiver):
  1. Extract timestamp and signature from header
  2. Check: is timestamp within 5 minutes of now? (prevents replay attacks)
  3. Reconstruct: signed_content = f'{timestamp}.{body}'
  4. Compute: expected = HMAC_SHA256(shared_secret, signed_content)
  5. Compare: expected === received_signature?
  6. Match -> authentic. Mismatch -> reject (attacker or corruption).

  Without verification:
  Attacker sends POST to https://merchant.com/webhook
  with fake { "type": "payment.succeeded", "amount": 99999 }
  Merchant processes fake payment. Money lost.

Retry Strategy and Failure Handling

Retry Schedule (Exponential Backoff):

  Attempt   Delay After Failure    Total Time Elapsed
  -------   ------------------     ------------------
  1         Immediate              0 min
  2         1 minute               1 min
  3         5 minutes              6 min
  4         30 minutes             36 min
  5         2 hours                2.5 hours
  6         8 hours                10.5 hours
  7         24 hours               34.5 hours
  8         48 hours               3.4 days (give up)

  Endpoint Health States:
  Healthy:   3+ consecutive successes
  Degraded:  1-2 recent failures (still delivering)
  Disabled:  5+ consecutive failures over 5 days
             -> Auto-disable + email notification to owner
             -> 'Your webhook endpoint has been disabled due to
                persistent failures. Please fix and re-enable.'

Webhook Best Practices

• Persist events before delivery - never lose events even if the delivery system fails.
• Per-endpoint isolation - a slow Merchant A should not delay deliveries to Merchant B. Use separate queues.
• Include event ID for idempotency - clients may receive the same event twice during retries. The event ID lets them deduplicate.
• Send minimal payload with resource URLs - include IDs and an API URL to fetch full details. Prevents large payloads and stale data.
• Provide webhook testing tools - let developers send test events to verify their endpoint before going live.
• Delivery logs dashboard - show every attempt with status code, response time, and response body for debugging.
• Event type filtering - let clients subscribe to specific events (payment.succeeded) not all events (reduces noise).

Interview Tip

Webhooks appear in notification system, payment platform, and API platform designs. The key architecture: event store (never lose) -> fan-out by subscription -> per-endpoint delivery queues (isolation) -> workers with retry and backoff. Always mention HMAC signature verification (timestamp-based to prevent replay), idempotency via event IDs, and the auto-disable policy for persistently failing endpoints. The advanced insight: webhooks should be fire-and-forget from the main request path - never slow down the primary flow to deliver a webhook. Use an async pipeline.

<

/blockquote>

Key Takeaway

Webhooks push events to clients instantly, eliminating polling. Include HMAC signature verification with timestamp validation, retry with exponential backoff (up to 3 days), idempotency via unique event IDs, and event type filtering. Provide delivery logs, testing tools, and auto-disable for failing endpoints.