The Outbox Pattern: Guaranteed Message Delivery Without Dual Writes

The Lost Order Notification

A customer places an order. Your Order Service saves it to PostgreSQL (success!) then tries to publish an OrderCreated event to Kafka so the notification service can send a confirmation email. But Kafka is temporarily unreachable. The order exists in the database, but no event was published. No email. No inventory update. No payment processing. The customer refreshes the page, sees their order, but never gets a confirmation. This is the dual write problem.

The Dual Write Problem Visualized

The Dual Write Problem:

  [Order Service]
      |
  1.  +--INSERT order--> [PostgreSQL]  SUCCESS
      |
  2.  +--publish event-> [Kafka]       FAIL (Kafka down!)
      |
  Result: order in DB, no event published.
  Inventory not reserved. Email not sent. Payment not started.

  What if we reverse the order?
  1. Publish to Kafka first -> Kafka SUCCESS
  2. Insert to DB -> DB FAIL (connection error!)
  Result: event published, but no order exists.
  Downstream services process a ghost order.

  No matter the order: a crash between step 1 and 2
  leaves the system in an inconsistent state.
  Two systems. No shared transaction. No way to be atomic.

The Outbox Pattern Solution

The Outbox Pattern:

  [Order Service]
      |
  BEGIN TRANSACTION (single database!)
  |  INSERT INTO orders (id, user_id, total, status)
  |  VALUES ('ord-123', 42, 150.00, 'PENDING');
  |
  |  INSERT INTO outbox (id, event_type, payload, status)
  |  VALUES ('evt-456', 'OrderCreated',
  |          '{"order_id": "ord-123", ...}', 'PENDING');
  COMMIT TRANSACTION
      |
  Both writes are ATOMIC. Same database. Same transaction.
  If either fails, both roll back.

  Later (async):
  [Message Relay] reads outbox table
      |
  1.  SELECT * FROM outbox WHERE status = 'PENDING'
  2.  Publish each event to Kafka
  3.  UPDATE outbox SET status = 'SENT' WHERE id = 'evt-456'
      |
  Event delivery is EVENTUAL but GUARANTEED.

The Message Relay: Polling vs CDC

Two approaches to relay events from outbox to Kafka:

POLLING (simple):
  Every 1 second:
    SELECT * FROM outbox WHERE status = 'PENDING' LIMIT 100
    Publish each to Kafka
    UPDATE status = 'SENT'

  Pros: simple to implement
  Cons: polling load on DB, 1s latency minimum

CDC - Change Data Capture (production-grade):
  [PostgreSQL WAL] --Debezium--> [Kafka]

  Debezium reads the database transaction log (WAL)
  Every INSERT to the outbox table is captured
  Published to Kafka in near real-time (<100ms)
  Zero polling. Zero DB overhead.

  CDC vs Polling:
  Feature        Polling       CDC (Debezium)
  ------------   ----------    ---------------
  Latency        ~1 second     ~50-100ms
  DB load        Moderate      Zero
  Complexity     Low           Medium
  Infrastructure Just code     Debezium + Kafka Connect
  Production     Small scale   Large scale

Handling Duplicates

The outbox guarantees AT-LEAST-ONCE delivery.
Duplicates can happen if relay crashes after
publishing but before marking 'SENT'.

Make consumers idempotent:

  # Consumer with idempotency:
  event = consume_from_kafka()

  # Check if already processed:
  if db.exists('processed_events', event.id):
      return  # already handled, skip

  # Process the event:
  process_order(event.payload)

  # Mark as processed:
  db.insert('processed_events', event.id)
  commit_kafka_offset()

  Same event processed twice? Second time is a no-op.

Interview Tip

When discussing event-driven microservices, say: 'To avoid the dual write problem of saving to a database and publishing to Kafka, I would use the outbox pattern. The service writes both business data and the event to the same database in a single transaction. A separate relay process reads pending events from the outbox table and publishes them to Kafka. For production, I would use Debezium CDC to capture outbox inserts from the database WAL with near-zero latency and no polling overhead. Since the relay provides at-least-once delivery, consumers must be idempotent - I would use the event ID as a dedup key.'

<

/blockquote>

Key Takeaway

The outbox pattern atomically pairs business data writes with event publishing by writing events to a local database table first. A relay process delivers the events asynchronously. This eliminates the dual write problem and guarantees eventual message delivery even when the message broker is temporarily unavailable.