Change Data Capture: Streaming Database Changes in Real-Time

How Airbnb Keeps 10 Systems in Sync Without Dual Writes

When a host updates their listing on Airbnb, that change needs to propagate to search (Elasticsearch), pricing (ML models), availability calendars, analytics, mobile push notifications, and more. Airbnb tried dual writes at first - write to PostgreSQL AND publish to Kafka in the same request. The problem: when Kafka was slow, the API slowed down. When Kafka was down, either the write failed (bad UX) or data got out of sync (worse). Their solution: Change Data Capture (CDC). Writes go ONLY to PostgreSQL. Debezium reads PostgreSQL's WAL (Write-Ahead Log) and streams every change to Kafka. Downstream systems consume from Kafka. Zero application code changes, zero dual-write risks, sub-100ms latency.

Why Dual Writes Fail

The Dual Write Problem:

  Application code:
    db.save(listing)           # Step 1: Write to database
    kafka.publish(listing)     # Step 2: Publish to Kafka

  Failure scenario 1: Kafka publish fails
    Database has the update. Kafka does not. Search shows stale data.

  Failure scenario 2: App crashes between steps
    Database has the update. Kafka never gets it. Silent data loss.

  Failure scenario 3: Database write fails after Kafka publish
    Kafka has an event for data that does NOT exist in the database!

  None of these are atomic. You cannot make two systems
  transactionally consistent without distributed transactions
  (which are slow and complex).

CDC Solution:

  Application:      db.save(listing)     # Only one write!
  Debezium (async):  WAL -> Kafka        # Captures the change
  Downstream:        Kafka -> Elasticsearch, Analytics, etc.

  The database write IS the event. No dual write. No inconsistency.

How CDC Works Under the Hood

CDC Architecture:

  +--Application--+
  | INSERT INTO   |     (normal write)
  | listings ...  |
  +-------+-------+
          |
          v
  +--PostgreSQL---+
  | Table: listings|     1. Row written to table
  | WAL: --------|-----> 2. Change recorded in Write-Ahead Log
  +-------+-------+      (PostgreSQL writes WAL for crash recovery
          |               - CDC just reads what's already there!)
          v
  +--Debezium-----+
  | Reads WAL     |      3. Debezium connects as a replication slot
  | Transforms    |      4. Converts WAL entries to structured events
  | Publishes     |      5. Publishes to Kafka topics
  +-------+-------+
          |
          v
  +--Kafka--------+      6. Topic: 'db.public.listings'
  | Change Events |      Message: {
  |               |        "op": "u",  // u=update, c=create, d=delete
  |               |        "before": { "price": 100, ... },
  |               |        "after":  { "price": 120, ... },
  |               |        "ts_ms":  1705312800000
  +-------+-------+      }
          |
     +----+-----+-----+
     |          |     |
     v          v     v
  Elastic   Analytics  Cache
  Search    Pipeline   Invalidator

CDC Use Cases

Where CDC Shines:

  Use Case                  Without CDC                 With CDC
  ----------------------    ----------------------      -----------------------
  Search sync               Dual write (unreliable)     Auto-sync via WAL
  Cache invalidation        TTL-based (stale data)      Instant invalidation
  Data warehouse ETL        Nightly batch (24h delay)   Near-real-time (<1 min)
  Microservice events       Outbox pattern (complex)    Zero code changes
  Audit logging             Application-level (gaps)    Complete WAL capture
  Cross-DC replication      Logical replication         CDC + Kafka (flexible)

  CDC Tools Comparison:
  Tool              Database Support            Deployment
  ----------------  -------------------------   ------------------
  Debezium          PG, MySQL, Mongo, Oracle    Open source, Kafka Connect
  AWS DMS           All AWS databases           Managed service
  Fivetran          150+ connectors             SaaS ($$$$)
  pg_logical        PostgreSQL only             Built-in
  Maxwell           MySQL only                  Open source
  Striim            Enterprise databases        Enterprise ($$$$)

CDC Operational Considerations

• Replication slots: PostgreSQL WAL retention grows until Debezium reads it. If Debezium is down for hours, WAL fills the disk. Monitor replication slot lag.
• Schema evolution: when you ALTER TABLE, Debezium must handle the schema change. Use a schema registry (Confluent Schema Registry) to manage versions.
• Initial snapshot: when CDC starts on an existing table, it takes a consistent snapshot of current data, then streams changes from that point. For large tables, this can take hours.
• Ordering: CDC guarantees per-row ordering (changes to row X arrive in order) but not cross-row ordering.
• Performance: CDC reads the WAL that PostgreSQL already writes. Zero additional load on the database. No triggers, no polling.

Interview Tip

CDC appears whenever you need to keep multiple data stores in sync. The key phrase: 'capture changes from the database transaction log instead of dual writes.' Always mention why dual writes fail (non-atomic, inconsistency on partial failure). Draw the architecture: PostgreSQL WAL -> Debezium -> Kafka -> downstream consumers. Mention that CDC has zero database performance impact because it reads the WAL that PostgreSQL already writes for crash recovery. The advanced insight: CDC replaces the Outbox pattern in many cases - the transaction log IS the outbox, and Debezium reads it directly.

<

/blockquote>

Key Takeaway

CDC captures database changes from the transaction log and streams them to downstream systems in real-time. It eliminates dual-write reliability issues, has zero database performance impact, and enables event-driven architectures without application code changes. Debezium + Kafka is the standard open-source CDC stack.