Kafka Streams and Apache Flink: Stream Processing at Massive Scale

Uber's 7 Trillion Events Per Day

Uber processes over 7 trillion events per day through Apache Flink. Every ride request, GPS update, surge price calculation, and fraud check is a stream processing job. Kafka delivers the events. But what turns 'ride started at location X' into 'there is surge pricing in downtown because demand spiked 3x in 5 minutes'? That is the stream processing layer.

Kafka Streams: Processing Inside Your App

Kafka Streams Architecture:

  +--Your Application (JVM)--+
  |                          |
  |  [Kafka Streams Library]  |  <- Not a separate cluster!
  |  |                    |  |     Just a library in your app.
  |  | [RocksDB]          |  |  <- Local state store
  |  | (state/aggregation) |  |
  |  +--------------------+  |
  +---------|------|----------+
            |      |
       read |      | write
            v      v
   [Kafka Topic A] [Kafka Topic B]
   (input events)  (output results)

Kafka Streams key capabilities:
  - Windowing:   Count events in 5-min time windows
  - KTable:      Materialized view (latest value per key)
  - Joins:       Join event stream with lookup table
  - Aggregation: Sum, count, average per key

When to use:
  - Simple transformations (filter, map, enrich)
  - Moderate state (fits on one machine)
  - You want zero extra infrastructure

Apache Flink: Heavy Lifting at Scale

Apache Flink Architecture:

  [JobManager]              <- Orchestrator
       |
  +----+----+----+
  |    |    |    |
  [TM] [TM] [TM] [TM]      <- TaskManagers (workers)
  |    |    |    |
  [State Backend: RocksDB + S3 checkpoints]

Flink excels at:

  1. Complex Event Processing (CEP):
     Detect: 'small test charge' followed by
     'large charge' within 2 minutes = FRAUD

  2. Massive State:
     Aggregate terabytes of state across cluster
     (Kafka Streams limited to single machine state)

  3. Exactly-Once Processing:
     Chandy-Lamport checkpoints to durable storage
     Recover without double-counting on failure

  4. Unified Batch + Stream:
     Same code processes historical and live data

The Event Time vs Processing Time Problem

The Late-Arriving Event Problem:

  Timeline (processing time):
  |--14:00--|--14:01--|--14:02--|--14:03--|

  Events arriving:
  14:00:01  Event A (event_time: 14:00:01)  -> on time
  14:00:05  Event B (event_time: 14:00:05)  -> on time
  14:02:30  Event C (event_time: 13:58:00)  -> LATE!
            (mobile app was offline, sent events late)

  If computing 'messages per minute' for minute 13:58:
  - Processing time: Event C counted in 14:02 (WRONG)
  - Event time:      Event C counted in 13:58 (CORRECT)

  Watermarks solve this:
  'We believe all events before T have arrived'
  Events arriving after the watermark are handled
  by allowed-lateness policies (process or discard)

Kafka Streams vs Flink Decision Guide

Factor              Kafka Streams      Apache Flink
------------------  -----------------  -------------------
Deployment          Library in app     Separate cluster
State size          Single machine     Distributed (TB+)
Exactly-once        Within Kafka       End-to-end
Event time support  Basic watermarks   Advanced (CEP)
Operational cost    Zero (it is a jar) High (cluster mgmt)
Language            Java/Kotlin        Java/Scala/Python
SQL support         No                 Flink SQL
Learning curve      Moderate           Steep

Use Kafka Streams: < 10 streaming jobs, simple logic
Use Flink:         Complex CEP, massive state, SQL queries

Interview Tip

When designing a real-time processing system, say: 'For stream processing, I would choose based on complexity. For simple event transformations like enrichment and filtering, Kafka Streams is perfect - it is just a library, no cluster to manage, and state is stored locally in RocksDB. For complex requirements like fraud detection patterns, terabyte-scale aggregations, or exactly-once guarantees end-to-end, I would use Apache Flink with its checkpoint-based recovery and CEP support. Both handle event-time processing with watermarks to handle late-arriving data correctly.'

<

/blockquote>

Key Takeaway

Kafka Streams is great for straightforward stream processing embedded in your application with no extra infrastructure. Flink is the choice for complex, large-scale pipelines with demanding state management. Both solve the fundamental problem of turning a stream of raw events into meaningful, real-time results.