The Real-Time Processing Toolkit
By 2014, the industry had Kafka for capturing event streams. What it lacked was a mature set of patterns for what to do with those streams. Three developments that year and the next defined the real-time processing playbook that most companies still use today: the Lambda Architecture for hybrid batch and stream processing, WebSockets for pushing data to browsers, and Apache Flink for production-grade stream processing with strong correctness guarantees.
Lambda Architecture (2014)
Nathan Marz, creator of Apache Storm at Twitter, proposed a pattern for systems that needed both real-time freshness AND historically accurate results. The Lambda Architecture runs two parallel pipelines. The batch layer processes the complete historical dataset periodically (a MapReduce job running nightly) for accuracy. The speed layer processes recent events in real-time (Storm processing events as they arrive) for freshness. A serving layer merges both views when answering queries.
Elegant in theory. A nightmare to operate. You maintain two separate codebases computing the same logical result, then reconcile their outputs. When business logic changes, you change it in two places. When bugs are found, they exist in two systems. This complexity eventually drove adoption of the Kappa Architecture, proposed by Kreps: just use the stream pipeline for everything, including reprocessing historical data. Lambda vs Kappa is a classic interview trade-off question: Lambda gives you accuracy plus freshness but doubles operational complexity. Kappa simplifies by treating everything as a stream but requires your streaming system to handle reprocessing.
WebSockets Go Mainstream (2014)
HTTP's request-response model works for loading pages but fails for real-time features. Chat messages, live scores, collaborative editing, real-time dashboards. All require the server to push updates without being polled. Long polling (client asks 'anything new?' every second) was a hack that wasted server connections and did not scale. WebSockets established a persistent bidirectional connection using a single TCP connection, with both sides able to send messages at any time.
The hard part is not opening a WebSocket. It is managing millions of them. Each connection holds stateful context (user identity, subscriptions, room membership). Multiple WebSocket servers need a way to route messages to connections on other servers. This is why chat systems and real-time collaboration tools always pair WebSockets with a pub/sub layer. A user sends a message, the WebSocket server publishes it to Redis Pub/Sub, and every other WebSocket server receives it and pushes to the relevant connected clients.
WebSockets solve the push problem. Redis Pub/Sub (or Kafka) solves the cross-server fan-out problem. You need both to build real-time systems that scale beyond a single server.
Apache Flink and Event-Time Processing (2015)
Apache Storm (Twitter, 2011) proved stream processing was possible. Spark Streaming (2013) made it easier with micro-batches. Flink was the first framework to handle event-time processing correctly with exactly-once semantics. The distinction matters enormously: processing-time means when the server received the event, event-time means when the event actually happened.
For analytics this distinction is critical. A ride-sharing app calculating surge pricing needs to know when rides were requested, not when the events arrived at the processing server. Mobile users with intermittent connectivity generate late-arriving events. A 9:05 PM event might arrive at 9:12 PM. Should it be counted in the 9:00 PM window? Flink's watermarks and windowing operators handle this systematically. Google's Dataflow paper (2015) independently reached similar conclusions, and Apache Beam unified the programming model across Flink, Spark, and Dataflow.
Further Reading
Nathan Marz (2012): 'How to beat the CAP theorem' where Lambda Architecture was introduced
Jay Kreps (2014): 'Questioning the Lambda Architecture' where Kappa Architecture was proposed
Akidau et al. (2015): 'The Dataflow Model' (Google), the theoretical foundation for event-time processing
Flink documentation on event time and watermarks
Window Types: How Stream Processors Group Continuous Data
Windowing converts an infinite stream into finite, processable chunks. The window type determines how those chunks are defined. Tumbling windows are fixed-size, non-overlapping: events from 9:00 to 9:05 go into one window, 9:05 to 9:10 into the next. Each event belongs to exactly one window. Sliding windows overlap: a 5-minute window sliding every 1 minute means each event belongs to 5 windows. More expensive but better for moving averages and trend detection. Session windows group events based on activity: a new window starts when activity resumes after an inactivity gap. Natural for user session analytics where session length varies.
Tumbling window: fixed size, no overlap. Hourly sales totals, daily active users. Each event in exactly one window.
Sliding window: fixed size, overlaps. 5-minute moving average computed every 1 minute. Each event in multiple windows.
Session window: dynamic size based on inactivity gap. User sessions, clickstream grouping. Window length varies.
Global window: one infinite window for the entire stream. Requires a custom trigger to emit results.
Watermark: the system's estimate of 'how late can events arrive?' Watermark advances past a timestamp to close the window for that timestamp.
State Management in Stream Processors: The Hard Part
Stateless stream processing (each event processed independently) is straightforward. Stateful processing (counts, aggregations, joins across events) requires maintaining state across millions of events. Flink manages state in 'keyed state': each key (user ID, order ID) has its own isolated state store. This state is partitioned across the cluster and backed by a distributed key-value store (RocksDB by default). Checkpoints periodically snapshot state to durable storage (HDFS, S3), enabling exactly-once recovery. If a task manager fails, Flink restarts from the last checkpoint and reprocesses events from the corresponding Kafka offset.
ValueState: single value per key. Running count, latest event timestamp.
ListState: list of values per key. Collecting events before a flush.
MapState: map per key. User profile accumulation.
RocksDB state backend: stores state on disk (not just memory). Enables large state (TBs) with slower access.
Savepoints vs checkpoints: checkpoints are automatic recovery points, savepoints are operator-triggered snapshots for upgrades and migrations.
What Interviewers Test About Stream Processing
Stream processing questions often start with 'design a real-time dashboard' or 'design a fraud detection system.' The interviewer is checking whether you know when to use stream processing versus batch, how you handle late-arriving events, and how you ensure exactly-once semantics end-to-end.
Know: the difference between processing time (when the server processed it) and event time (when it actually happened), and why event time is usually what you want
Know: how watermarks work and the trade-off between watermark aggressiveness and result latency
Know: why exactly-once end-to-end requires both the stream processor (Flink's checkpointing) AND the sink (idempotent writes or transactions)
Know: when Lambda Architecture's dual-pipeline complexity is worth it vs when Kappa (stream only) is simpler
Know: Kafka Streams for lightweight stream processing embedded in the application vs Flink for heavy stateful pipelines requiring dedicated cluster management
Window Types: How Stream Processors Group Continuous Data
Windowing converts an infinite stream into finite, processable chunks. The window type determines how those chunks are defined. Tumbling windows are fixed-size, non-overlapping: events from 9:00 to 9:05 go into one window, 9:05 to 9:10 into the next. Each event belongs to exactly one window. Sliding windows overlap: a 5-minute window sliding every 1 minute means each event belongs to 5 windows. More expensive but better for moving averages and trend detection. Session windows group events based on activity: a new window starts when activity resumes after an inactivity gap. Natural for user session analytics where session length varies.
Tumbling window: fixed size, no overlap. Hourly sales totals, daily active users. Each event in exactly one window.
Sliding window: fixed size, overlaps. 5-minute moving average computed every 1 minute. Each event in multiple windows.
Session window: dynamic size based on inactivity gap. User sessions, clickstream grouping. Window length varies.
Global window: one infinite window for the entire stream. Requires a custom trigger to emit results.
Watermark: the system's estimate of 'how late can events arrive?' Watermark advances past a timestamp to close the window for that timestamp.
State Management in Stream Processors: The Hard Part
Stateless stream processing (each event processed independently) is straightforward. Stateful processing (counts, aggregations, joins across events) requires maintaining state across millions of events. Flink manages state in 'keyed state': each key (user ID, order ID) has its own isolated state store. This state is partitioned across the cluster and backed by a distributed key-value store (RocksDB by default). Checkpoints periodically snapshot state to durable storage (HDFS, S3), enabling exactly-once recovery. If a task manager fails, Flink restarts from the last checkpoint and reprocesses events from the corresponding Kafka offset.
ValueState: single value per key. Running count, latest event timestamp.
ListState: list of values per key. Collecting events before a flush.
MapState: map per key. User profile accumulation.
RocksDB state backend: stores state on disk (not just memory). Enables large state (TBs) with slower access.
Savepoints vs checkpoints: checkpoints are automatic recovery points, savepoints are operator-triggered snapshots for upgrades and migrations.
What Interviewers Test About Stream Processing
Stream processing questions often start with 'design a real-time dashboard' or 'design a fraud detection system.' The interviewer is checking whether you know when to use stream processing versus batch, how you handle late-arriving events, and how you ensure exactly-once semantics end-to-end.
Know: the difference between processing time (when the server processed it) and event time (when it actually happened), and why event time is usually what you want
Know: how watermarks work and the trade-off between watermark aggressiveness and result latency
Know: why exactly-once end-to-end requires both the stream processor (Flink's checkpointing) AND the sink (idempotent writes or transactions)
Know: when Lambda Architecture's dual-pipeline complexity is worth it vs when Kappa (stream only) is simpler
Know: Kafka Streams for lightweight stream processing embedded in the application vs Flink for heavy stateful pipelines requiring dedicated cluster management
Related reading
Window Types: How Stream Processors Group Continuous Data
Windowing converts an infinite stream into finite, processable chunks. The window type determines how those chunks are defined. Tumbling windows are fixed-size, non-overlapping: events from 9:00 to 9:05 go into one window, 9:05 to 9:10 into the next. Each event belongs to exactly one window. Sliding windows overlap: a 5-minute window sliding every 1 minute means each event belongs to 5 windows. More expensive but better for moving averages and trend detection. Session windows group events based on activity: a new window starts when activity resumes after an inactivity gap. Natural for user session analytics where session length varies.
- Tumbling window: fixed size, no overlap. Hourly sales totals, daily active users. Each event in exactly one window.
- Sliding window: fixed size, overlaps. 5-minute moving average computed every 1 minute. Each event in multiple windows.
- Session window: dynamic size based on inactivity gap. User sessions, clickstream grouping. Window length varies.
- Global window: one infinite window for the entire stream. Requires a custom trigger to emit results.
- Watermark: the system's estimate of 'how late can events arrive?' Watermark advances past a timestamp to close the window for that timestamp.
State Management in Stream Processors: The Hard Part
Stateless stream processing (each event processed independently) is straightforward. Stateful processing (counts, aggregations, joins across events) requires maintaining state across millions of events. Flink manages state in 'keyed state': each key (user ID, order ID) has its own isolated state store. This state is partitioned across the cluster and backed by a distributed key-value store (RocksDB by default). Checkpoints periodically snapshot state to durable storage (HDFS, S3), enabling exactly-once recovery. If a task manager fails, Flink restarts from the last checkpoint and reprocesses events from the corresponding Kafka offset.
- ValueState: single value per key. Running count, latest event timestamp.
- ListState: list of values per key. Collecting events before a flush.
- MapState: map per key. User profile accumulation.
- RocksDB state backend: stores state on disk (not just memory). Enables large state (TBs) with slower access.
- Savepoints vs checkpoints: checkpoints are automatic recovery points, savepoints are operator-triggered snapshots for upgrades and migrations.
What Interviewers Test About Stream Processing
Stream processing questions often start with 'design a real-time dashboard' or 'design a fraud detection system.' The interviewer is checking whether you know when to use stream processing versus batch, how you handle late-arriving events, and how you ensure exactly-once semantics end-to-end.
- Know: the difference between processing time (when the server processed it) and event time (when it actually happened), and why event time is usually what you want
- Know: how watermarks work and the trade-off between watermark aggressiveness and result latency
- Know: why exactly-once end-to-end requires both the stream processor (Flink's checkpointing) AND the sink (idempotent writes or transactions)
- Know: when Lambda Architecture's dual-pipeline complexity is worth it vs when Kappa (stream only) is simpler
- Know: Kafka Streams for lightweight stream processing embedded in the application vs Flink for heavy stateful pipelines requiring dedicated cluster management
