Real-Time Analytics Pipelines: From Raw Events to Live Dashboards

Twitter's 500 Million Tweets Per Day

Twitter (X) shows trending topics that update every few minutes. Behind this is a pipeline processing 500 million tweets per day, computing hashtag frequencies, applying geographic filters, and comparing against historical baselines - all with single-digit-second latency. The difference between batch analytics ('what happened yesterday') and real-time analytics ('what is happening NOW') is the difference between a newspaper and a live news ticker.

Lambda vs Kappa Architecture

Lambda Architecture (two paths):

  [Events] ---> [Kafka] ---+---> [Speed Layer (Flink)]
                           |     (approximate, seconds)
                           |           |
                           |     [Serving Layer]-->[Dashboard]
                           |           |
                           +---> [Batch Layer (Spark)]
                                 (exact, hours)

  Problem: TWO codebases doing the same computation.
  Maintaining both is painful.

Kappa Architecture (one path):

  [Events] ---> [Kafka] ---> [Flink] ---> [OLAP DB] ---> [Dashboard]
                  |
                  | (replay from offset 0 for reprocessing)
                  +---> [Flink v2] ---> [New OLAP table]

  Everything is a stream. Reprocessing = replay the topic.
  ONE codebase. Simpler. Used by LinkedIn, Uber.

The End-to-End Pipeline

Complete Real-Time Analytics Pipeline:

  [Mobile/Web SDK]  [Backend Services]  [IoT Devices]
        |                  |                  |
        v                  v                  v
  [Kafka Ingestion Topics]
        |
        v
  [Flink Processing]
  - Sessionize user journeys
  - Enrich with user metadata
  - Filter bots and spam
  - Aggregate per time window
        |
        v
  [OLAP Database]
  (Druid / ClickHouse / Pinot)
        |
        v
  [Dashboard / Alerting]
  - Grafana dashboards (< 100ms query)
  - PagerDuty alerts on anomalies
  - A/B test metric computation

  End-to-end latency: event -> dashboard = 5-30 seconds

OLAP Databases: The Query Engine

OLAP Database Comparison:

Database     Ingestion    Query Speed   Best For
-----------  ----------   -----------   ----------------------
Druid        Real-time    ~100ms        Time-series, pre-agg
ClickHouse   Near-RT      ~200ms        Ad-hoc analytics, scans
Pinot        Real-time    ~50ms         User-facing analytics
BigQuery     Batch/stream ~seconds      Data warehouse queries
Redshift     Batch        ~seconds      BI/reporting

Why not PostgreSQL?
  PostgreSQL: row-oriented, scans ALL columns per row
  ClickHouse: column-oriented, reads ONLY queried columns

  Query: 'SELECT avg(latency) FROM events WHERE date > X'
  PostgreSQL scans: all 50 columns of every row = slow
  ClickHouse scans: just the latency and date columns = fast
  10 billion rows in < 1 second (ClickHouse)

Cost vs Latency Trade-offs

Analytics Latency vs Cost Spectrum:

  Approach         Latency      Monthly Cost (10TB/day)
  ---------------  ----------   ----------------------
  Batch (Spark)    Hours        $2,000-5,000
  Micro-batch      Minutes      $5,000-15,000
  True streaming   Seconds      $15,000-50,000
  Sub-second       < 1 second   $50,000+

  Rule of thumb: each order-of-magnitude latency
  reduction roughly doubles infrastructure cost.
  Only pay for the latency you actually NEED.

  Fraud detection:    needs sub-second (worth the cost)
  Trending topics:    minutes is fine
  Revenue dashboard:  hourly refresh is acceptable

Interview Tip

When designing an analytics system, say: 'I would use a Kappa architecture with Kafka for event ingestion, Flink for stream processing (sessionization, enrichment, aggregation), and ClickHouse or Apache Druid for the query layer. Events flow from producers to Kafka topics, Flink reads and transforms them, then writes to the OLAP database. Dashboards query the OLAP DB with sub-100ms latency. For reprocessing, I replay the Kafka topic through an updated Flink job. I would choose latency requirements carefully since each order of magnitude improvement roughly doubles cost.'

<

/blockquote>

Key Takeaway

Real-time analytics pipelines combine stream processing (Kafka/Flink) with columnar OLAP databases (Druid/ClickHouse). Kappa architecture keeps it simple by treating everything as a replayable stream. The key engineering challenge is balancing latency, cost, and query complexity.