Worked Example: Designing an Ad Click Aggregation System (like Meta's)
You'll learn to
- -Apply the design loop to a real-time aggregation pipeline where correctness has direct financial consequences
- -Understand windowed stream aggregation, watermarking, and why "exactly-once" matters here specifically
Most of the systems this course has designed so far can tolerate a little imprecision: a slightly stale feed, a slightly delayed message. This one can't. Every click event this system counts eventually turns into a line on an advertiser's bill, which changes which trade-offs are acceptable and which aren't.
Step 1: Clarify Requirements
- -Functional: capture every ad click, deduplicate it, attribute it to the correct advertiser campaign, and aggregate counts into real-time metrics that feed both dashboards and billing.
- -Out of scope: bot/fraud-click filtering and ad targeting itself. Assume every click reaching the system is a legitimate impression worth counting.
- -Non-functional: billing numbers must be correct, full stop; a small amount of dashboard staleness (numbers a minute or two behind) is completely acceptable, but a miscounted click is not.
Step 2: Estimate Scale
Assume 1 million click events per second across 100,000 active advertisers, aggregated into 1-minute tumbling windows.
That last row is the entire design constraint. A system that processes 1M clicks/sec but occasionally double-counts or drops one isn't "mostly correct" for a billing pipeline, it's wrong in a way that either overcharges a customer or gives away revenue. Speed matters, but not more than that.
Step 3: High-Level Design
Click beacons are accepted quickly and published to a queue partitioned by advertiser ID, so per-advertiser event order is preserved downstream.
The stream processor aggregates each window and fans results out to whichever store fits that consumer's need.
Step 4: Deep Dive: Windows, Watermarks, and Exactly-Once
A tumbling window is a fixed, non-overlapping time bucket (00:00:00-00:01:00, then 00:01:00-00:02:00, and so on); every event belongs to exactly one window based on its timestamp. The problem: a click from a phone on a slow network might not arrive until after its window has technically closed. A watermark tells the processor to wait a configurable extra period past each window boundary before finalizing it, giving late data a real chance to be counted, at the cost of a small, bounded delay before that window's numbers are final.
- -Deduplication: every click carries a unique impression ID. A Bloom filter gives a fast, cheap first-pass check for "have I possibly seen this before," backed by an authoritative store for the final answer, catching both accidental double-clicks and retried network requests replaying the same beacon.
- -"Exactly-once" in practice: truly guaranteeing a message is delivered exactly one time is generally not achievable in a distributed system. What's actually built is idempotent processing keyed by impression ID, so processing the same click event twice produces the same end state as processing it once, which is what "exactly-once semantics" means in real systems.
The Worker in this design runs periodic batch reconciliation: cross-checking the streaming aggregates against the billing database and correcting any drift. Pairing a fast streaming path with a slower, ground-truth batch pass is the Lambda architecture pattern named in this level's objectives, and it exists specifically because "fast" and "certainly correct" are hard to guarantee simultaneously in one path.
Step 5: Trade-offs & Failure Modes
- -Freshness vs. completeness: a short watermark wait produces fresher-looking dashboards but risks finalizing a window before some legitimately late clicks arrive; a longer wait is more complete but delays when a window's numbers are "done."
- -Malformed events: routed to a dead letter queue rather than blocking the stream processor, so one unparseable event can't stall the entire pipeline. A human or batch job reviews the dead letter queue later.
- -Reconciliation as a safety net: the batch reconciler assumes the real-time path, however carefully built, can still drift under load or partial failure, and treats the billing database as the source of truth that gets corrected against, not the stream.
Since this system is ultimately just counting clicks, why not accept at-least-once delivery and treat an occasional double-count as noise?
"At-least-once is simpler to build, and a rare double-count probably doesn't matter much."
"It would matter here specifically because these counts drive advertiser billing, not just an internal metric. A double-counted click overcharges a customer; a dropped one gives away revenue the platform is owed. That's a fundamentally different cost than, say, a public view counter being off by a few, which is why this design spends real complexity on idempotent, impression-ID-keyed processing instead of accepting at-least-once as good enough."
What is a tumbling window in stream processing?
Level 20: Ad Click Aggregation (Meta) in the HLD Lab is exactly this system. Go build the design you just walked through.