How LinkedIn Processes 7 Trillion Messages Per Day
LinkedIn runs one of the largest Kafka deployments in the world - processing over 7 trillion messages per day across thousands of topics. When a user likes a post, that event fans out to the activity feed, notifications, analytics, search indexing, and anti-spam services - all through Kafka. The secret to handling this scale is not bigger servers but more partitions. LinkedIn splits high-volume topics into hundreds of partitions, each consumed by a separate consumer instance. A single topic with 120 partitions and 120 consumers processes 120 messages in parallel. Kafka is not fast because of any single operation - it is fast because it parallelizes everything.
Topics, Partitions, and Offsets
Kafka Topic Architecture:
Topic: 'order-events' (3 partitions)
Partition 0: [msg0] [msg1] [msg2] [msg3] [msg4] [msg5]
^offset 0 ^offset 5 (newest)
Partition 1: [msg0] [msg1] [msg2] [msg3]
^offset 0 ^offset 3 (newest)
Partition 2: [msg0] [msg1] [msg2] [msg3] [msg4]
^offset 0 ^offset 4 (newest)
Key Properties:
- Messages within a partition are STRICTLY ORDERED
- NO ordering guarantee ACROSS partitions
- Each message has a unique (partition, offset) address
- Messages are immutable once written (append-only log)
- Messages persist for retention period (7 days default)Partition Key Strategy
How Partition Keys Work:
Producer sends: key='user-42', value={event data}
Kafka: partition = hash('user-42') % num_partitions
= 2947826 % 3 = 1 -> Partition 1
ALL events for user-42 go to Partition 1 (always)
ALL events for user-73 might go to Partition 0
Choosing the Right Key:
Use Case Key Why
------------------- -------------- ---------------------------
Order processing order_id All events for one order stay ordered
User activity user_id User's events processed in sequence
IoT telemetry device_id Per-device ordering
Multi-tenant tenant_id Tenant isolation + ordering
Payment processing transaction_id All payment steps stay ordered
BAD keys:
- Timestamp: all messages go to same partition at same time (hot partition)
- Country: 'US' gets 60% of traffic (skew)
- null key: round-robin (no ordering guarantee)
Hot Partition Problem:
Partition 0: [|||||||||||||||||||||||] 80% of traffic (key='US')
Partition 1: [||||] 10%
Partition 2: [||||] 10%
Solution: composite key = country + user_id (spreads US traffic)Consumer Groups Deep Dive
Consumer Group Mechanics:
Topic: 'orders' (6 partitions)
Consumer Group A (Order Processing):
+------------+ +------------+ +------------+
| Consumer 1 | | Consumer 2 | | Consumer 3 |
| P0, P1 | | P2, P3 | | P4, P5 |
+------------+ +------------+ +------------+
Each partition assigned to EXACTLY ONE consumer.
Each consumer reads 2 partitions.
Consumer Group B (Analytics) - reads SAME messages independently:
+------------+ +------------+
| Consumer 1 | | Consumer 2 |
| P0, P1, P2 | | P3, P4, P5 |
+------------+ +------------+
Scaling scenarios:
6 partitions + 3 consumers = 2 partitions/consumer (balanced)
6 partitions + 6 consumers = 1 partition/consumer (max parallel)
6 partitions + 8 consumers = 2 consumers IDLE (wasted!)
^-- max useful consumers = num partitions
Rebalancing (when consumer dies):
Before: C1=[P0,P1] C2=[P2,P3] C3=[P4,P5]
C2 dies: C1=[P0,P1,P2] C3=[P3,P4,P5] (automatic redistribution)
C2 back: C1=[P0,P1] C2=[P2,P3] C3=[P4,P5] (rebalances again)Partition Sizing Guide
How Many Partitions?
Rule of Thumb:
partitions = max(target_throughput / per_consumer_throughput,
target_throughput / per_producer_throughput)
Example:
Target: 100K msgs/sec. Each consumer handles 10K msgs/sec.
Partitions needed: 100K / 10K = 10 partitions minimum
Practical Guidelines:
Throughput Partitions Notes
--------------- ---------- --------------------------
< 10K msgs/sec 3-6 Small topics, low overhead
10K-100K msgs/sec 6-30 Medium topics
100K-1M msgs/sec 30-120 High-volume topics
> 1M msgs/sec 120-500 LinkedIn/Uber scale
Tradeoffs of more partitions:
+ Higher parallelism and throughput
- More file handles on brokers (each partition = 2 files)
- Slower leader elections during broker failures
- Longer rebalancing time when consumers join/leave
- More memory for offset tracking
WARNING: You can ADD partitions later but CANNOT REMOVE them.
Start conservative, scale up as needed.Interview Tip
Kafka partitions are fundamental to any event-driven system design. When drawing Kafka in an interview, always specify partition count and partition key. Explain: partitions = max parallelism ceiling, partition key = ordering guarantee. Common trap: interviewer asks 'how do you guarantee ordering?' Answer: 'messages with the same key go to the same partition, and partitions provide strict ordering. We choose the partition key based on what needs ordering - user_id for user activity, order_id for order processing.' Mention rebalancing when consumers join/leave and explain that more consumers than partitions means wasted resources. The advanced insight: hot partitions from skewed keys (like country) are the #1 performance issue - solve with composite keys.
Key Takeaway
Partitions enable parallel processing and ordered delivery per key. Consumer groups distribute partitions among consumers for horizontal scaling. Your partition count sets the ceiling for parallelism - you cannot have more active consumers than partitions. Choose partition keys carefully to ensure related messages stay ordered and load is distributed evenly across partitions.