Every Data Problem Is a Log Problem
In 2013, Jay Kreps wrote a blog post called 'The Log: What every software engineer should know about real-time data's unifying abstraction.' It explained something he had discovered while building LinkedIn's data infrastructure: at its core, almost every data problem at scale is a problem about logs. Not log files for debugging. An append-only, ordered sequence of records with a position offset.
Databases use logs (the write-ahead log ensures durability). Replication uses logs (the binlog streams changes to replicas). Event sourcing uses logs (every state change is an event appended to a sequence). If you put a durable, distributed log at the center of your architecture, every system can consume events at its own pace. A slow consumer does not block a fast one. A new system can replay from position zero to rebuild its state. The log becomes the single source of truth for all downstream systems.
The append-only log is the most underrated data structure in system design. It solves replication, event sourcing, audit trails, and inter-service communication with a single abstraction.
Apache Kafka (2011)
LinkedIn needed to stream hundreds of billions of events per day between dozens of internal systems. Traditional message queues (RabbitMQ, ActiveMQ) were designed for point-to-point messaging, not streaming at planetary scale. They deleted messages after delivery. They could not handle multiple independent consumers reading the same stream. And at LinkedIn's write rates, they became bottlenecks.
Kreps, Narkhede, and Rao built Kafka around one insight: treat messages like a distributed commit log. Producers append to partitioned topics. Consumers read at their own offset. Messages persist on disk and replicate for durability. Unlike traditional queues where messages die after consumption, Kafka retains messages by time or size. Multiple consumer groups can independently process the same stream from different offsets.
How Kafka Works
A Kafka topic is divided into partitions. Each partition is an ordered, immutable sequence of messages stored on disk. Producers write to partitions in parallel (keys can be used to route related messages to the same partition for ordering guarantees). Consumers in a consumer group divide up the partitions among themselves, each reading from assigned partitions independently.
The disk persistence is counterintuitive. Disks feel slow, but sequential writes to disk are extremely fast, often faster than random writes to memory. Kafka writes are sequential appends. Reads are sequential scans from an offset. Operating system page cache handles the hot data in memory automatically. A single Kafka broker can handle millions of messages per second on commodity hardware.
- Topics: logical channels that producers write to and consumers read from
- Partitions: each topic splits into N partitions for parallel reads and writes. More partitions enable more throughput.
- Consumer groups: multiple consumers in a group split partitions between them. Different groups read independently at their own pace.
- Offsets: each message in a partition has a sequential offset. Consumers track their own offset. Replay by resetting to offset 0.
- Retention: messages stay on disk until their retention period (time or bytes) expires. Not deleted on consumption.
Kafka vs Traditional Message Queues
The key difference is the consumption model. Traditional queues (RabbitMQ, SQS) are about delivering messages to a single consumer and deleting them. Work queues, job processing. Kafka is about streaming an ordered log of events to multiple independent consumers who each maintain their own position. Event streaming, CDC (change data capture), real-time analytics.
In practice: use a traditional queue when you want task distribution (process this job once, by any available worker). Use Kafka when you want event streaming (every service that cares about this event should see it, at their own pace). The news feed fanout, the audit log, the real-time dashboard, and the search index update can all consume the same Kafka topic independently. With a traditional queue, you would need a separate queue for each consumer.
Further Reading
- Jay Kreps (2013): 'The Log: What every software engineer should know about real-time data's unifying abstraction' (engineering.linkedin.com)
- Kreps, Narkhede & Rao (2011): 'Kafka: a Distributed Messaging System for Log Processing'
- Kafka documentation: kafka.apache.org, particularly the design section
- Confluent's 'Kafka: The Definitive Guide', free online
Kafka Delivery Guarantees: Choosing Your Consistency Level
Kafka exposes three delivery guarantee levels for producers, and the right choice depends entirely on your workload. At-most-once: producers fire and forget. Messages may be lost on broker failure. Throughput is maximized. Acceptable for non-critical metrics where occasional loss is tolerable. At-least-once: producers retry on failure using acknowledgment. Messages may be delivered more than once on retry. Consumers must be idempotent (processing the same message twice produces the same result). This is the default mode and the right choice for most event-driven systems. Exactly-once: Kafka's transactional API ensures each message is delivered and processed exactly once, even across producer failures and consumer group rebalances. Highest overhead, required for financial event processing.
- acks=0: fire and forget. No confirmation from broker. Highest throughput, possible message loss.
- acks=1: leader confirms receipt. Message lost if leader fails before replicating to followers.
- acks=all (acks=-1): all in-sync replicas confirm. Durability guaranteed. Lowest throughput.
- Idempotent producer: enabled with enable.idempotence=true. Assigns sequence numbers to deduplicate retries at the broker.
- Transactions: producers can atomically write to multiple partitions and topics. Consumers read committed messages only with isolation.level=read_committed.
Partition Key Design: The Most Impactful Kafka Decision You Will Make
Kafka guarantees ordering only within a partition. If you need all events for a user to be processed in order, all user events must go to the same partition. The partition key (usually the message key hash mod partition count) controls this. Using user_id as the key ensures all events for a user land in the same partition, giving you per-user ordering. Using a random or null key distributes load evenly but provides no ordering guarantees. Getting partition key design wrong creates either ordering bugs (random keys) or partition hotspots (keys with high cardinality imbalance like a few power users generating 80% of events).
- High-cardinality keys (user_id, order_id): even distribution, per-entity ordering. Ideal for most use cases.
- Low-cardinality keys (country, category): hotspot risk. A popular country saturates one partition.
- Null keys: round-robin distribution. Maximum throughput, zero ordering guarantees.
- Sticky partitioner (Kafka 2.4+): batches messages with null keys to the same partition temporarily, improving batch compression.
- Partition count is immutable after creation (changing it breaks key-to-partition mapping). Size partitions for peak throughput at creation time.
What Interviewers Test About Kafka
Kafka questions in senior interviews probe whether you understand the operational realities of running a distributed log at scale. Knowing that Kafka 'is fast' is not enough. Interviewers want to hear about consumer lag monitoring, replication factors, partition count planning, and the operational differences between Kafka and traditional message queues.
- Know: why Kafka consumers in a group cannot exceed the partition count (one consumer per partition maximum)
- Know: what consumer lag is, how to monitor it, and how to respond to it (scale consumers, optimize processing)
- Know: the role of Zookeeper in Kafka (controller election, topic metadata) vs KRaft (Kafka's newer built-in consensus)
- Know: log compaction vs log retention: compaction keeps the latest value per key (for state rebuilding), retention drops old messages by time or size
- Know: the difference between Kafka Streams (library for stateful stream processing on Kafka) and Kafka Connect (source/sink connectors for external systems)
Kafka Delivery Guarantees: Choosing Your Consistency Level
Kafka exposes three delivery guarantee levels for producers, and the right choice depends entirely on your workload. At-most-once: producers fire and forget. Messages may be lost on broker failure. Throughput is maximized. Acceptable for non-critical metrics where occasional loss is tolerable. At-least-once: producers retry on failure using acknowledgment. Messages may be delivered more than once on retry. Consumers must be idempotent (processing the same message twice produces the same result). This is the default mode and the right choice for most event-driven systems. Exactly-once: Kafka's transactional API ensures each message is delivered and processed exactly once, even across producer failures and consumer group rebalances. Highest overhead, required for financial event processing.
- acks=0: fire and forget. No confirmation from broker. Highest throughput, possible message loss.
- acks=1: leader confirms receipt. Message lost if leader fails before replicating to followers.
- acks=all (acks=-1): all in-sync replicas confirm. Durability guaranteed. Lowest throughput.
- Idempotent producer: enabled with enable.idempotence=true. Assigns sequence numbers to deduplicate retries at the broker.
- Transactions: producers can atomically write to multiple partitions and topics. Consumers read committed messages only with isolation.level=read_committed.
Partition Key Design: The Most Impactful Kafka Decision You Will Make
Kafka guarantees ordering only within a partition. If you need all events for a user to be processed in order, all user events must go to the same partition. The partition key (usually the message key hash mod partition count) controls this. Using user_id as the key ensures all events for a user land in the same partition, giving you per-user ordering. Using a random or null key distributes load evenly but provides no ordering guarantees. Getting partition key design wrong creates either ordering bugs (random keys) or partition hotspots (keys with high cardinality imbalance like a few power users generating 80% of events).
- High-cardinality keys (user_id, order_id): even distribution, per-entity ordering. Ideal for most use cases.
- Low-cardinality keys (country, category): hotspot risk. A popular country saturates one partition.
- Null keys: round-robin distribution. Maximum throughput, zero ordering guarantees.
- Sticky partitioner (Kafka 2.4+): batches messages with null keys to the same partition temporarily, improving batch compression.
- Partition count is immutable after creation (changing it breaks key-to-partition mapping). Size partitions for peak throughput at creation time.
What Interviewers Test About Kafka
Kafka questions in senior interviews probe whether you understand the operational realities of running a distributed log at scale. Knowing that Kafka 'is fast' is not enough. Interviewers want to hear about consumer lag monitoring, replication factors, partition count planning, and the operational differences between Kafka and traditional message queues.
- Know: why Kafka consumers in a group cannot exceed the partition count (one consumer per partition maximum)
- Know: what consumer lag is, how to monitor it, and how to respond to it (scale consumers, optimize processing)
- Know: the role of Zookeeper in Kafka (controller election, topic metadata) vs KRaft (Kafka's newer built-in consensus)
- Know: log compaction vs log retention: compaction keeps the latest value per key (for state rebuilding), retention drops old messages by time or size
- Know: the difference between Kafka Streams (library for stateful stream processing on Kafka) and Kafka Connect (source/sink connectors for external systems)