Skip to content
Kafka: a Distributed Messaging System for Log Processing 4 min

Jay Kreps, Neha Narkhede, Jun Rao · NetDB 2011

Kafka: a Distributed Messaging System for Log Processing

Why Kafka is an append-only log, not a queue: partitions, offsets, consumer groups and replay, explained with a playground you can break.

Read the original paper

Systems & Distributed · Yellow Belt · Pro · 4 min read

Kafka: a Distributed Messaging System for Log Processing

Every busy website produces a river of small facts: this person viewed that page, that server logged this error, this order was placed. At LinkedIn the systems built to move such data around were struggling. Traditional message queues were designed for careful, one-at-a-time delivery of important jobs. Log streams are the opposite: huge, constant, and mostly fine to replay.

Kafka's authors made a bold simplification. Treat the stream as an append-only log on disk. Let consumers pull from it at their own speed, remembering their own position. That one idea made it fast enough for the firehose, and flexible enough that dozens of different systems could read the same stream.

Kafka stores each stream as a set of partitioned, append-only logs. Messages are found by offset, not by ID. Consumers pull data and keep their own place, so brokers stay simple and fast, and many independent consumer groups can read the same data.

The problem with queues for logs

An ordinary message broker keeps a bookkeeping record for every single message: who has it, whether it was acknowledged, whether to resend it. That is wonderful for a payment job and wasteful for a billion page views. The paper's own tests showed per-message overhead of 9 bytes in Kafka against 144 bytes in ActiveMQ.

Kafka throws most of the bookkeeping away. The broker stores messages in order and does not track who has read what. The reader is responsible for remembering how far it got.

Producers append messages to partitions spread over several brokers. Consumers in a group pull messages at their own pace. ZooKeeper keeps track of who is alive and who reads what.

A topic is a named stream, such as "page-views." A topic is split into partitions, and partitions are spread over several brokers (servers). A producer can choose the partition by a key (all events for one user go to one partition) or let it be random.

Keep reading with Pro

This paper is part of Pro. Pro and Architect open the full breakdown, with animated diagrams, interactive elements you can play with and a quiz. A few papers stay free to read in full.

Still to read

3 more sections, 1 more diagram and 2 interactive elements to play with

  • A log is simple, and simple is fast
  • Consumer groups: sharing the work, and replaying
  • What Kafka does and does not promise

Found this breakdown useful?

Share it with someone else wrestling with this paper.

Discussion0

Join the Discussion

Sign in to leave comments, reply to others, or like insights.

Sign In to ScaleDojo

No comments yet. Be the first to start the thread!

Enjoyed this? Get more like it.

New paper breakdowns, levels, and one concept worth knowing, straight to your inbox.

No spam, ever. Unsubscribe in one click.

Enjoyed this content?

Your support keeps us creating free resources

We put a lot of hours into researching and writing these guides. If it helped you, consider buying us a coffee. Every bit goes toward keeping ScaleDojo's content free and growing.

$

One-time payment via Stripe. ScaleDojo account required.