Every busy website produces a river of small facts: this person viewed that page, that server logged this error, this order was placed. At LinkedIn the systems built to move such data around were struggling. Traditional message queues were designed for careful, one-at-a-time delivery of important jobs. Log streams are the opposite: huge, constant, and mostly fine to replay.
Kafka's authors made a bold simplification. Treat the stream as an append-only log on disk. Let consumers pull from it at their own speed, remembering their own position. That one idea made it fast enough for the firehose, and flexible enough that dozens of different systems could read the same stream.
Kafka stores each stream as a set of partitioned, append-only logs. Messages are found by offset, not by ID. Consumers pull data and keep their own place, so brokers stay simple and fast, and many independent consumer groups can read the same data.
The problem with queues for logs
An ordinary message broker keeps a bookkeeping record for every single message: who has it, whether it was acknowledged, whether to resend it. That is wonderful for a payment job and wasteful for a billion page views. The paper's own tests showed per-message overhead of 9 bytes in Kafka against 144 bytes in ActiveMQ.
Kafka throws most of the bookkeeping away. The broker stores messages in order and does not track who has read what. The reader is responsible for remembering how far it got.
A topic is a named stream, such as "page-views." A topic is split into partitions, and partitions are spread over several brokers (servers). A producer can choose the partition by a key (all events for one user go to one partition) or let it be random.
