The Limits of Request-Response
REST APIs are synchronous: the client makes a request and blocks until the server responds. This model breaks down when processing takes longer than a client timeout, when multiple downstream services must coordinate, when you need to replay or audit historical events, or when your system must handle traffic spikes without dropping requests. Asynchronous messaging solves each of these.
AMQP: A Standard for Message Queuing
AMQP (Advanced Message Queuing Protocol) was created in 2003 by JPMorgan Chase to standardize financial messaging. Before AMQP, every enterprise message broker (MQSeries, TIBCO, WebLogic) used proprietary protocols that locked you into a vendor. AMQP defined a wire-level protocol that any broker could implement and any client could speak.
AMQP introduced concepts that define message broker APIs: exchanges (where messages are published), queues (where messages are stored for consumers), bindings (routing rules connecting exchanges to queues), acknowledgments (consumers confirm processing before the broker removes the message), and dead letter queues (where messages go after repeated failure). RabbitMQ, the most popular AMQP broker, still uses these concepts as its primary API surface.
Apache Kafka: The Distributed Commit Log
Jay Kreps, Neha Narkhede, and Jun Rao built Kafka at LinkedIn in 2010 and open-sourced it in 2011. Kafka is not a message queue - it is a distributed commit log. Messages are appended to ordered, partitioned logs (topics). Consumers read from a position in the log (offset) and maintain their own progress. Messages are retained by default for 7 days (configurable to indefinitely). Multiple consumers can independently read the same message.
The commit log model changes the API contract fundamentally. Traditional message queues destroy messages after delivery. Kafka persists them. This means you can replay any event, add a new consumer that processes all historical events, and recover from consumer bugs by resetting the offset. The log becomes the source of truth, not just a pipe.
Kafka's throughput numbers are deceptive in interviews. Yes, Kafka can handle millions of messages per second. But the right question is not 'how fast?' - it's 'do you need replay, multiple consumers, and persistent event history?' If the answer is yes, Kafka. If you just need reliable delivery from A to B, RabbitMQ with AMQP is simpler.
Event Sourcing: The Log as the Database
Event sourcing extends the Kafka model to application architecture. Instead of storing current state, you store every event that changed state. The current state is derived by replaying the event log. An account balance is not stored as $1,250 - it is computed by replaying every deposit and withdrawal event from account creation.
Event sourcing gives you a complete audit log for free, the ability to rebuild any derived state, and temporal queries (what was the state at time T?). The cost is that queries against current state require projections (read models built by consuming the event stream), and event schema evolution is harder than schema migrations.
Consumer Groups, Partitions, and Ordering Guarantees
Kafka's partition model determines both throughput and ordering guarantees in ways that are critical to understand before designing your event schema. Messages within a partition are strictly ordered. Messages across partitions are not. If ordering matters - say, all events for a specific order must be processed in sequence - you must ensure all events for that order go to the same partition. The standard pattern is to use the entity ID (order ID, user ID) as the partition key. All events for order-12345 hash to the same partition and are processed in sequence.
Consumer groups let multiple consumers process a topic in parallel. Each partition is assigned to exactly one consumer in a consumer group at a time. With 12 partitions and 12 consumers, all consumers process in parallel. With 12 partitions and 3 consumers, each consumer processes 4 partitions. Adding more consumers than partitions creates idle consumers - extra consumers get no partitions. Partition count determines your maximum parallelism. Choose partition count based on your throughput requirements and scale it up proactively; shrinking partitions requires repartitioning.
Event Sourcing Trade-offs in Practice
Event sourcing's benefits are substantial: complete audit trail, temporal queries, ability to replay events to fix bugs in projections, and natural integration with CQRS. The costs are also substantial. Snapshots become necessary as event streams grow (replaying 10 million events to get the current state of an account is impractical). Event schema migration is harder than column migration in a relational database: you need to either version your events or write migration code that transforms old events to new schemas on read.
The practical verdict: event sourcing is the right choice for domains where audit and temporal query requirements are genuine business requirements (financial transactions, compliance systems, collaborative editing). It is wrong for most CRUD applications where the current state is what matters and history is irrelevant. The complexity cost of event sourcing is significant. Do not introduce it without clear business requirements that justify it.
- Kafka partitions are the unit of parallelism, ordering, and availability. Design your partition key before designing your event schema.
- The at-least-once delivery guarantee means consumers must be idempotent. Use a deduplication key in each event and check it before processing.
- Kafka Streams and Apache Flink are stream processing frameworks built on Kafka. They enable stateful stream processing: windowed aggregations, joins between event streams, pattern detection.
- Avro schema registry (Confluent Schema Registry) enforces schema compatibility at publish time and enables schema evolution with full compatibility tracking.
- Lag monitoring is the most important Kafka operational metric: how far behind is the consumer from the latest partition offset? Sustained lag growth indicates a consumer that cannot keep up with the producer.
Exactly-Once Semantics: The Hardest Guarantee
Exactly-once delivery is the most desired and hardest to guarantee in distributed messaging. Kafka introduced exactly-once semantics (EOS) in version 0.11 (2017) for producer-to-broker delivery and idempotent consumers in Kafka Streams. The implementation requires a sequence number per producer-partition pair to deduplicate retries, and a two-phase commit protocol for consumer offsets and output records. The performance cost is real: enabling exactly-once semantics reduces throughput by approximately 20% compared to at-least-once delivery.
For most business applications, idempotent consumers with at-least-once delivery are the practical approach. Design each message consumer to be idempotent: processing the same message twice produces the same outcome. The idempotency key is typically the message's event ID. Store processed event IDs in a deduplication table (or use a Redis set with TTL). Check the ID before processing. This pattern is simpler than true exactly-once delivery, handles 99.9% of production duplicate scenarios, and has no throughput penalty.
Consumer Groups, Partitions, and Ordering Guarantees
Kafka's partition model determines both throughput and ordering guarantees in ways that are critical to understand before designing your event schema. Messages within a partition are strictly ordered. Messages across partitions are not. If ordering matters - say, all events for a specific order must be processed in sequence - you must ensure all events for that order go to the same partition. The standard pattern is to use the entity ID (order ID, user ID) as the partition key. All events for order-12345 hash to the same partition and are processed in sequence.
Consumer groups let multiple consumers process a topic in parallel. Each partition is assigned to exactly one consumer in a consumer group at a time. With 12 partitions and 12 consumers, all consumers process in parallel. With 12 partitions and 3 consumers, each consumer processes 4 partitions. Adding more consumers than partitions creates idle consumers - extra consumers get no partitions. Partition count determines your maximum parallelism. Choose partition count based on your throughput requirements and scale it up proactively; shrinking partitions requires repartitioning.
Event Sourcing Trade-offs in Practice
Event sourcing's benefits are substantial: complete audit trail, temporal queries, ability to replay events to fix bugs in projections, and natural integration with CQRS. The costs are also substantial. Snapshots become necessary as event streams grow (replaying 10 million events to get the current state of an account is impractical). Event schema migration is harder than column migration in a relational database: you need to either version your events or write migration code that transforms old events to new schemas on read.
The practical verdict: event sourcing is the right choice for domains where audit and temporal query requirements are genuine business requirements (financial transactions, compliance systems, collaborative editing). It is wrong for most CRUD applications where the current state is what matters and history is irrelevant. The complexity cost of event sourcing is significant. Do not introduce it without clear business requirements that justify it.
- Kafka partitions are the unit of parallelism, ordering, and availability. Design your partition key before designing your event schema.
- The at-least-once delivery guarantee means consumers must be idempotent. Use a deduplication key in each event and check it before processing.
- Kafka Streams and Apache Flink are stream processing frameworks built on Kafka. They enable stateful stream processing: windowed aggregations, joins between event streams, pattern detection.
- Avro schema registry (Confluent Schema Registry) enforces schema compatibility at publish time and enables schema evolution with full compatibility tracking.
- Lag monitoring is the most important Kafka operational metric: how far behind is the consumer from the latest partition offset? Sustained lag growth indicates a consumer that cannot keep up with the producer.
Exactly-Once Semantics: The Hardest Guarantee
Exactly-once delivery is the most desired and hardest to guarantee in distributed messaging. Kafka introduced exactly-once semantics (EOS) in version 0.11 (2017) for producer-to-broker delivery and idempotent consumers in Kafka Streams. The implementation requires a sequence number per producer-partition pair to deduplicate retries, and a two-phase commit protocol for consumer offsets and output records. The performance cost is real: enabling exactly-once semantics reduces throughput by approximately 20% compared to at-least-once delivery.
For most business applications, idempotent consumers with at-least-once delivery are the practical approach. Design each message consumer to be idempotent: processing the same message twice produces the same outcome. The idempotency key is typically the message's event ID. Store processed event IDs in a deduplication table (or use a Redis set with TTL). Check the ID before processing. This pattern is simpler than true exactly-once delivery, handles 99.9% of production duplicate scenarios, and has no throughput penalty.
Consumer Groups, Partitions, and Ordering Guarantees
Kafka's partition model determines both throughput and ordering guarantees in ways that are critical to understand before designing your event schema. Messages within a partition are strictly ordered. Messages across partitions are not. If ordering matters - say, all events for a specific order must be processed in sequence - you must ensure all events for that order go to the same partition. The standard pattern is to use the entity ID (order ID, user ID) as the partition key. All events for order-12345 hash to the same partition and are processed in sequence.
Consumer groups let multiple consumers process a topic in parallel. Each partition is assigned to exactly one consumer in a consumer group at a time. With 12 partitions and 12 consumers, all consumers process in parallel. With 12 partitions and 3 consumers, each consumer processes 4 partitions. Adding more consumers than partitions creates idle consumers - extra consumers get no partitions. Partition count determines your maximum parallelism. Choose partition count based on your throughput requirements and scale it up proactively; shrinking partitions requires repartitioning.
Event Sourcing Trade-offs in Practice
Event sourcing's benefits are substantial: complete audit trail, temporal queries, ability to replay events to fix bugs in projections, and natural integration with CQRS. The costs are also substantial. Snapshots become necessary as event streams grow (replaying 10 million events to get the current state of an account is impractical). Event schema migration is harder than column migration in a relational database: you need to either version your events or write migration code that transforms old events to new schemas on read.
The practical verdict: event sourcing is the right choice for domains where audit and temporal query requirements are genuine business requirements (financial transactions, compliance systems, collaborative editing). It is wrong for most CRUD applications where the current state is what matters and history is irrelevant. The complexity cost of event sourcing is significant. Do not introduce it without clear business requirements that justify it.
- Kafka partitions are the unit of parallelism, ordering, and availability. Design your partition key before designing your event schema.
- The at-least-once delivery guarantee means consumers must be idempotent. Use a deduplication key in each event and check it before processing.
- Kafka Streams and Apache Flink are stream processing frameworks built on Kafka. They enable stateful stream processing: windowed aggregations, joins between event streams, pattern detection.
- Avro schema registry (Confluent Schema Registry) enforces schema compatibility at publish time and enables schema evolution with full compatibility tracking.
- Lag monitoring is the most important Kafka operational metric: how far behind is the consumer from the latest partition offset? Sustained lag growth indicates a consumer that cannot keep up with the producer.
Exactly-Once Semantics: The Hardest Guarantee
Exactly-once delivery is the most desired and hardest to guarantee in distributed messaging. Kafka introduced exactly-once semantics (EOS) in version 0.11 (2017) for producer-to-broker delivery and idempotent consumers in Kafka Streams. The implementation requires a sequence number per producer-partition pair to deduplicate retries, and a two-phase commit protocol for consumer offsets and output records. The performance cost is real: enabling exactly-once semantics reduces throughput by approximately 20% compared to at-least-once delivery.
For most business applications, idempotent consumers with at-least-once delivery are the practical approach. Design each message consumer to be idempotent: processing the same message twice produces the same outcome. The idempotency key is typically the message's event ID. Store processed event IDs in a deduplication table (or use a Redis set with TTL). Check the ID before processing. This pattern is simpler than true exactly-once delivery, handles 99.9% of production duplicate scenarios, and has no throughput penalty.