Message Queue Patterns: Pub/Sub, Point-to-Point
You'll learn to
- -Distinguish a queue (point-to-point) from a topic (publish/subscribe) by who receives each message
- -Explain competing consumers and consumer groups with a worked example
- -Choose between a queue and a topic for work distribution and for fan-out
- -Explain acknowledgements, and what happens when a consumer dies mid-message
The queue in the last chapter needs its own vocabulary. Not every queue behaves the same way, and the difference between a queue that hands each message to one worker and a topic that broadcasts it to many shapes the whole system around it. Getting the choice wrong is a common design mistake: use a queue where you needed a broadcast and only one of your services hears the news.
An Analogy: A Deli Counter and a Newspaper
At a deli counter, each ticket is called once, and whichever server is free takes it. No two servers ever handle the same ticket. A newspaper works the other way: one edition is printed and a copy goes to every subscriber, and each household reads it independently. Real systems often combine the two: each household has a shared mailbox, so the copy a household receives is handled by only one member of that household.
- -The deli counter is a queue, also called point-to-point: each message is claimed by exactly one consumer.
- -The newspaper is a topic, also called publish/subscribe: each message is delivered to every subscriber.
- -The household's shared mailbox is a consumer group: a copy per group, but only one consumer in the group processes it.
The diagram shows both patterns side by side:
Point-to-point: each message is claimed by exactly one worker. Pub/Sub: each message is delivered to every subscriber independently.
Competing Consumers: A Worked Example
Say an order is placed and an event is published to a topic. Three services care: email, inventory and analytics, and each is a consumer group. Every message reaches all three groups. Inside the email group there are four workers, and they compete: each message is processed by exactly one of them. If 100 orders arrive, the email group, the inventory group and the analytics group each see all 100 messages, and each of the four email workers handles about 25. Adding a fifth email worker reduces each one's share to about 20 without touching any other service. This is how queues give you horizontal scaling for free.
Choosing Between Them
- -Distribute work among identical workers (resize an image, send an email): a queue.
- -Tell several different services that something happened (order placed): a topic.
- -Both at once, which is the usual case in a larger system: a topic with a consumer group per interested service.
The first card describes the queue component as it appears in the lab:
A message queue is like a to-do list between two people who work different shifts. The day-shift person writes tasks on the list ('Process order #1234,' 'Send welcome email to alice@example.com,' 'Generate monthly invoice for Acme Corp'). The night-shift person comes in, picks tasks off the list one by one, and processes them. They never need to talk to each other directly. The list (queue) acts as a buffer between them. In software, a producer (like your web server) puts messages into the queue, and a consumer (like a worker service) pulls messages out and processes them. The beautiful thing: if the night shift person is slow, the tasks just accumulate in the list. Nothing is lost. If the day shift suddenly produces 10x more tasks (Black Friday!), the list absorbs the spike and the night shift works through it at its own pace. The queue DECOUPLES the producer from the consumer. Neither needs to know about the other, and neither crashes because the other is slow.
Examples: Apache Kafka (distributed log, supports replay), AWS SQS (fully managed, nearly infinite scale), RabbitMQ (feature-rich routing), Google Pub/Sub, Redis Streams, Azure Service Bus
And the second describes the topic, the publish/subscribe component:
If a message queue is like a to-do list where ONE person picks up each task, Pub/Sub (Publish-Subscribe) is like a radio broadcast. One person speaks (the publisher) and EVERYONE tuned in hears the message. When your Order Service publishes 'OrderPlaced: order #5678, user #42, total $150,' EVERY subscriber receives a copy: the Inventory Service (to reserve stock), the Payment Service (to charge the card), the Email Service (to send confirmation), and the Analytics Service (to update dashboards). Each subscriber acts independently. They don't know about each other. This is the foundation of event-driven architectures where services communicate through events instead of direct API calls. The publisher doesn't care who's listening or how many subscribers there are. It just shouts into the void, and the Pub/Sub system ensures every subscriber receives the message.
Examples: AWS SNS (fan-out to SQS/Lambda/email/SMS), Google Pub/Sub, Apache Kafka topics, Redis Pub/Sub (in-memory, not durable), NATS (lightweight, high-performance)
Acknowledgements and Failed Consumers
What if a consumer takes a message and crashes halfway through? The queue does not delete a message when it hands it out. It hides the message for a limited time, often called the visibility timeout, say 30 seconds, and waits for an acknowledgement. If the consumer acknowledges in time, the message is deleted. If the consumer crashes, no acknowledgement arrives, the timeout ends, and the message reappears for another consumer. That protects you from losing work, but it also means a message can be processed twice, for example if a worker is merely slow rather than dead. This is why consumers should be written to be safe to repeat, an idea the later chapter on idempotency develops.
A Note on Ordering
A single queue keeps messages in order, but with several consumers working in parallel, messages finish in no guaranteed order. When order matters for one entity, such as all the events of one user, route every message for that user to the same partition or consumer by hashing the user id, the same idea as the shard key. You then get order within each user without serializing the whole system.
The Wiki card below compares queue and topic patterns in more depth:
How you decouple producers from consumers, handle backpressure, and ensure reliable message delivery.
The post office is the queue. You (producer) drop letters in the mailbox without knowing who the recipient is or whether they're home. The postal service (queue) holds the letter. The recipient (consumer) picks up their mail when they're ready. The post office guarantees delivery even if the recipient was on vacation when you sent it.
Point-to-Point (Queue): One producer sends to a queue. One of potentially many consumers reads from it. Each message is processed exactly once. Use case: distributing work across a pool of workers. AWS SQS, RabbitMQ queues.
Publish-Subscribe (Topic): One producer publishes to a topic. ALL subscribed consumers receive a copy. Use case: broadcasting an event to multiple services (order_placed event → notify inventory service, billing service, email service simultaneously). AWS SNS, Kafka topics.
Dead Letter Queue (DLQ): When a message fails processing N times (after retries), it is moved to a DLQ for manual inspection instead of blocking the main queue. Essential for production reliability - a malformed message ("poison pill") can otherwise block an entire queue forever.
Competing Consumers: Multiple consumer instances read from the same queue, each processing different messages concurrently. This is how you horizontally scale message processing - run 50 workers instead of 1 to consume faster.
Message Ordering: Standard queues (SQS Standard, Kafka with multiple partitions) do NOT guarantee ordering. FIFO queues (SQS FIFO, Kafka with a single partition keyed by ID) guarantee order for a given partition key but limit throughput. For most use cases, exact ordering is not required and adds unnecessary cost.
- -Queue = work distribution (point-to-point). Topic = event broadcast (pub-sub).
- -DLQs are mandatory in production - never leave poison pills blocking your queue.
- -Design consumers to be idempotent - at-least-once delivery is almost universal.
- -Visibility timeout (SQS) or ack timeout (RabbitMQ): how long a consumer has to process before the message is re-queued.
- -Kafka = durable log (messages retained for days/weeks, replay possible). SQS = queue (messages deleted on ack).
In one line: a queue gives each message to one consumer, a topic gives a copy to every subscriber, and consumer groups combine both. Acknowledgements keep work from being lost, which is why a message can occasionally be delivered twice.
Interview Signal is part of Pro
See a real weak answer next to a real strong one for this exact topic.
Quiz is part of Pro
Test what you just read with a short quiz, and bank the XP.
Discussion0
Join the Discussion
Sign in to leave comments, reply to others, or like insights.
No comments yet. Be the first to start the thread!