Message Queue Patterns: Pub/Sub, Point-to-Point
You'll learn to
- -Distinguish a queue from a topic
The queue in the last diagram needs its own vocabulary: not every "queue" behaves the same way, and the difference between a work queue and a broadcast topic shapes the whole system around it.
Point-to-point: each message is claimed by exactly one worker. Pub/Sub: each message is delivered to every subscriber independently.
A message queue is like a to-do list between two people who work different shifts. The day-shift person writes tasks on the list ('Process order #1234,' 'Send welcome email to alice@example.com,' 'Generate monthly invoice for Acme Corp'). The night-shift person comes in, picks tasks off the list one by one, and processes them. They never need to talk to each other directly. The list (queue) acts as a buffer between them. In software, a producer (like your web server) puts messages into the queue, and a consumer (like a worker service) pulls messages out and processes them. The beautiful thing: if the night shift person is slow, the tasks just accumulate in the list. Nothing is lost. If the day shift suddenly produces 10x more tasks (Black Friday!), the list absorbs the spike and the night shift works through it at its own pace. The queue DECOUPLES the producer from the consumer. Neither needs to know about the other, and neither crashes because the other is slow.
Examples: Apache Kafka (distributed log, supports replay), AWS SQS (fully managed, nearly infinite scale), RabbitMQ (feature-rich routing), Google Pub/Sub, Redis Streams, Azure Service Bus
If a message queue is like a to-do list where ONE person picks up each task, Pub/Sub (Publish-Subscribe) is like a radio broadcast. One person speaks (the publisher) and EVERYONE tuned in hears the message. When your Order Service publishes 'OrderPlaced: order #5678, user #42, total $150,' EVERY subscriber receives a copy: the Inventory Service (to reserve stock), the Payment Service (to charge the card), the Email Service (to send confirmation), and the Analytics Service (to update dashboards). Each subscriber acts independently. They don't know about each other. This is the foundation of event-driven architectures where services communicate through events instead of direct API calls. The publisher doesn't care who's listening or how many subscribers there are. It just shouts into the void, and the Pub/Sub system ensures every subscriber receives the message.
Examples: AWS SNS (fan-out to SQS/Lambda/email/SMS), Google Pub/Sub, Apache Kafka topics, Redis Pub/Sub (in-memory, not durable), NATS (lightweight, high-performance)
How you decouple producers from consumers, handle backpressure, and ensure reliable message delivery.
The post office is the queue. You (producer) drop letters in the mailbox without knowing who the recipient is or whether they're home. The postal service (queue) holds the letter. The recipient (consumer) picks up their mail when they're ready. The post office guarantees delivery even if the recipient was on vacation when you sent it.
Point-to-Point (Queue): One producer sends to a queue. One of potentially many consumers reads from it. Each message is processed exactly once. Use case: distributing work across a pool of workers. AWS SQS, RabbitMQ queues.
Publish-Subscribe (Topic): One producer publishes to a topic. ALL subscribed consumers receive a copy. Use case: broadcasting an event to multiple services (order_placed event → notify inventory service, billing service, email service simultaneously). AWS SNS, Kafka topics.
Dead Letter Queue (DLQ): When a message fails processing N times (after retries), it is moved to a DLQ for manual inspection instead of blocking the main queue. Essential for production reliability - a malformed message ("poison pill") can otherwise block an entire queue forever.
Competing Consumers: Multiple consumer instances read from the same queue, each processing different messages concurrently. This is how you horizontally scale message processing - run 50 workers instead of 1 to consume faster.
Message Ordering: Standard queues (SQS Standard, Kafka with multiple partitions) do NOT guarantee ordering. FIFO queues (SQS FIFO, Kafka with a single partition keyed by ID) guarantee order for a given partition key but limit throughput. For most use cases, exact ordering is not required and adds unnecessary cost.
- -Queue = work distribution (point-to-point). Topic = event broadcast (pub-sub).
- -DLQs are mandatory in production - never leave poison pills blocking your queue.
- -Design consumers to be idempotent - at-least-once delivery is almost universal.
- -Visibility timeout (SQS) or ack timeout (RabbitMQ): how long a consumer has to process before the message is re-queued.
- -Kafka = durable log (messages retained for days/weeks, replay possible). SQS = queue (messages deleted on ack).
You need both a payment-processing worker AND an analytics pipeline to react to every "order placed" event. Queue or Pub/Sub?
"A queue: it's the more common, simpler choice."
"Pub/Sub. A plain queue delivers each message to exactly one consumer: if both the payment worker and the analytics pipeline are competing consumers on the same queue, only one of them will ever see a given message, and you'll lose events. Pub/Sub delivers a copy of every message to every independent subscriber, which is exactly the 'multiple independent services should all react to the same event' shape this needs."