Message Queue Patterns
Async ArchitectureHow you decouple producers from consumers, handle backpressure, and ensure reliable message delivery.
The post office is the queue. You (producer) drop letters in the mailbox without knowing who the recipient is or whether they're home. The postal service (queue) holds the letter. The recipient (consumer) picks up their mail when they're ready. The post office guarantees delivery even if the recipient was on vacation when you sent it.
Point-to-Point (Queue): One producer sends to a queue. One of potentially many consumers reads from it. Each message is processed exactly once. Use case: distributing work across a pool of workers. AWS SQS, RabbitMQ queues.
Publish-Subscribe (Topic): One producer publishes to a topic. ALL subscribed consumers receive a copy. Use case: broadcasting an event to multiple services (order_placed event → notify inventory service, billing service, email service simultaneously). AWS SNS, Kafka topics.
Dead Letter Queue (DLQ): When a message fails processing N times (after retries), it is moved to a DLQ for manual inspection instead of blocking the main queue. Essential for production reliability - a malformed message ("poison pill") can otherwise block an entire queue forever.
Competing Consumers: Multiple consumer instances read from the same queue, each processing different messages concurrently. This is how you horizontally scale message processing - run 50 workers instead of 1 to consume faster.
Message Ordering: Standard queues (SQS Standard, Kafka with multiple partitions) do NOT guarantee ordering. FIFO queues (SQS FIFO, Kafka with a single partition keyed by ID) guarantee order for a given partition key but limit throughput. For most use cases, exact ordering is not required and adds unnecessary cost.
Idempotency: At-least-once delivery (the safe default in most queues) means a message might be delivered multiple times due to network retries. Your consumer MUST be idempotent - processing the same message twice should have the same effect as processing it once. Use unique message IDs and check-before-process.
- Queue = work distribution (point-to-point). Topic = event broadcast (pub-sub).
- DLQs are mandatory in production - never leave poison pills blocking your queue.
- Design consumers to be idempotent - at-least-once delivery is almost universal.
- Visibility timeout (SQS) or ack timeout (RabbitMQ): how long a consumer has to process before the message is re-queued.
- Kafka = durable log (messages retained for days/weeks, replay possible). SQS = queue (messages deleted on ack).
Curated Curation & Deep Insights
ScaleDojo CertifiedOur system architects are vetting high-quality, authorized video guides for Message Queue Patterns with zero third-party platform links.
We are preparing premium, zero-competitor deep-dives for Message Queue Patterns. Authorized reading references will appear automatically.