Dead Letter Queues & Delivery Guarantees
You'll learn to
- -Explain what a poison pill is and how a DLQ handles it
The Poison Pill Problem
What happens when a message is malformed, or triggers a bug, and a worker can never successfully process it? Left alone, that single message gets retried forever, burning worker capacity and (in ordered queues) potentially blocking every message behind it. This is called a "poison pill."
A Dead Letter Queue (DLQ) is the 'Return to Sender' pile at the post office. Imagine a letter that the postal service tries to deliver 3 times, but the address is wrong or the recipient refuses it every time. Instead of trying forever (blocking all the other mail behind it) or throwing it away (losing it permanently), the post office puts it in a special bin for undeliverable mail where someone can look at it later. In software, when a message from your queue fails processing 3 times (maybe the data is malformed, maybe a downstream service is broken), the queue moves that message to the DLQ. This is CRITICAL because without a DLQ, a single bad message (called a 'poison pill') can block your entire queue forever. Your consumer keeps trying to process it, keeps failing, and every message behind it is stuck waiting. With a DLQ, the bad message gets safely set aside, the rest of the queue keeps flowing, and an engineer can investigate the DLQ later to fix the problem and reprocess the messages.
Examples: AWS SQS DLQ (automatic after maxReceiveCount), Kafka dead letter topics, RabbitMQ dead-letter exchanges, Azure Service Bus dead-letter sub-queue, Celery task_reject_on_worker_lost
After N failed attempts, a poison-pill message is moved out of the main queue and into a DLQ instead of retrying forever.
A Dead Letter Queue is the safety valve: after N failed attempts, a message is moved out of the main queue into the DLQ for manual inspection, instead of being retried indefinitely. Production message queues without a DLQ are a common source of silent, hard-to-diagnose outages.
Delivery Guarantees
- -At-most-once: a message is delivered zero or one times. Simple, but messages can be silently lost.
- -At-least-once: a message is delivered one or more times. The safe default for most queues, but your consumer might see the same message twice.
- -Exactly-once: each message is processed exactly one time, no duplicates, no loss. Genuinely achieving this across a network is notoriously difficult; the Idempotency chapter later in this course explains why "exactly-once" in practice usually means "at-least-once delivery plus an idempotent consumer."
Why can't a queue just guarantee true exactly-once delivery and remove this whole problem?
"Some message queues advertise exactly-once, so it must be solvable."
"The fundamental issue is a network can always fail silently between 'the worker finished processing' and 'the worker told the queue it finished': the queue has no way to distinguish 'never processed' from 'processed but the acknowledgment got lost,' so it has to guess, and guessing safely means redelivering. That's why the honest, achievable guarantee is at-least-once delivery plus an idempotent consumer, not true exactly-once; some systems brand this combination as 'exactly-once processing,' but the delivery itself is still at-least-once underneath."