Dead Letter Queues & Delivery Guarantees

Queues make a system forgiving: if a worker is slow or briefly down, messages wait. But a queue also has a failure mode that a direct call does not: a message that can never be processed. And because messages can be delivered more than once, you need to know exactly what a queue promises. This chapter covers the safety valve for the first problem and the vocabulary for the second.

A post office tries to deliver every letter. If the address does not exist, it does not keep sending a van to the same wrong door for ever, and it does not leave that letter at the front of the sorting line where it blocks everyone else's mail. After a few attempts the letter goes to a special office for undeliverable mail, where a person looks at it and decides what to do.

• -The letter that can never be delivered is a poison pill: a message that fails every time it is processed.
• -The attempts the post office makes are retries.
• -The special office for undeliverable mail is the dead letter queue (DLQ).
• -The person who inspects it is you: someone has to look at what lands there.

What happens when a message is malformed, or triggers a bug, and a worker can never process it? Left alone, that single message is retried for ever, burning worker capacity, and in an ordered queue it can block every message behind it. Picture 10,000 orders waiting behind one that always fails: not one of them moves.

The diagram shows the safety valve. After a set number of failed attempts, the message leaves the main queue for the DLQ, and processing carries on.

The card below describes the component as it appears in the lab:

A DLQ is the standard remedy, and a production queue without one is a common source of silent, hard-to-diagnose outages.

Not every failure is a poison pill. A transient failure, such as a timeout or a briefly unavailable database, will probably succeed if tried again in a moment, so retry it, waiting a little longer each time (exponential backoff: 1 second, 2, 4, 8) so you do not hammer a service that is already struggling. A permanent failure, such as invalid data, will never succeed however many times you try, so there is no point retrying it: send it to the DLQ quickly. A limit on attempts, commonly three to five, covers the cases you cannot tell apart.

Three words describe what a queue promises about delivery, and a postal comparison makes them concrete:

• -At-most-once: like sending a postcard. It is delivered zero or one times, never twice, but it can be lost with no one knowing. It is simple, but messages can silently disappear.
• -At-least-once: like registered mail that is resent until someone signs for it. It is delivered one or more times, so nothing is lost, but if the signature slip is lost on the way back, the receiver gets it twice. This is the safe default for most queues, and your consumer must expect duplicates.
• -Exactly-once: each message takes effect exactly one time, with no loss and no duplicates. Achieving this across a network is notoriously hard. In practice it usually means at-least-once delivery plus an idempotent consumer, one that notices a message it has already handled and ignores it. The chapter on idempotency later in the course shows how.

A concrete case: a payment message is delivered twice because the acknowledgement was lost. If the consumer simply charges the card each time, the customer is charged twice. If the consumer records each message id as it handles it, the second delivery is recognized and skipped.

• -Alert when it is not empty: a DLQ that nobody watches is a silent hole where messages go to be forgotten.
• -Inspect: find out why the messages failed, a bug in your code or bad data from a producer.
• -Fix, then redrive: after the cause is fixed, move the messages back to the main queue to be processed.
• -Set a retention period: DLQ messages should not live for ever, but should live long enough to be investigated.