Skip to content
API Design Learn/Event-Driven APIs
Browsing as a guest. Sign in to save your progress and earn XP as you complete chapters.

Dead Letter Queues & Retry Policies

4 min read

You'll learn to

  • -Retry transient failures with exponential backoff and a limit, never forever and never instantly
  • -Move a message that cannot succeed to a dead letter queue so it stops blocking the stream
  • -Tell transient from permanent failures, and watch the waiting time that distinction saves

A consumer will eventually meet a message it truly cannot process: malformed data, a dependency that has been down for hours, or a bug that makes one particular message always crash the handler. Without a deliberate policy, that one message can block everything behind it in the same stream indefinitely.

An Analogy: The Jammed Parcel on the Conveyor Belt

In a sorting office, one parcel with an unreadable label can jam the belt, and every parcel behind it waits. The sensible rule is to try the label scanner a few times, with pauses in case it was a smudge, and then lift the parcel off into a "problem" tray for a person to look at, while the belt carries on. The tray is the dead letter queue. A tray no one ever checks is just a slower way to lose parcels.

Retry With Exponential Backoff

Retrying immediately and repeatedly can make a transient problem worse. Hammering an already struggling dependency with instant retries adds exactly the load it can least handle. Exponential backoff means each retry waits longer than the last, usually up to a cap: 2 seconds, 4, 8, 16, and then no more than 60. That gives a temporary failure real time to clear, instead of retrying at a fixed, aggressive rhythm whether or not the problem has had a chance to recover.

Dead Letter Queues: Isolating What Retrying Cannot Fix

After the retry budget is used up, a message that still fails is moved to a dead letter queue (DLQ), a separate holding area for messages that need human investigation. It is neither retried forever nor quietly dropped. Crucially, moving it out of the main stream means one permanently broken message no longer blocks all the legitimate messages queued behind it.

Not Every Failure Deserves the Same Retry Treatment

A transient failure, such as a downstream timeout or a network blip, benefits from retrying, because the next attempt may well succeed. A message that fails validation, or triggers a deterministic bug, fails identically on every retry. Retrying it with backoff just delays its inevitable trip to the DLQ. Telling the two kinds apart, the same retriable versus non-retriable distinction as in gRPC error handling, and sending permanent failures straight to the DLQ, saves the whole retry budget.

The demo processes four messages. Message 2 is a poison message that can never succeed. Message 3 fails twice with a timeout and then works. It runs the stream with a policy that retries everything, and then with one that sends permanent failures straight to the DLQ. Compare the waiting times.

Press Run: the same four messages, 36 seconds of waiting against 6

A DLQ that nobody monitors is just a quieter way to lose messages. The pattern delivers value only with alerting and a real process for investigating and, where appropriate, replaying the messages once the underlying problem is fixed.

Interview Signal is part of Pro

See a real weak answer next to a real strong one for this exact topic.

Quiz is part of Pro

Test what you just read with a short quiz, and bank the XP.

Ready to Build This?

Design Dead Letter Detective in the API Design Lab's gRPC & Event-Driven act.

Discussion0

Join the Discussion

Sign in to leave comments, reply to others, or like insights.

Sign In to ScaleDojo

No comments yet. Be the first to start the thread!