The Uber Eats Order That Crashed 10,000 Deliveries
In 2020, an Uber Eats partner restaurant sent an order update with a malformed JSON payload - a single emoji character in a field that expected ASCII. The order processing consumer crashed on deserialization. The message went back to the queue. Consumer restarted, grabbed the same message, crashed again. This single poison message blocked the entire queue for that region. 10,000 food deliveries were delayed because one bad message kept crashing the consumer in an infinite retry loop. The fix: a dead letter queue that catches messages after 3 failed attempts and moves them aside for investigation.
Dead Letter Queue Architecture
DLQ Flow:
Producer --> [Main Queue] --> Consumer (attempt 1) --> FAIL
| |
|<------------ retry (backoff) --------|
|
Consumer (attempt 2) --> FAIL
|
|<------------ retry (backoff x2) -----|
|
Consumer (attempt 3) --> FAIL
|
|-- max retries exceeded -->
|
[Dead Letter Queue]
|
Store with metadata:
- Original queue name
- Failure reason / stack trace
- Retry count (3)
- First failure timestamp
- Last failure timestamp
- Original message headers
|
[Alert: ops team / PagerDuty]
|
Engineer investigates
|
Fix deployed --> [Replay Tool] --> [Main Queue]Types of Message Failures
Failure Classification:
Type Retry? DLQ? Example
--------------- ------ ----- --------------------------
Transient Yes After N Database timeout, network blip
Poison message No Yes Malformed JSON, invalid schema
Business error No Yes Order already cancelled
Dependency down Yes After N Payment gateway unavailable
Bug in consumer No Yes NullPointerException on valid data
Retry Strategy:
Attempt 1: immediate retry
Attempt 2: wait 1 second
Attempt 3: wait 5 seconds
Attempt 4: wait 30 seconds
Attempt 5: -> DLQ (total wait: ~36 seconds)
For transient errors: exponential backoff (1s, 2s, 4s, 8s, 16s)
For poison messages: fail fast, DLQ immediately (no point retrying)DLQ Implementation by Platform
Platform-Specific DLQ Setup:
AWS SQS:
Main Queue --> maxReceiveCount: 3 --> DLQ
Built-in: just create a second queue and set redrive policy.
Cloud-native, zero code, just configuration.
RabbitMQ:
x-dead-letter-exchange: 'dlx'
x-dead-letter-routing-key: 'dlq.orders'
x-message-ttl: 300000 (5 min before retry from DLQ)
Per-queue DLQ with automatic routing.
Kafka:
No built-in DLQ! Manual implementation required:
try:
process(message)
consumer.commit()
except RetryableError:
if retry_count < MAX_RETRIES:
producer.send('orders-retry', message) # Retry topic
else:
producer.send('orders-dlq', message) # DLQ topic
except NonRetryableError:
producer.send('orders-dlq', message) # Immediate DLQ
Best practice: separate retry topic (with delay) from DLQ topic.DLQ Operations Playbook
Monitor DLQ depth - set alerts when depth > 0 (any message in DLQ is a problem to investigate).
Include rich metadata - original queue, error message, stack trace, retry count, timestamps. Without this, debugging is guesswork.
Build replay tooling - after fixing the bug, replay DLQ messages back to the main queue. Make this a one-click operation.
Set TTL on DLQ messages - messages older than 7-30 days are likely stale. Archive or purge them.
Categorize failures - separate transient failures (will succeed on replay) from permanent failures (need code fix).
Never ignore DLQ - a growing DLQ means data is being lost. Treat it like a production incident.
Interview Tip
When designing any message-driven system, always mention DLQs proactively. Interviewers want to see you think about failure cases, not just the happy path. The standard pattern: main queue with retry policy (3-5 attempts with exponential backoff), then DLQ for investigation and replay. For Kafka, explain that you need to implement DLQ manually with a separate topic since Kafka has no built-in DLQ. Mention the operational aspects: monitoring, alerting, replay tooling, and TTL. The phrase that impresses: 'every production queue needs a corresponding dead letter queue with monitoring and replay capability.'
Key Takeaway
Dead letter queues prevent poison messages from blocking your pipeline. After a configured number of retries with exponential backoff, failed messages move to the DLQ with rich metadata for investigation. Every production queue needs a DLQ, monitoring on DLQ depth, and one-click replay tooling. A growing DLQ is always a production incident.