Exactly-Once & the Outbox Pattern
You'll learn to
- -Explain why exactly-once delivery cannot be guaranteed by the transport, and what at-least-once means
- -Make a consumer idempotent so duplicates have no effect, and run it on a duplicated delivery
- -Use the transactional outbox so a saved change and its event are never separated by a crash
Message queues and event streams almost always promise at-least-once delivery, not exactly-once. At-least-once means a message is delivered one or more times, never zero. Understanding why matters, because a system that assumes true exactly-once delivery is built on a guarantee that does not really exist.
An Analogy: The Recorded Delivery Letter
You post a letter and wait for the signed receipt. The receipt never comes back. Did the letter get lost, or did the receipt get lost? You cannot tell from where you stand. If you send the letter again, you risk the recipient getting two. If you do not, you risk them getting none. Sending again is the safer choice, so the recipient must be able to cope with a duplicate.
Why True Exactly-Once Is Effectively Unachievable
A publisher sends a message, and the network or the broker fails before the acknowledgment arrives. Unsure whether the message got there, the publisher retries. If the original did arrive and only the acknowledgment was lost, the retry creates a duplicate. The publisher cannot tell "the message was lost" from "the message arrived but the ack was lost" without extra machinery, and that machinery is the idempotent consumer.
Idempotent Consumers: Exactly-Once Effect
The practical goal is at-least-once delivery with an exactly-once effect. The transport may deliver the same event twice, but the consumer detects the duplicate using the event's unique id, the event_id from the domain-events chapter, and ignores it. The net effect on the system is as if the event had been handled once.
The Outbox Pattern: Atomically Coupling a Write With Publishing
There is a second, separate problem. How do you guarantee that an event is published if and only if the database write it describes succeeded? Writing to a database and publishing to a broker are two separate systems. If the write succeeds and the service crashes before publishing, the event is lost. If the publish succeeds and the write then fails, a phantom event describes something that never happened.
BEGIN;
INSERT INTO orders (id, status) VALUES (501, 'shipped');
INSERT INTO outbox_events (event_type, payload, published)
VALUES ('order.shipped', '{"order_id": 501}', false);
COMMIT;
-- both writes succeed or both roll back TOGETHER - a single local
-- transaction, no distributed transaction across two separate systemsThe outbox table lives in the same database as the business data, so saving the change and saving the "event to publish" record happen in one ordinary local transaction. Both succeed or both roll back. A separate background process, the relay, then reads unpublished rows and publishes them to the broker, marking each as published once the broker confirms. The demo shows all three ideas in one run: a lost acknowledgment producing a duplicate, a consumer that ignores it, and a naive save losing an event that the outbox version keeps.
The outbox and idempotent consumers solve complementary halves of the same problem. The outbox guarantees an event is never silently lost on the publishing side. Idempotent consumers guarantee a redelivered event is never processed twice on the receiving side. A robust exactly-once-effect system needs both.
Interview Signal is part of Pro
See a real weak answer next to a real strong one for this exact topic.
Quiz is part of Pro
Test what you just read with a short quiz, and bank the XP.
Design Exactly Once in the API Design Lab's gRPC & Event-Driven act.
Discussion0
Join the Discussion
Sign in to leave comments, reply to others, or like insights.
No comments yet. Be the first to start the thread!