Skip to content
API Design Learn/Event-Driven APIs
Browsing as a guest. Sign in to save your progress and earn XP as you complete chapters.

Exactly-Once & the Outbox Pattern

4 min read

You'll learn to

  • -Explain why exactly-once delivery cannot be guaranteed by the transport, and what at-least-once means
  • -Make a consumer idempotent so duplicates have no effect, and run it on a duplicated delivery
  • -Use the transactional outbox so a saved change and its event are never separated by a crash

Message queues and event streams almost always promise at-least-once delivery, not exactly-once. At-least-once means a message is delivered one or more times, never zero. Understanding why matters, because a system that assumes true exactly-once delivery is built on a guarantee that does not really exist.

An Analogy: The Recorded Delivery Letter

You post a letter and wait for the signed receipt. The receipt never comes back. Did the letter get lost, or did the receipt get lost? You cannot tell from where you stand. If you send the letter again, you risk the recipient getting two. If you do not, you risk them getting none. Sending again is the safer choice, so the recipient must be able to cope with a duplicate.

Why True Exactly-Once Is Effectively Unachievable

A publisher sends a message, and the network or the broker fails before the acknowledgment arrives. Unsure whether the message got there, the publisher retries. If the original did arrive and only the acknowledgment was lost, the retry creates a duplicate. The publisher cannot tell "the message was lost" from "the message arrived but the ack was lost" without extra machinery, and that machinery is the idempotent consumer.

Idempotent Consumers: Exactly-Once Effect

The practical goal is at-least-once delivery with an exactly-once effect. The transport may deliver the same event twice, but the consumer detects the duplicate using the event's unique id, the event_id from the domain-events chapter, and ignores it. The net effect on the system is as if the event had been handled once.

The Outbox Pattern: Atomically Coupling a Write With Publishing

There is a second, separate problem. How do you guarantee that an event is published if and only if the database write it describes succeeded? Writing to a database and publishing to a broker are two separate systems. If the write succeeds and the service crashes before publishing, the event is lost. If the publish succeeds and the write then fails, a phantom event describes something that never happened.

The outbox table - written in the SAME local transaction as the business data
BEGIN;
INSERT INTO orders (id, status) VALUES (501, 'shipped');
INSERT INTO outbox_events (event_type, payload, published)
  VALUES ('order.shipped', '{"order_id": 501}', false);
COMMIT;
-- both writes succeed or both roll back TOGETHER - a single local
-- transaction, no distributed transaction across two separate systems

The outbox table lives in the same database as the business data, so saving the change and saving the "event to publish" record happen in one ordinary local transaction. Both succeed or both roll back. A separate background process, the relay, then reads unpublished rows and publishes them to the broker, marking each as published once the broker confirms. The demo shows all three ideas in one run: a lost acknowledgment producing a duplicate, a consumer that ignores it, and a naive save losing an event that the outbox version keeps.

Press Run: two copies delivered but one effect, and only the outbox version keeps its event after a crash

The outbox and idempotent consumers solve complementary halves of the same problem. The outbox guarantees an event is never silently lost on the publishing side. Idempotent consumers guarantee a redelivered event is never processed twice on the receiving side. A robust exactly-once-effect system needs both.

Interview Signal is part of Pro

See a real weak answer next to a real strong one for this exact topic.

Quiz is part of Pro

Test what you just read with a short quiz, and bank the XP.

Ready to Build This?

Design Exactly Once in the API Design Lab's gRPC & Event-Driven act.

Discussion0

Join the Discussion

Sign in to leave comments, reply to others, or like insights.

Sign In to ScaleDojo

No comments yet. Be the first to start the thread!