Message Delivery Guarantees: At-Least-Once vs Exactly-Once

The Double-Charge That Cost Stripe $0

The Double-Charge That Cost Stripe $0

In payment processing, a single duplicate message can charge a customer twice, which is exactly why message delivery guarantees matter so much in practice. Stripe processes billions of dollars in payments, and network failures happen constantly. When a payment confirmation message is lost between services, the system retries. Without idempotency, that retry charges the customer again. Stripe solved this with idempotency keys: every payment request includes a unique key. If the same key appears twice, the second request returns the original result without processing again, no new charge, no duplicate row, nothing. That's the punchline in the title: the double-charge bug that should have cost real money ended up costing Stripe exactly $0, because the idempotency key caught the duplicate before it ever touched the ledger. This pattern, at-least-once delivery plus idempotent consumers, is how every production system achieves effectively exactly-once processing.

The Three Delivery Guarantees

The Three Delivery Guarantees

Before getting into how systems fake exactly-once delivery, it helps to lay out the three guarantees a message queue can actually offer. Only one of them is achievable without extra engineering effort, and it's not the one most people assume.

Message Delivery Spectrum:

  At-Most-Once (fire and forget):
  Producer ---msg---> Queue ---msg---> Consumer
                        (no ACK required)
  If queue crashes or consumer fails: message LOST forever.
  Speed: fastest    Reliability: lowest   Data loss: possible
  Use for: metrics, logs, analytics (losing 0.1% is OK)

  At-Least-Once (retry until ACK):
  Producer ---msg---> Queue ---msg---> Consumer
                              <--ACK---
  If no ACK: queue resends. Consumer might process TWICE.
  Speed: fast        Reliability: high    Duplicates: possible
  Use for: most applications (with idempotent consumers)

  Exactly-Once (the holy grail):
  Producer ---msg---> Queue ---msg---> Consumer
                              <--ACK---
            (plus deduplication + transactional processing)
  Speed: slowest     Reliability: highest  Neither loss nor dups
  Use for: financial transactions, billing, inventory
  Reality: achieved via at-least-once + idempotency

Notice that the "exactly-once" row doesn't describe a different network protocol, it describes at-least-once delivery with extra bookkeeping bolted on. That's not a limitation of any particular queue, it's a limitation of distributed systems in general.

Why Exactly-Once Is Almost Impossible

The reason comes down to a classic problem: you cannot make "process the message" and "acknowledge the message" a single atomic operation when a network sits between the two parties. Something can always fail in the gap between them.

The Two Generals Problem Applied to Messaging:

  Consumer processes message successfully at 10:00:00.000
  Consumer sends ACK to queue at              10:00:00.001
  Network drops the ACK packet                10:00:00.002
  Queue never receives ACK                    10:00:00.003
  Queue thinks consumer failed                10:00:00.500
  Queue resends the SAME message              10:00:01.000
  Consumer receives it AGAIN                  10:00:01.001

  The consumer processed it. The queue does not know.
  This is FUNDAMENTAL to distributed systems - you cannot
  make 'process' and 'acknowledge' atomic across a network.

  Even Kafka's 'exactly-once semantics' (EOS) only works
  WITHIN Kafka (produce-consume-produce chains).
  Once you interact with an external database, you need
  application-level idempotency.

That last point trips up a lot of engineers who assume Kafka's EOS setting solves the problem everywhere. It solves it for chains that stay inside Kafka. The moment a consumer writes to a database, calls an API, or sends an email, you're back to needing idempotency at the application layer. If you want a deeper look at how Kafka's transactional guarantees actually work under the hood, this piece on AMQP, Kafka, and event sourcing is worth reading.

Making Consumers Idempotent

Watch out: the most common way idempotency breaks in real codebases is a stray balance += amount or a bare INSERT that slipped in during a refactor. Someone adds a "quick fix" months later, doesn't realize the handler can be called twice for the same message, and now you've got a silent double-charge bug that only shows up under load or during a network blip. Grep your consumers for non-idempotent writes before you trust the dedup table to save you.

Since the queue can't guarantee exactly-once, the responsibility shifts to the consumer. There are a handful of patterns that show up again and again in production systems, and most codebases end up combining two or three of them.

Idempotency Patterns:

  1. Deduplication Table:
  +--------------+------------+--------+
  | message_id   | processed  | result |
  +--------------+------------+--------+
  | msg-abc-123  | 10:00:00   | OK     |  <- First time: process + record
  | msg-abc-123  | (duplicate)|        |  <- Second time: skip, return cached result
  +--------------+------------+--------+

  2. Natural Idempotency:
  SET balance = 500        <- Idempotent (same result every time)
  balance += 50            <- NOT idempotent (adds 50 each time!)
  SET status = 'shipped'   <- Idempotent
  INSERT INTO orders(...)  <- NOT idempotent (creates duplicates!)

  3. Conditional Updates:
  UPDATE orders SET status = 'shipped'
  WHERE id = 123 AND status = 'paid'    <- Only transitions from 'paid'
  Second execution: status is already 'shipped', WHERE fails, no change.

  4. Version/Sequence Numbers:
  UPDATE accounts SET balance = 500, version = 4
  WHERE id = 42 AND version = 3
  Second execution: version is already 4, WHERE fails, no change.

The deduplication table is the most general-purpose option since it works regardless of what the message actually does, but it costs you a database write on every message and needs its own cleanup policy. Conditional updates and version numbers are cheaper when your data already has a natural state machine, which is often the case for orders, shipments, or account balances. If you're designing this kind of consumer alongside an outbox for reliable publishing, the transactional outbox pattern pairs naturally with deduplication tables on the receiving end.

Delivery Guarantees by Platform

Gotcha with SQS FIFO: the built-in dedup window is only 5 minutes. If your queue is backed up and a retry happens after that window closes, SQS treats it as a brand new message and your "exactly-once" guarantee quietly disappears. If your consumers can lag behind under load, don't rely on the FIFO window alone, add your own message-ID dedup table as a backstop.

Different messaging platforms handle this problem differently, and the defaults matter a lot when you're picking infrastructure for a new system. Here's how the major ones stack up.

Kafka's transactional API is the one place where exactly-once is a first-class feature rather than something you build yourself. Here's roughly what that looks like in practice:

Kafka Exactly-Once (Transactional):
  producer.beginTransaction()
  producer.send('output-topic', processedMessage)
  producer.sendOffsetsToTransaction(consumedOffsets)
  producer.commitTransaction()
  // If any step fails, ALL are rolled back atomically

Notice the scope, though. This transaction covers the consume-process-produce chain inside Kafka. It says nothing about a side effect like writing to a separate SQL database or calling a payment API, which is exactly why idempotency at the consumer level still matters even when you're using Kafka's strongest guarantees.

Interview Tip

This is one of the most frequently tested distributed systems concepts. The key insight interviewers want: true exactly-once is impossible across system boundaries (Two Generals Problem). The practical answer is at-least-once delivery plus idempotent consumers. Show three idempotency patterns: (1) deduplication table with message IDs, (2) conditional updates with state machine transitions, (3) version numbers for optimistic concurrency. When discussing payment systems, mention Stripe's idempotency key pattern. For Kafka specifically, explain that EOS works within Kafka's transaction boundary but not for external side effects.

Architecture overview

Key Takeaway

True exactly-once delivery is nearly impossible in distributed systems due to the fundamental problem of making 'process' and 'acknowledge' atomic across a network. The practical solution is at-least-once delivery plus idempotent consumers. Design every message consumer to handle duplicates safely using deduplication tables, conditional updates, or version numbers.

If you're building out these patterns as part of a broader system design study plan, it's worth pairing this with how caching and message delivery interact, since cache invalidation runs into the exact same duplicate-event problem from a different angle.