The Bank Transfer That Happened Twice
A stream processor reads a bank transfer event: 'Move $500 from Account A to Account B.' It processes the transfer, writes to the database, and then crashes before it can acknowledge the message to Kafka. Kafka re-delivers the event. The processor runs again. Account A just lost $1,000 instead of $500. This is why exactly-once processing is treated as the holy grail of distributed stream processing, and why it's genuinely hard to get right.
The Three Delivery Guarantees
Before you can reason about exactly-once semantics, you need to be honest about what your system is actually promising today. Most production pipelines fall into one of three buckets, and picking the wrong one for the job is where outages start.
Delivery Guarantee Comparison:
At-Most-Once (fire and forget):
[Producer] --msg--> [Broker] (no ACK waited)
If msg lost in transit -> gone forever
Use for: unimportant metrics, logs you can lose
At-Least-Once (retry until ACK):
[Producer] --msg--> [Broker] --ACK--> [Producer]
If ACK lost -> producer retries -> DUPLICATE
[Consumer] --process--> [DB] --crash before offset commit--
Kafka re-delivers -> processed TWICE
Use for: most things, IF operations are idempotent
Exactly-Once (the holy grail):
Each message processed PRECISELY ONCE
Even across crashes, retries, network failures
Use for: financial transactions, billing, inventoryNotice that at-least-once isn't a lesser guarantee, it's often the more honest one. Once you accept that duplicates can happen, the design question shifts to making duplicates harmless rather than pretending they won't occur.
Why Exactly-Once Is So Hard
The difficulty isn't in any single step of the pipeline, it's in the gaps between steps. Every hop from reading a message to committing its result is a place where a crash can leave the system in an ambiguous state.
The Crash Window Problem:
[Read from Kafka] -> [Process] -> [Write to DB] -> [Commit offset]
1 2 3 4
Crash after step 3, before step 4:
- DB has the result (step 3 committed)
- Kafka offset NOT advanced (step 4 never happened)
- Kafka re-delivers the message
- Processor writes to DB AGAIN
- Result: DUPLICATE processing
Crash after step 2, before step 3:
- DB does NOT have the result
- Kafka re-delivers (good!)
- Processor writes to DB (now correct)
- But what if step 2 had side effects? (sent an email?)
The gap between ANY two steps is a crash window.
Every gap is a potential duplicate or loss.This is the same underlying tension you run into with cache invalidation: two systems (Kafka's offset and your database) need to agree on the truth, and there's no free way to update both atomically unless the infrastructure does it for you. If you've read the piece on caching strategies, the trade-offs here will feel familiar, it's the same problem of keeping two sources of state in sync without a shared transaction.
How Kafka Achieves Exactly-Once
Kafka solves this with two separate mechanisms that work together rather than one silver bullet.
Kafka's Two Mechanisms:
1. Idempotent Producer (dedup at broker):
Producer assigns sequence numbers to messages.
Broker tracks: (producer_id, partition, sequence).
Duplicate sequence? Broker silently drops it.
Result: no duplicate WRITES to Kafka.
2. Kafka Transactions (atomic read-process-write):
BEGIN TRANSACTION
Read from input-topic (offsets 100-105)
Process events
Write results to output-topic
Commit consumer offsets (100-105)
END TRANSACTION
Either ALL of these happen, or NONE do.
Crash mid-transaction? Kafka aborts everything.
Consumer offsets + output writes are ATOMIC.
This is how Kafka Streams achieves exactly-once.The idempotent producer handles duplicate writes at the broker level, while transactions bundle the consumer offset commit and the downstream write into a single atomic unit. If you want the fuller picture of how Kafka fits into an event-driven architecture, including where event sourcing and AMQP-style brokers diverge, the AMQP, Kafka, and Event Sourcing post covers that ground well.
How Flink Achieves Exactly-Once
Flink takes a different approach, one built around distributed snapshots rather than transactional writes to a single broker.
Flink's Checkpoint Mechanism (Chandy-Lamport):
[Source] -> [Map] -> [Aggregate] -> [Sink]
| |
barrier ----+----------+--- checkpoint barrier
| |
[snapshot] [snapshot]
| |
[S3 / HDFS checkpoint storage]
Every N seconds, Flink inserts a barrier into the stream.
When barrier reaches each operator:
1. Operator snapshots its state to S3
2. Passes barrier downstream
When ALL operators have checkpointed:
Checkpoint is COMPLETE.
On failure:
1. Restore all operators from last checkpoint
2. Replay input from that checkpoint's offset
3. Combined with idempotent sinks = exactly-onceThe checkpoint barrier is really just a coordination signal, it makes sure every operator agrees on a consistent point in the stream before anyone's state is considered durable. On its own, though, a checkpoint only guarantees that Flink's internal state is consistent. Getting exactly-once semantics all the way to an external system still depends on the sink behaving idempotently on replay, which is the same principle you'll see if you look at outbox-based delivery patterns.
The Practical Fallback: Idempotent Sinks
Most teams don't need Kafka Streams transactions or Flink checkpoints. What they need is a way to make at-least-once delivery safe, and that usually comes down to a unique key and a database constraint.
When exactly-once infra is too heavy, use:
at-least-once delivery + idempotent operations
Example (payment processing):
INSERT INTO payments (transaction_id, amount, status)
VALUES ('txn-abc-123', 500, 'completed')
ON CONFLICT (transaction_id) DO NOTHING;
Even if processed 3 times, only 1 row exists.
The transaction_id is the idempotency key.
This is what Stripe does: every API call has
an Idempotency-Key header. Retries are safe.This is basically the same idea behind the transactional outbox pattern: instead of trying to make two separate systems commit atomically, you record the event and its unique identifier in a place that naturally rejects duplicates. It's less elegant than true exactly-once delivery, but it's far easier to reason about during an incident, and it doesn't require every operator in your pipeline to support checkpointing.
Interview Tip
When discussing stream processing guarantees, say: 'True exactly-once requires atomic coordination between reading, processing, and writing. Kafka achieves this with idempotent producers (sequence-based dedup) and transactions (atomic offset commits and writes). Flink uses Chandy-Lamport checkpoints to snapshot state and replay from checkpoints on failure. For most practical systems, I would use at-least-once delivery with idempotent sinks, using a unique transaction ID as a dedup key so processing the same event twice is a no-op. This gives exactly-once SEMANTICS without the complexity of exactly-once DELIVERY.'
Architecture overview
Key Takeaway
Exactly-once is achieved by combining idempotent producers, transactional commits, and recoverable state. Kafka and Flink both provide this but at a performance cost. When exactly-once infrastructure is too heavy, design idempotent sinks as a simpler alternative.
Related Reading
If you're building out a broader event-driven system and want the surrounding context, these go well with this post:
AMQP, Kafka, and Event Sourcing: The Async API Story, for the messaging fundamentals behind Kafka's exactly-once mechanics.
AsyncAPI, CloudEvents, and the Transactional Outbox Pattern, for the idempotency and delivery-guarantee patterns referenced in the fallback section above.
Caching Strategies: Invalidation, Placement, and Real-World Trade-offs, for a related look at keeping two systems of record consistent under failure.
LLD: Master the Building Blocks of Low-Level Design, if you want to work through these coordination problems at the code level.