Ordering Keys, Partitions and Replay

Events describe things that happened, and for some things the order is the whole point. 'Order shipped' before 'order paid' would corrupt a customer's history, while two different customers' events can arrive in any order and nobody notices. A good event API says exactly which orderings it promises, because promising more than that costs throughput.

A supermarket with one lane serves everyone in strict order, and the line is long. With ten lanes, each customer's items stay together in one lane, so their own items are scanned in order, but nobody promises that customer A finishes before customer B. Partitioned streams work the same way: order inside a lane, parallelism across lanes.

A single total order needs a single queue and a single writer, which caps throughput at what one machine can do. A partitioned log splits the stream into many independent ordered logs. Each event is routed to a partition by hashing a key, so every event with the same key lands in the same partition and stays in order. Each partition is read by one consumer at a time, and more partitions mean more consumers working in parallel.

Here is the same stream in three shapes: one lane, many lanes split by key, and a second reader that replays the retained log without disturbing the first. Watch where each event for one order ends up.

The key is the thing whose events must stay in order: an order id, an account id, a conversation id. Choose the smallest entity that still protects the rule. Too coarse (everything keyed by country) and one partition does all the work. A key that is too fine loses ordering you needed. Document the promise in the API: 'events are ordered per `order_id`, not across orders.'

A hot key is the limit of this design. If one account produces most of the traffic, its partition is overloaded while the others sit idle, and adding partitions does not help because that key still maps to one. The usual answers are to split the key when ordering within a smaller group is enough (for example, per account and per device), or to give the very hot keys their own stream.

Brokers retry until they are sure a message was delivered, so a consumer will see duplicates, and a retry can reorder messages. The consumer's defense is a per-key sequence number. It applies an event only if it is the next one expected, ignores anything at or below the last applied number, and holds back anything that arrives early until the gap is filled. Press Run to see all three.

The second 'paid' is dropped as a duplicate. 'Shipped' arrives before 'packed', so it waits (the log says it is holding 4), and when 'packed' arrives both are applied in the right order. Real consumers also bound the wait: if the missing event never comes, they ask for a replay of that range instead of holding forever.

Because the stream is a retained log with offsets, you can read it again. That is how you build a new consumer from history, recover after a bug, or rebuild a corrupted view. Replay is a feature of the API, not an afterthought: a request such as `POST /subscriptions/{id}/replays` with a start offset or time and an end returns an operation the client can poll, so a large backfill does not hold a connection open.

• -Replay in its own consumer group and rate limit, so it never starves live consumers of throughput.
• -Mark replayed events so a consumer can tell them apart, and disable or make idempotent any side effect such as sending an email.
• -Retention is finite. For history older than the log, rebuild from a snapshot of the state and replay only the events after it.
• -Record what was replayed, from where and by whom, because a replay changes data downstream.