Idempotency & Exactly-Once Illusions

Here is a situation that happens in every networked system. A client sends a payment request and waits. Five seconds pass with no reply. Did the server never receive the request, or did it process the payment and the reply got lost on the way back? The client cannot tell. If it retries, it risks charging the customer twice; if it does not, it risks not charging at all. This chapter is about how to make that uncertainty harmless.

In the asynchronous module, at-least-once delivery was named as the practical default for queues, and the same reality applies to any request that crosses a network. A timeout does not say whether the work happened, so a client retrying to be safe may be sending a duplicate of a request that already succeeded. Preventing every duplicate across an unreliable network is not realistic. The answer is to make duplicates harmless.

Pressing the call button of a lift a second time does not summon a second lift. The lift is called, and pressing again changes nothing: the operation is idempotent. Compare a vending machine where each press of the add coin button adds another coin. Pressing it twice is not the same as pressing it once. Networks make people press buttons twice all the time, because they cannot see whether the first press worked, so designs must make the important buttons behave like the lift.

For operations that are not naturally safe to repeat, such as a payment, the standard tool is the idempotency key. The client generates a unique key for each logical operation (this one payment attempt) and sends it with the request. The retry carries the same key:

The server remembers keys it has seen. If the same key arrives again, it does not charge again; it returns the original result. Retrying a payment can then never double-charge. The diagram shows the check:

• -Look the key up before doing anything. If it exists and the work is finished, return the stored response.
• -Record the key and the effect in the same database transaction, so a crash cannot leave the customer charged with no record, which would let a retry charge them again.
• -Make the key unique in the store, so two copies of the same request arriving at the same moment cannot both pass the check: one proceeds, and the other waits for its result or receives a 409 Conflict.
• -Reject a reused key that comes with a different request body, which is a client bug rather than a retry.
• -Keep keys for a limited time, such as 24 hours, which is longer than any sensible retry window.

Many operations do not need a key because repeating them changes nothing more. Setting a value is idempotent: set the user's email to a@b.com gives the same result however many times it is applied. Adding to a value is not: add 10 to the balance gives a different result every time. The same goes for append a comment and send an email. Where you can, design operations as set-to-a-value rather than change-by-an-amount; and where you cannot, use a key. PUT and DELETE in an API are idempotent by definition, and POST usually needs help.