Exponential Backoff & Jitter

Token bucket controls how fast one side sends requests; exponential backoff controls how a client behaves when a request fails. It shows up constantly, both in Forge and in real systems: network hiccups, a downstream service returning a 503, a lock that is momentarily held elsewhere. Level 26 asks you to implement the retry policy nearly every production HTTP client and RPC framework ships by default.

Imagine a service goes down for two seconds, and a thousand clients all retry immediately the instant a request fails. All thousand retries land on the service in the same instant it starts recovering, overwhelming it again before it can stabilize - the service now spends all its capacity handling retries instead of real traffic, so it looks like it never recovered. Retrying is necessary, but retrying immediately and identically is the actual cause of many real production outages lasting far longer than the original blip.

The first half of the fix is simple: instead of retrying immediately, wait, and make each subsequent wait longer than the last - typically doubling, so 1s, 2s, 4s, 8s and so on up to some cap. This spreads retries out over time instead of concentrating them all in the same instant, and it backs off harder the more consecutive failures there have been, which matches the intuition that a service down for a while probably needs more time to recover, not less.

Doubling alone is not enough, because every client computing the exact same sequence (1s, 2s, 4s...) from the exact same failure event still retries in lockstep with every other client - you have just moved the thundering herd from t=0 to t=1, t=2, t=4. Jitter breaks that synchronization by adding randomness to each wait, so retries from different clients spread out across a range instead of landing at the same instant. A common version, full jitter, picks a wait uniformly between zero and the computed backoff value rather than using the backoff value exactly.