Advanced Throttling Strategies

The basic rate-limiting chapter earlier in this course covered communicating limits to clients via headers; this chapter covers the algorithms and policy design behind the limit itself, at the scale of an API gateway sitting in front of every request to every backend service - the same token bucket and sliding window algorithms from LLD Fundamentals' rate limiter case study, now applied at gateway scale with tiered policies.

A token bucket per client (or per API key) allows short bursts up to the bucket's capacity while enforcing a steady average rate over time - the same algorithm from the LLD Fundamentals case study, but now the gateway needs to track buckets for potentially millions of distinct clients, which typically means the bucket state lives in a shared, fast store (like Redis) rather than in a single process's memory, so the limit is enforced consistently regardless of which gateway instance actually handles a given request.

A single uniform rate limit across every client treats a free-tier hobbyist and an enterprise customer with a negotiated contract identically, which is rarely the actual business intent - tiered limits, keyed by the client's plan or identity (established during authentication at the gateway), let the throttling policy reflect what each client has actually paid for or been granted, rather than a one-size-fits-all number.

Not every endpoint costs the same to serve - a cheap `GET /users/42` lookup and an expensive `POST /reports/generate` that kicks off real computation shouldn't necessarily share the same budget. Layering per-endpoint (or per-endpoint-category) limits on top of per-client limits lets the gateway protect specifically expensive operations more tightly, without needlessly restricting a client's access to cheap, high-volume endpoints just because they've been using an expensive one heavily.