Advanced Throttling Strategies
You'll learn to
- -Apply token buckets at gateway scale, with the bucket state in a shared store
- -Give different plans different limits, and charge expensive endpoints more tokens
- -Show with numbers why local counters on several gateways multiply the real limit
The earlier rate limiting chapter covered how to tell clients about a limit using headers. This chapter covers the algorithms and the policy behind the limit itself, at the scale of a gateway sitting in front of every request to every service. It uses the same token bucket and sliding window ideas as the LLD rate limiter, now with plans and shared state.
An Analogy: The Arcade Tokens
At an arcade, each visitor gets a card with tokens that slowly refill. A simple game costs one token. A big ride costs twenty. A premium pass gives a bigger card that refills faster. If the arcade has two entrances and each keeps its own tally, a visitor can walk in through both and play twice as much as their pass allows. That last point is the heart of the chapter.
Token Bucket at Gateway Scale
A token bucket per client, or per API key, allows short bursts up to the bucket's size while holding a steady average rate over time. At gateway scale there may be millions of distinct clients, so the bucket state normally lives in a shared, fast store such as Redis (an in-memory data store) and not in one process's memory. Then the limit is enforced the same way whichever gateway instance handles a given request.
Tiered Limits by Plan
{
"free_tier": {"requests_per_minute": 60, "burst_capacity": 10},
"pro_tier": {"requests_per_minute": 600, "burst_capacity": 100},
"enterprise_tier": {"requests_per_minute": 6000, "burst_capacity": 1000}
}One uniform limit treats a free-tier hobbyist and an enterprise customer with a negotiated contract identically, which is rarely the business intent. Tiered limits, keyed by plan or identity as established during authentication at the gateway, let the policy reflect what each client has paid for or been granted.
Per-Endpoint Limits, Not Just Per-Client
Not every endpoint costs the same. A cheap GET /users/42 lookup and an expensive POST /reports/generate that starts real computation should not necessarily share one budget. Charging more tokens for expensive operations protects them more tightly without restricting a client's access to cheap, high-volume endpoints just because it used one expensive one. The demo shows plans, endpoint costs and the two-gateway problem. In the last two lines, 24 requests alternate between two gateways.
A distributed rate limiter adds real latency to every request, because the shared-state lookup must happen in the request path before the request is let through. Weigh that cost against the coarser but faster alternative of eventually consistent local counters on each gateway instance.
Interview Signal is part of Pro
See a real weak answer next to a real strong one for this exact topic.
Quiz is part of Pro
Test what you just read with a short quiz, and bank the XP.
Design Throttle Engine in the API Design Lab's Full System Design act.
Discussion0
Join the Discussion
Sign in to leave comments, reply to others, or like insights.
No comments yet. Be the first to start the thread!