Skip to content
API Design Learn/API Gateways & Aggregation
Browsing as a guest. Sign in to save your progress and earn XP as you complete chapters.

Advanced Throttling Strategies

4 min read

You'll learn to

  • -Apply token buckets at gateway scale, with the bucket state in a shared store
  • -Give different plans different limits, and charge expensive endpoints more tokens
  • -Show with numbers why local counters on several gateways multiply the real limit

The earlier rate limiting chapter covered how to tell clients about a limit using headers. This chapter covers the algorithms and the policy behind the limit itself, at the scale of a gateway sitting in front of every request to every service. It uses the same token bucket and sliding window ideas as the LLD rate limiter, now with plans and shared state.

An Analogy: The Arcade Tokens

At an arcade, each visitor gets a card with tokens that slowly refill. A simple game costs one token. A big ride costs twenty. A premium pass gives a bigger card that refills faster. If the arcade has two entrances and each keeps its own tally, a visitor can walk in through both and play twice as much as their pass allows. That last point is the heart of the chapter.

Token Bucket at Gateway Scale

A token bucket per client, or per API key, allows short bursts up to the bucket's size while holding a steady average rate over time. At gateway scale there may be millions of distinct clients, so the bucket state normally lives in a shared, fast store such as Redis (an in-memory data store) and not in one process's memory. Then the limit is enforced the same way whichever gateway instance handles a given request.

Tiered Limits by Plan

Different limits for different client tiers, not one blanket policy
{
  "free_tier":    {"requests_per_minute": 60,   "burst_capacity": 10},
  "pro_tier":     {"requests_per_minute": 600,  "burst_capacity": 100},
  "enterprise_tier": {"requests_per_minute": 6000, "burst_capacity": 1000}
}

One uniform limit treats a free-tier hobbyist and an enterprise customer with a negotiated contract identically, which is rarely the business intent. Tiered limits, keyed by plan or identity as established during authentication at the gateway, let the policy reflect what each client has paid for or been granted.

Per-Endpoint Limits, Not Just Per-Client

Not every endpoint costs the same. A cheap GET /users/42 lookup and an expensive POST /reports/generate that starts real computation should not necessarily share one budget. Charging more tokens for expensive operations protects them more tightly without restricting a client's access to cheap, high-volume endpoints just because it used one expensive one. The demo shows plans, endpoint costs and the two-gateway problem. In the last two lines, 24 requests alternate between two gateways.

Press Run: plans differ, a report costs 20 tokens, and local counters let 20 requests through instead of 10

A distributed rate limiter adds real latency to every request, because the shared-state lookup must happen in the request path before the request is let through. Weigh that cost against the coarser but faster alternative of eventually consistent local counters on each gateway instance.

Interview Signal is part of Pro

See a real weak answer next to a real strong one for this exact topic.

Quiz is part of Pro

Test what you just read with a short quiz, and bank the XP.

Ready to Build This?

Design Throttle Engine in the API Design Lab's Full System Design act.

Discussion0

Join the Discussion

Sign in to leave comments, reply to others, or like insights.

Sign In to ScaleDojo

No comments yet. Be the first to start the thread!