Why Rate Limiting Actually Matters
GitHub has been hit by some of the largest DDoS attacks ever recorded. In 2015, sustained traffic (widely attributed to China's "Great Cannon" injection attack) hammered the site for days. In 2018, GitHub took a 1.35 Tbps hit from a memcached amplification attack, at the time the largest DDoS ever measured. Neither incident was stopped by API rate limiting alone, they were absorbed through a combination of upstream DDoS scrubbing, Akamai's edge network, and traffic shaping. But that's exactly the point: without some layer standing between raw internet traffic and your application servers, every one of those malicious requests gets processed as if it were legitimate, and your own infrastructure becomes the weapon used against you.
Rate limiting is one piece of that defense. It won't stop a multi-terabit volumetric attack by itself, but it's what keeps a single abusive client, a broken retry loop, or a scraper script from taking down a service that would otherwise handle normal traffic just fine. Think of it as the layer that separates "one client is being unreasonable" from "the whole system falls over."
Rate Limiting Algorithms Compared
There are five algorithms you'll run into repeatedly, and each makes a different tradeoff between accuracy, memory usage, and how much burst traffic it tolerates.
Algorithm Comparison:
Algorithm Burst Accuracy Memory Complexity Best For
----------- ------ ---------- -------- ---------- ----------------------
Fixed Window High Low O(1) Simple Quick implementation
Sliding Log None Exact O(n) Medium Audit/billing
Sliding Window Low High O(1) Medium General purpose
Token Bucket Medium High O(1) Medium API gateways (common)
Leaky Bucket None High O(1) Medium Smooth output rate
Fixed Window Problem:
Window 1 (12:00-12:01) Window 2 (12:01-12:02)
|............[50 requests]| |[50 requests]............|
^100 requests in 2 seconds^
Limit: 50/minute, but 100 requests in a 2-second burst!
Sliding Window Fix:
At 12:00:45 (75% through current window):
count = (0.25 * prev_window_count) + current_window_count
If prev=40, current=30: count = 10 + 30 = 40 (under limit)The fixed window bug above trips up a lot of people the first time they see it. It looks correct on paper (50 requests per minute, evenly enforced) until you realize nothing stops all 50 from landing in the last second of one window and all 50 more in the first second of the next. Sliding window fixes this with a weighted average across the two most recent windows, which is why it's the default choice for most general-purpose APIs.
Token Bucket, Deep Dive
If you've read up on low-level design fundamentals, you've probably seen token bucket come up as the go-to example for rate limiting interviews, and for good reason. It allows short bursts while still enforcing a sustained average rate, which matches how real client traffic actually behaves.
Token Bucket (Most Popular for APIs):
Bucket capacity: 10 tokens (max burst)
Refill rate: 2 tokens/second (sustained rate)
Time 0s: [**********] 10 tokens (full)
Burst! [ ] 0 tokens (10 requests at once - allowed!)
Time 1s: [** ] 2 tokens (refilled 2)
Time 2s: [**** ] 4 tokens (refilled 2 more)
Request: [*** ] 3 tokens (1 consumed)
Time 3s: [***** ] 5 tokens (refilled 2)
...
Time 5s: [*********_] 9 tokens (capped at 10, no overflow)
Implementation (Redis + Lua for atomicity):
-- Redis Lua script (atomic token bucket)
local key = KEYS[1]
local capacity = tonumber(ARGV[1]) -- 10
local refill_rate = tonumber(ARGV[2]) -- 2 tokens/sec
local now = tonumber(ARGV[3])
local bucket = redis.call('HMGET', key, 'tokens', 'last_refill')
local tokens = tonumber(bucket[1]) or capacity
local last = tonumber(bucket[2]) or now
-- Refill tokens based on elapsed time
local elapsed = now - last
tokens = math.min(capacity, tokens + elapsed * refill_rate)
if tokens >= 1 then
tokens = tokens - 1
redis.call('HMSET', key, 'tokens', tokens, 'last_refill', now)
redis.call('EXPIRE', key, 300)
return 1 -- ALLOWED
else
return 0 -- REJECTED (429 Too Many Requests)
endThe Lua script matters more than it looks. Rate limit counters get read and written by many requests concurrently, often from multiple gateway instances, and if you do the read-then-write as two separate Redis calls you get a race condition where two requests both see 1 token available and both get allowed through. Running the whole check as one atomic Lua script closes that gap. This is the same kind of consideration that comes up when picking a cache placement strategy, if you want more on how Redis behaves under concurrent access and where it fits in your stack, the piece on caching strategies and placement trade-offs covers a lot of the same ground.
Multi-Layer Rate Limiting
In production, rate limiting isn't one check, it's several, each tuned to what that layer is actually protecting.
Production Rate Limiting Architecture:
Internet
|
[CDN/Edge] --- Layer 1: IP-based, 1000 req/min
| (Cloudflare, AWS WAF)
| Stops DDoS before it hits your servers
|
[API Gateway] --- Layer 2: API key-based, tiered
| Free: 100 req/hour
| Pro: 1000 req/hour
| Enterprise: 10000 req/hour
|
[Service] --- Layer 3: Per-endpoint
| POST /login: 5 req/min (brute force prevention)
| GET /search: 30 req/min (expensive query)
| GET /status: 600 req/min (cheap health check)
|
[Database] --- Layer 4: Connection-level
max_connections = 100 (last resort protection)
Rate Limit by:
- IP address: unauthenticated endpoints (beware NAT - 1 IP = 1000 users)
- API key: public APIs (Stripe, GitHub, Twitter)
- User ID: authenticated endpoints (fairest)
- Endpoint: expensive ops get stricter limits
- Global: total system capacity protectionLayer 2 is usually where the interesting design decisions live, since it's where your API gateway enforces per-key or per-tier quotas. If you're deciding how much logic belongs in the gateway versus the service itself, the breakdown in API Gateway, BFF, and multi-protocol architecture is worth reading alongside this, since rate limiting is usually one of several cross-cutting concerns (auth, routing, protocol translation) that gateway sits in front of.
Worth calling out: IP-based limiting at the edge is a blunt instrument. Behind a corporate NAT or a mobile carrier's shared IP pool, one IP address can represent hundreds or thousands of real users, so a limit that's reasonable for