A rate limit says “this caller may make N requests per time period”. It protects shared capacity from one heavy user, keeps costs predictable, and slows down abuse like credential stuffing. In an interview, “add a rate limiter” is easy to say. The questions that follow are where the design lives: limit whom, using which time model, enforced where, and what does the caller see when they hit it?
Who is being limited?
The identity you count against decides whether the limit is fair.
- API key or OAuth client. Best for public APIs: each integration has its own budget, and limits can follow a pricing tier.
- User id. Good for logged-in apps, so one user with three devices shares one budget.
- Tenant. For B2B products, a per-tenant limit stops one customer’s batch job from hurting every other customer, often with per-user limits inside it.
- IP address. The only option for unauthenticated traffic, and the least fair. A corporate network or a mobile carrier’s NAT can put thousands of people behind one IPv4 address, so a limit of 100 requests a minute per IP may block a whole office. With IPv6, a single user can rotate through addresses in a /64, so limiting per full address does nothing; limit per prefix instead.
Endpoints often need more than one. A login endpoint should limit attempts per account (to stop guessing one person’s password from many IPs) and per IP (to stop one machine trying many accounts).
A common misconception is that rate limiting and overload protection are the same thing. A rate limit is a per-caller policy. Your service can still be overloaded when every caller is within their limit, because a thousand well-behaved callers can exceed capacity together. You still need load shedding for that.
Four algorithms, one burst
The algorithms differ in how they treat time. Here’s a burst designed to expose them: the limit is 100 requests per minute, and a client sends 100 requests between t = 59.0 and 59.9 seconds, then another 100 between t = 60.0 and 60.9. Each limiter is a few lines of Node; here’s the fixed window and the token bucket:
function fixedWindow() {
const counts = new Map();
return (t) => { const w = Math.floor(t / WINDOW); const c = counts.get(w) ?? 0; if (c >= LIMIT) return false; counts.set(w, c + 1); return true; };
}
function tokenBucket(capacity = LIMIT, refillPerSec = LIMIT / WINDOW) {
let tokens = capacity, last = 0;
return (t) => { tokens = Math.min(capacity, tokens + (t - last) * refillPerSec); last = t; if (tokens < 1) return false; tokens -= 1; return true; };
}limit 100 per minute; 200 requests sent between t=59.0 s and t=60.9 s:
fixed window allowed 200
sliding log allowed 100
sliding window counter allowed 102
token bucket (100, 1.67/s) allowed 103Fixed window counts requests per calendar minute. The first 100 fall in minute 0 and the next 100 in minute 1, so all 200 pass within two seconds: double the intended rate. That boundary burst is the classic weakness to mention, and it matters when the limit protects something fragile.
Sliding log stores the timestamp of every accepted request and counts those in the last 60 seconds. It’s exact (100 allowed) but stores one entry per request, which gets expensive for high limits.
Sliding window counter keeps only two counters, the current and previous fixed windows, and estimates the rolling count by weighting the previous window by how much of it still overlaps. It allowed 102, close to exact, with constant memory. Cloudflare has written about using this approach at scale.
Token bucket holds up to 100 tokens and refills at 100 per minute (1.67 a second). Each request takes a token. The full bucket allowed the first 100 instantly, then only what refilled in the next two seconds, 103 in total.
Bursts are a feature
The token bucket has two knobs, and they mean different things. Capacity is the largest burst you’ll accept. Refill rate is the sustained rate. Real clients are bursty: a page load fires 20 API calls at once, then nothing for a minute. A bucket with capacity 20 and refill of 1 a second lets that page load through instantly while still capping a scraper at 60 a minute.
Over a longer run, the two models converge:
one client sending 10 requests/s for 10 minutes (limit 100/min):
fixed window allowed 1000 of 6000 (100/min average)
token bucket (100, 1.67/s) allowed 1099 of 6000 (110/min average)The token bucket allowed its initial burst of 100 and then exactly the refill rate. The extra 99 is the burst allowance, spent once. If that initial burst is too generous for what you’re protecting, lower the capacity rather than the rate.
Enforcing across many gateways
A single process can keep counters in memory. A real API runs on many gateway nodes, and a client’s requests land on different ones. There are two broad approaches.
A central counter, usually Redis. Each request runs an atomic script that updates the client’s bucket and returns allow or deny. Atomicity matters: a GET followed by a SET from two gateways at once lets both through. A Lua script or a single INCR with expiry keeps it to one atomic step (see rate limiting in Redis with Lua for tested scripts). The cost is a network round trip, usually well under a millisecond in the same zone, plus a decision about what happens when Redis is unreachable. Most public APIs fail open (allow traffic) rather than reject every request because the limiter is down.
Local limits on each node. The tempting shortcut is to split a global limit evenly: with 10 nodes and a 100 per second limit, each node allows 10 per second per client. It only works when every client’s traffic is spread evenly across nodes, which long-lived connections and sticky routing prevent:
global limit 100/s split as 10/s on each of 10 gateway nodes; client sends 150/s:
traffic spread over 10 nodes: 100/s allowed
traffic spread over 5 nodes: 50/s allowed
traffic spread over 2 nodes: 20/s allowedA paying client entitled to 100 a second gets 20 because its HTTP/2 connections happen to land on two nodes. Hybrids exist: nodes keep local buckets and periodically sync usage to a central store, accepting a small overshoot in exchange for no per-request network hop. Whether that overshoot is acceptable depends on what the limit protects. Billing quotas need precision; abuse protection rarely does.
Telling clients what happened
When a request is limited, return 429 Too Many Requests with a Retry-After header saying how many seconds to wait. Many APIs also send remaining-budget headers on every response (GitHub uses X-RateLimit-Limit, X-RateLimit-Remaining and X-RateLimit-Reset, and an IETF draft standardises RateLimit fields), so well-behaved clients can slow down before hitting the wall. Clients should honour Retry-After and add random jitter, or every limited client retries at the same instant and gets limited again.
Don’t return 503 for rate limiting. It tells clients and monitors that your service is failing, when in fact it’s doing exactly what it should.
Interview practice: tiers on a public API
“Design rate limiting for a public API with a free tier (60 requests a minute) and a paid tier (1,000 a minute, with short bursts allowed). It runs on 20 gateway nodes in two regions.”
Answer
I’d limit per API key, since that maps to the customer and their tier, and add a per-IP limit on unauthenticated endpoints like signup and token exchange. For the algorithm I’d use a token bucket: free tier with capacity 20 and refill of 1 per second, paid tier with capacity 200 and refill of about 16.7 per second. Capacity gives paid customers the bursts they asked for without raising their sustained rate.
For enforcement I’d keep buckets in a Redis cluster in each region, updated by an atomic Lua script per request, keyed by API key. Each region enforces independently, so a customer splitting traffic across regions could get up to twice their limit. For a rate limit that’s an acceptable trade against putting a cross-region round trip on every request; if billing depends on exact counts, I’d meter usage asynchronously from logs instead. If Redis is unavailable, gateways fail open with a conservative local limit, so an outage of the limiter doesn’t become an outage of the API. Limited requests get 429 with Retry-After, every response carries remaining-budget headers, and I’d keep a separate load-shedding layer, since all customers being within their limits doesn’t guarantee the backend can take the sum.