Rate Limiting, Actually Explained

A rate limiter is a gatekeeper: it looks at how many requests a client has already made recently, and decides whether this one gets through. That's the entire idea — a counter and a comparison. If you understand that sentence, you understand rate limiting conceptually.

Everything below this point is about keeping that gatekeeper correct and fast once there isn't just one gatekeeper — there are thousands of them, running on different machines, all supposedly enforcing the same rule for the same client at the same time.

Before picking an algorithm, it's worth being explicit about what this system is actually on the hook for — the same way you'd scope it out loud in an interview before drawing a single box.

One constraint shapes everything downstream: this sits on the hot path of every request, so it has to add close to zero latency and keep working even when part of its own infrastructure is degraded. That constraint is why this guide keeps coming back to “check less often” and “fail open vs. fail closed” instead of just “add more Redis.”

Numbers here are explicit back-of-envelope estimates, not measured production values — but they're what determines whether “just use Redis” actually holds up, and why this guide keeps coming back to checking less often instead of scaling Redis harder.

Before drilling into any one piece, here's the system end-to-end: a request hits the gateway, gets a fast local check, and only pays for a strict Redis round trip when it's actually close to its limit. The API contract and data model below are what every later section refines — nothing past this point replaces this picture, it only explains why each piece of it exists.

What the client actually sees is usually headers, not a body a human reads — the same shape Stripe, GitHub, and most public APIs use, so an HTTP client can back off automatically without parsing anything custom:

RateLimitPolicy is deliberately a separate entity from Counter so the platform team can change a limit without touching, resetting, or migrating any in-flight counts.

The fix is to stop counting locally and count in one shared place instead — usually Redis. Every request does an atomic increment against a key like ratelimit:{client_id}:{endpoint}:{window} and compares the result to the policy's limit.

That fixes correctness — there's exactly one count now, not one per server. It also creates two new problems. Every request now pays a network round trip to Redis before it can proceed, which eats into a latency budget that's supposed to be close to zero. And Redis is now a dependency the whole platform shares, so a busy client — or a busy shard — can slow things down for everyone else hitting that same node. Getting the count right was the easy part.

Two separate moves get this to production scale, and they answer two different questions — refining the architecture already shown above, not replacing it.

How the check happens: run the check-and-increment as a single Lua script executed inside Redis — one round trip instead of a read followed by a conditional write. That closes a real race where two concurrent requests could both read “under limit” before either one increments. Shard the Redis layer by client_id so no single node absorbs the whole platform's traffic.

How often it has to happen: most requests, for most clients, aren't anywhere near their limit. Keep a small local approximation at the edge — an in-memory count, reconciled against Redis periodically — and only fall through to a strict, synchronous Redis check when a client is getting close to their actual limit. The common case becomes a local, sub-millisecond check; the network round trip is reserved for the case that actually needs precision.

At the ~1M req/s scale sized earlier, if most clients aren't near their limit, this split keeps the volume of strict Redis checks to a small fraction of that total — the local check absorbs the rest.

Once the counter lives somewhere shared and the checks are fast, one real design decision is left: how do you decide “over the limit”? Three common answers behave differently in a way that matters. Pick one below, pick a traffic pattern, and press run.

Fixed window's boundary bug isn't something a single-server test would ever catch — it only shows up once real traffic clusters around a reset, which is exactly what the “burst at the boundary” pattern above just showed you.

In practice a single request is usually checked against more than one policy at once — a per-user quota and a coarser per-IP abuse limit, say. The cheap move: check whichever policy is most likely to reject first, usually the tightest, most abuse-oriented one, so a request that's going to be rejected anyway fails fast — before paying for a Redis round trip against every applicable policy.

Redis itself runs with replicas per shard, so a single node failure triggers a fast replica promotion rather than a shard-wide outage — acceptable here because a counter that's a few seconds stale after failover is still well within the tolerance of a system that's already an approximate, soft limit at the margin.

The limiter only works if client_id can't be trivially spoofed — if it's derived from a header the caller controls rather than an authenticated identity or a properly-derived IP, a rate limit is just a suggestion. Sharding by client_id isn't only a performance optimization, either — it's what stops one noisy tenant from degrading Redis latency for every other tenant sharing the cluster.