Should a rate limiter use token bucket or sliding window?

Token bucket if you want to allow occasional bursts above the average rate (typical for APIs). Sliding window counter if you need a smoother, more predictable enforcement with no burst allowance. Fixed window is the simplest to build but allows up to 2x the intended rate right at the window boundary — usually not worth the simplicity trade-off once you know that.

Why is a distributed rate limiter harder than an in-memory counter?

Because a client's requests get load-balanced across many stateless API servers. If each server keeps its own counter, the effective limit becomes the configured limit multiplied by the number of servers a client happens to hit — the count has to live somewhere shared, which is what makes this a distributed systems problem rather than a one-liner.

What happens if the rate limiter itself goes down?

This is a real product decision, not just an engineering detail: fail open (let requests through, risking abuse) or fail closed (block everything, risking a self-inflicted outage). Most production systems fail open for read-heavy, low-risk endpoints and fail closed for anything expensive or abuse-prone, like login or payment endpoints.

What's the detail that separates a senior answer from a mid-level one here?

Recognizing that checking the limit on every single request against a shared store adds real latency at scale, and that the fix isn't 'add more Redis' — it's reducing how often you actually need to ask, by pushing an approximate local check to the edge and only paying for a strict shared check when a client is close to their limit.

Distributed Rate Limiter System Design

An API that doesn't rate limit its clients eventually gets taken down by one of them — not always maliciously, sometimes it's just a retry loop with a bug. This post walks through designing a rate limiter the way you'd talk through it in an interview: what looks obvious at first, where that breaks at scale, and what actually holds up.

Requirements and Goals of the System

Functional Requirements

Enforce a limit like "100 requests per user per minute" per API endpoint
Support different limits per client tier (free vs. paid) and per endpoint (a search endpoint and a checkout endpoint shouldn't share a limit)
Reject over-limit requests clearly (HTTP 429) with a Retry-After hint
Let the platform team change limits without an engineering deploy

Below the line (out of scope)

Issuing API keys or authenticating the client — we assume a client_id already exists by the time a request reaches us
Billing or usage metering for paid tiers — a related but separate system that happens to read similar counters
Network/edge-layer DDoS protection — a different problem operating below the application layer

Non-Functional Requirements

Scale: on the order of 1M requests/second platform-wide, with a limit check on nearly every request
Latency: the check sits on the hot path of every API call, so it needs to add close to zero — a few milliseconds at most, not tens
Consistency: a client shouldn't be able to exceed their limit by more than a small margin, even when their requests land on different servers each time
Availability: if the limiter itself is unavailable, the system needs an explicit answer for whether it fails open (allow) or fails closed (block) — this is a product decision, not a default

Capacity Estimation and Constraints

Traffic: ~1M requests/second platform-wide, and if the vast majority of clients aren't near their limit, the actual number of strict checks against the shared store is a small fraction of that — the local-approximation design below exists precisely to keep that fraction small.

Counter storage: counters are keyed per {client_id}:{endpoint}:{window}, so the number of live keys at any moment is roughly active_clients × active_endpoints, not one per request. For, say, 50 million active clients checked against an average of 3 endpoints each, that's ~150 million live keys — each a tiny integer with a TTL, easily a few tens of GB in Redis, comfortably shardable across a modest cluster.

Memory, not disk: this is the detail worth stating plainly — a rate limiter is a memory-bound problem, not a storage-bound one. Every key is small and short-lived (a TTL equal to the window length), so the system's real constraint is Redis cluster memory and ops/second, not disk capacity.

System APIs

http

POST /internal/ratelimit/check
{ "client_id", "endpoint", "cost": 1 }
→ { "allowed": true, "remaining": 42, "reset_at": "2026-07-28T10:01:00Z" }

PUT /admin/policies/{endpoint}
{ "scope": "per_user" | "per_ip" | "per_key", "limit": 100, "window_seconds": 60, "tier": "free" | "paid" }

The check endpoint is called by the API gateway on every request, in-process or over a fast internal call — never by the client directly.

High-Level Design

The core requirement to design around: enforce a shared limit accurately across many stateless API servers, without that shared state becoming the bottleneck.

Bad solution: each API server keeps its own in-memory counter per client.

This is free — zero latency, no extra infrastructure — and it's also wrong the moment there's more than one server. A client's requests get load-balanced across all of them, so a "100 requests/minute" limit silently becomes "100 × number of servers" in practice, since no single server ever sees the client's full request volume.

Good solution: move the counter into Redis, shared by every API server. Each request does an atomic increment against a key like ratelimit:{client_id}:{endpoint}:{window} and compares it to the policy's limit.

This fixes correctness — there's now exactly one count, not one per server — but introduces two new problems. First, every single request now pays a network round trip to Redis before it can proceed, which eats directly into the latency budget. Second, Redis is now a shared dependency for the entire platform's traffic, and a hot client (or a hot shard) can create a real bottleneck.

Great solution: keep Redis as the single source of truth, but be deliberate about two things: how the check-and-increment happens, and how often it has to happen at all.

For the "how": do the read-check-increment as a single Lua script executed atomically inside Redis, so it's one round trip instead of a read followed by a conditional write — avoiding a race where two concurrent requests both read "under limit" before either increments. Shard the Redis layer by client_id (consistent hashing) so no single node has to absorb the whole platform's traffic, and no single hot client can degrade every other client's latency.

For the "how often": most requests, for most clients, aren't anywhere near their limit. The gateway can keep a small local approximation — an in-memory token count that's periodically reconciled against Redis rather than checked on every request — and only fall through to a strict, synchronous Redis check when a client is getting close to their actual limit. This turns the common case (nowhere near the limit) into a local, sub-millisecond check, and reserves the network round trip for the case that actually needs precision.

Database Design

Client — whoever is being limited: a user, an API key, or an IP address, depending on scope
Endpoint — the API route a policy applies to; different endpoints reasonably have very different limits
RateLimitPolicy — the actual rule: scope (per-user/per-IP/per-key), max requests, window length, which tier it applies to
Counter — the live count of requests a specific client has made against a specific policy in the current window

The one modeling choice worth calling out: a Counter is keyed by {client_id}:{endpoint}:{window_start}, not a single running total per client. That means old windows simply expire off a TTL instead of needing a cleanup job, and it's what makes fixed and sliding-window algorithms implementable as plain key-value operations instead of a bespoke data structure. RateLimitPolicy is deliberately a separate entity from Counter so the platform team can change a limit without touching, resetting, or migrating any in-flight counts.

Detailed Component Design

Token bucket, sliding window, or fixed window — which one, and why?

Fixed window (reset the counter every N seconds on the clock) is the simplest to build and reason about, but it has a real flaw: a client can send their full limit right at the end of one window and again right at the start of the next, getting roughly 2x their intended rate in a short burst around the boundary. Sliding window counter fixes this by weighting the previous window's count proportionally to how much of it overlaps the current moment — almost as accurate as tracking every individual request timestamp, at a fraction of the memory. Token bucket is the right choice when bursts are actually desirable — a client that's been idle should be able to use up a saved allowance quickly, then settle into the steady-state rate — which is why it's the common default for public APIs.

How do we avoid a thundering herd right at the window boundary?

This is the fixed-window problem again, from the client's side: if a burst of requests gets rejected right up until a window resets, the same clients often retry at nearly the same instant the window rolls over, producing a spike synchronized to the wall clock. A sliding-window counter smooths this out structurally, since there's no single instant where the count resets to zero. On top of the algorithm choice, adding a small amount of random jitter to the Retry-After value discourages every rejected client from retrying at exactly the same moment.

How do multiple limits (per-user, per-IP, per-endpoint) apply to the same request?

In practice a single request is often checked against more than one policy at once — a per-user quota and a coarser per-IP abuse limit, say. The cheap move is to check whichever policy is most likely to reject first (usually the tightest, most abuse-oriented one) so a request that's going to be rejected anyway fails fast, before paying for a Redis round trip against every applicable policy.

Fault Tolerance and Replication

Failure	Impact	Response
Redis shard unreachable	Requests against that shard's clients can't be strictly checked	Explicit, endpoint-dependent choice: fail open (allow through, log for later review) for low-risk endpoints, fail closed for expensive or abuse-prone ones like login
Policy store unreachable	Rate limiter can't fetch a new/updated policy	The rate limiter service caches the last-known-good policy set locally in memory, refreshed periodically — a brief outage just means policy changes are delayed, not that limiting stops working
One Redis node overloaded by a hot client	Latency degrades for every other client sharing that node	Sharding by `client_id` bounds the blast radius to that shard; a sufficiently large single client may need its own dedicated shard
Rate limiter service instance crash	In-flight local-approximation state for that instance is lost	Stateless design — the instance simply restarts and re-syncs its local approximation from Redis; no client-visible correctness issue since Redis remains the source of truth

Redis itself runs with replicas per shard so a single node failure triggers a fast replica promotion rather than a shard-wide outage — acceptable here because a counter that's a few seconds stale after failover is still well within the tolerance of a system that's already an approximate, soft limit at the margin.

Extended Requirements

Observability & Monitoring

Signal	Why it matters	Alert on
Rate-limiter check latency (P99)	This sits on every request's hot path	Sustained P99 above a few milliseconds
Reject rate, overall and per-endpoint	A sudden spike often means a client bug (retry loop), not real abuse	Sharp spike vs. 24h baseline for one client/endpoint
Redis shard hot-keying	One oversized client can degrade the shard everyone else shares	Any single shard's ops/sec far above the cluster average
Local-cache vs. Redis check ratio	Confirms the approximate local check is actually absorbing most traffic	Ratio drops sharply (means most requests are now near their limit)

Security & Compliance

The limiter only works if client_id can't be trivially spoofed — if it's derived from a header the caller controls rather than an authenticated identity or a properly-derived IP, a rate limit is just a suggestion. Multi-tenant fairness is the other constraint worth naming explicitly: sharding by client_id isn't just a performance optimization, it's what stops one noisy or misbehaving tenant from degrading Redis latency for every other tenant sharing the cluster.

Leveled Expectations

Mid-level: Identifies that per-server in-memory counters break under load balancing, and lands on a shared Redis counter with INCR and a TTL.

Senior: Independently raises the fixed-window boundary problem and picks sliding window or token bucket accordingly; reasons about Redis as a shared bottleneck and proposes sharding by client; treats the atomic check-and-increment (not just "use Redis") as the actual correctness requirement.

Staff+: Pushes on the latency budget itself — proposes reducing Redis round trips via local approximation rather than just scaling Redis harder; treats fail-open vs. fail-closed as an explicit, endpoint-dependent product decision; reasons about cross-region deployments, where strict global consistency on a counter usually isn't worth the latency cost given the limit is inherently a soft target at the margin.