Design an LLM Gateway: Token Limits, Fair Share and Failover Across Providers

Your API gateway counts requests. Your model providers count tokens. That one unit mismatch is where most of the trouble starts: two requests that look identical to Envoy or nginx can differ a hundredfold in what they cost and how much of a provider's per-minute quota they consume, and you only learn the second number after the response has finished streaming.

Put a dozen tenants on top of that, two or three providers with their own quotas and their own bad afternoons, and a finance team that wants a hard monthly cap per plan, and "one API in front of several providers" stops being a proxy problem. Below is the design one request at a time, then the parts that break it, then a simulator where you can break it yourself.

This post sits one layer above LLM inference serving, which is about keeping your own GPUs busy. Here you don't own the GPUs. You rent quota from providers and resell it to tenants, and the design is about spending that quota well.

What the gateway owes its callers

Tenants call one endpoint with one key, in one request format. Behind it, the gateway owes them five things a plain reverse proxy doesn't give:

• Survive a provider outage. Providers publish their incidents on status pages (OpenAI, Anthropic); your users shouldn't have to read them.
• Live inside the quotas you rent. Every provider limits you per minute, and the units differ. OpenAI limits requests and tokens per minute (and per day); Anthropic limits requests, input tokens and output tokens per minute separately.
• Share those quotas fairly between tenants who pay different amounts.
• Stop spending at a cap, not report the overspend next month.
• Keep the provider keys out of tenant hands and out of logs.

Open-source gateways ship most of this as configuration now (Envoy AI Gateway advertises provider fallback and token limits per team, app or model; LiteLLM documents provider fallbacks). The configuration is easy to copy. Knowing which defaults are wrong for your traffic is the part worth understanding.

A request, start to finish

Step through the path. Every numbered step is a separate component with its own failure mode, and the arc at the end (step 6) is the one people forget: the real token count only exists after the stream ends.

Two things fall out of this picture. The limiter decides on a number it made up, so the design has to tolerate the estimate being wrong. And the response is a stream, so every decision after step 5 is constrained by what the client has already seen. The next sections take those two in turn.

Counting tokens before they exist

A per-request limit treats a 200-token classification call and a 90,000-token document summary as the same thing. Providers don't, so the gateway can't either.

Input tokens can be estimated before forwarding. Providers do exactly this themselves: Anthropic says its input-token limit is estimated at the start of each request and adjusted during it, and OpenAI's limiter uses an estimate based on the character count of the request. A character-based estimate per model family is good enough for admission. Calling a counting endpoint on the hot path isn't worth the extra round trip, and Anthropic notes its token counts are themselves estimates and that tokenizers change between model generations. An estimate tuned for one provider's tokenizer is wrong for another's, so keep one ratio per target model, not one per gateway.

Output tokens are the hard part, because nobody knows how long the answer will be. The two big providers handle it differently, and the difference is instructive. OpenAI counts a request against your limit as the larger of max_tokens and the estimate, which is conservative. Anthropic counts output tokens as they are produced and says max_tokens doesn't factor in at all.

For a gateway, reserving the full max_tokens sounds safe and starves everyone in practice, because many clients set it to the model maximum out of habit. A better reservation is the smaller of max_tokens and a recent high percentile of that tenant's real output lengths for that model. Then let the bucket go negative when a response runs long. The debt delays the tenant's next request instead of cutting off the current one mid-sentence.

There are two buckets per request, not one:

1. 01
   The tenant bucket, sized by plan. It answers "has this customer used its share?"
2. 02
   The provider bucket, sized by the quota you rent from that provider for that model. It answers "will this request get a 429 from upstream?"

Keep the provider bucket honest by reading the rate-limit headers each provider returns on every response (x-ratelimit-remaining-tokens at OpenAI, anthropic-ratelimit-input-tokens-remaining and friends at Anthropic). Your local count drifts; their header is the truth. When the provider bucket runs dry, the router sends the request somewhere else before the provider has to refuse it. The bucket mechanics themselves (refill rate, burst, where the counter lives across gateway replicas) are the same as any distributed limiter; the rate limiter design covers them in depth.

Sharing a quota you rent

Here's where a gateway differs most from a single-tenant client. The provider quota is a shared pool, and the question is who gets it when demand exceeds it.

The default most proxies implement is first come, first served. Under contention, every tenant is cut by the same fraction, so the tenant sending the most traffic gets the most capacity. A free-tier customer running a backfill script can push a paying customer's interactive requests into 429s, without breaking any rule.

Weighted fair share fixes that. Each tenant gets capacity in proportion to a weight (say, 2 for Pro and 1 for Free), but a tenant that needs less than its share only takes what it needs, and the remainder is redistributed to the others. It is max-min fairness with weights. It doesn't punish the heavy tenant when there's room, only when there's contention. In practice you implement it as one queue per tenant drained by something like deficit round robin, with the deficit counted in tokens rather than packets.

Try it. The defaults put three tenants against three providers with 26 requests a minute of combined quota; Noisy Co alone wants 24.

What to notice:

1. 01
   First come, first served: Acme asks for 6 a minute and gets 5, while Noisy Co gets 18 of its 24. Switch to weighted fair share and Acme and Beta get everything they ask for. Noisy Co still gets 16, because the others don't need the rest.
2. 02
   Take Provider B down with the breaker off: the gateway keeps routing nearly a third of the traffic to it, and every tenant sees errors in proportion to how much it sent.
3. 03
   Turn the breaker on: after one bad minute the breaker opens, Provider B drops out of the routing table, and the errors turn into 429s. Throttling is the honest result now that capacity really is smaller. Every few minutes the breaker lets one probe through to check whether B is back.
4. 04
   Press Play and wait: Noisy Co spends its illustrative $6 budget and is blocked outright, and its share flows back to the other two.

The simulator shows request counts to keep the picture readable. The real scheduler works in reserved tokens, as described above.

When a provider goes dark

Health-based routing needs a definition of "unhealthy" that matches what your users feel. Three signals, in rough order of usefulness:

• Server errors and overload responses. Anthropic can return 529 overloaded_error when its API sees high traffic across all users. That is a capacity signal about the provider, not about you.
• Time to first token. A provider that answers in 40 seconds instead of 2 is down for an interactive product even though it returns 200. Trip on a latency threshold, not just on errors.
• Upstream 429s. These mean your quota is exhausted, not that the provider is sick. Route around them, but don't trip the health breaker on them, or one tenant's burst will mark a healthy provider dead for everyone.

The breaker itself is the classic circuit breaker: closed while things work, open after a threshold of failures so calls fail fast instead of piling up on timeouts, half-open after a cooldown to let a trial request through.

The retry budget matters as much as the breaker. Allow one failover per request, two at most, and cap the total time spent before the first token. Unbounded retries during a provider-wide incident turn your gateway into a load amplifier against the providers that are still up.

There is one quiet cost to failover: it throws away warm caches. Provider prompt caches live inside one provider, and OpenAI notes its cached state lives on individual machines, so sticky routing is what earns cache hits. Prefer the provider a tenant's long prefix is already cached on, and treat a failover as costing a cache miss in your routing score.

The first token is the point of no return

Everything in the last section works because the client hadn't seen a byte. Once tokens are on the wire, the options shrink fast.

Mid-stream failures are real, not theoretical. Anthropic documents that an error can arrive after the API has already returned 200, as an error event inside the stream. Its own recovery guidance is to send a continuation request containing the partial response. It also says tool-use and thinking blocks can't be partially recovered: you resume from the last text block.

That continuation trick works with the same model. Across providers it produces an answer written by two models, with a seam a careful reader will find. My call: the gateway does not continue silently. It ends the stream with its own final event that says what happened, how many tokens were delivered, and whether a retry from scratch is safe. Offer continuation as an opt-in mode for plain-text chat only.

Two supporting pieces make this workable:

• Idempotency keys. The client sends one per logical request. If the client's connection drops and it retries, the gateway can replay a completed response from a short-lived store, or say "still running", instead of generating and billing twice. This matters more for agents, where a duplicate response can mean a duplicate tool call.
• A way to tell the client. Headers only work before the first byte. After that, the information has to ride inside the stream. Use a gateway-specific event type that SDKs can ignore, or a comment line, which the server-sent events spec tells clients to skip.

Here is a streamed request that fails over cleanly, step by step. The provider payloads are placeholders; the gateway's own format is ours to define.

Two caches with very different risks

"Add caching" means two unrelated things in a gateway. One is nearly free to adopt. The other can return the wrong answer.

The provider side is cheap and well documented. Anthropic bills cache writes at 1.25× and cache reads at 0.1× the base input price for the default 5-minute lifetime (reads cost less still on some models), and for most of its models cached reads don't count toward the input-token rate limit. Caching raises effective quota as well as cutting cost. OpenAI applies caching by default on supported models once a prefix reaches a minimum length. The gateway's job is to keep prefixes stable: system prompt and tool definitions first, per-request content last. The token optimization guide has the prompt-side techniques.

Watch the isolation boundary. Provider caches are isolated per organization, and on the Claude API per workspace. If every tenant goes through one gateway workspace, they share one cache scope. Hits still require an identical prefix, so no tenant reads another's answer, but cache accounting per tenant is gone. OpenAI lets you pass a prompt_cache_key per customer to keep that accounting separate; on the Claude side, a workspace per large tenant does the same job.

Budgets that stop spending

A budget that's checked after the request is a report, not a cap. Picture ten concurrent requests from one tenant: each one sees "$1 left", and together they spend several times that. The fix is the same reserve-then-settle pattern as the token limiter, applied to dollars.

At an illustrative $3 per million input tokens and $15 per million output tokens, a request with 3,000 input tokens and an 800-token output cap reserves at most 3,000 × $3/1M + 800 × $15/1M ≈ $0.021. If the tenant's remaining budget can't cover that reservation, the request stops at the gateway and never reaches a provider.

Three choices to make explicitly. Who pays for a failed attempt: if a provider failed mid-stream after billing you, eat it as the platform or pass it on, and say which in the plan's terms. Hard or soft cap per plan: a free tier gets a hard stop; an enterprise tenant usually wants an alert and a manager's override rather than an outage at 3 a.m. Where the ledger lives: the reservation needs an atomic decrement (a Redis script or a row lock), while the durable spend record can be an append-only usage log rolled up per day.

What to watch, and where the keys live

Everything above produces events you'll want on a dashboard. The useful cuts:

• Per tenant: tokens and dollars per day against budget, 429s from your limiter, the burn rate (how many days of budget are left at today's pace).
• Per provider and model: time to first token and time between tokens at p50, p95 and p99. These are two different latencies with different causes, so keep them as separate series. Add errors by class (overload, other 5xx, timeouts, mid-stream aborts, upstream 429s) and breaker state changes.
• Per route: failovers per minute, cache read share of input tokens, and the gap between estimated and actual tokens. A growing gap means your estimator needs retuning for a new model.

Keep tenant identifiers out of high-cardinality metric labels once you have more than a few hundred tenants. Per-tenant numbers belong in the usage ledger, which is already keyed by tenant. Don't log prompts or completions by default. Log their token counts and a hash, and make content logging a per-tenant opt-in for debugging.

Keys get their own picture, because the rules are strict and simple:

Tenants only ever hold a gateway key, stored as a hash, scoped to their models and revocable without touching a provider. Provider keys live in a vault and are injected by the adapter. They're never returned in errors, never logged, and rotated with two valid keys overlapping. Use separate provider workspaces or projects per environment so a leaked staging key can't spend production quota. When a large tenant brings its own provider contract, store that key encrypted under its own tenant entry and route that tenant only to it. Its traffic then also stops drawing on your shared quota.

Check yourself

Sources

• OpenAI, Rate limits: RPM, TPM and other limit types, max_tokens and estimate counting, rate-limit response headers.
• OpenAI, Prompt caching: on by default, machine-local cached state, prompt_cache_key, isolation between organizations.
• Anthropic, Rate limits: RPM, ITPM and OTPM, token bucket algorithm, estimate-then-adjust for input, real-time output counting, cache-aware ITPM, response headers.
• Anthropic, Errors: 529 overloaded_error, errors after a 200 on streams.
• Anthropic, Streaming: error events, error recovery and what can't be resumed.
• Anthropic, Prompt caching: cache write and read multipliers, lifetimes, workspace isolation.
• Anthropic, Token counting: counts are estimates; tokenizers differ by model.
• Provider status pages: status.openai.com, status.claude.com.
• Martin Fowler, CircuitBreaker.
• Wikipedia, Deficit round robin.
• WHATWG, Server-sent events.
• Envoy AI Gateway and LiteLLM reliability docs: open-source gateways with provider fallback and token limits.