Why can't an LLM serving platform just process one request at a time per GPU?

A GPU is built to do enormous numbers of matrix multiplications in parallel. Running one sequence at a time uses a tiny fraction of that parallelism, so far more GPUs are needed to hit a given throughput target than necessary — at genuinely high cost, since GPU-hours are the dominant expense of any inference platform.

What is continuous batching and why does it matter?

Static batching groups requests into a fixed batch and waits for the whole batch to finish before starting the next one — so a short request gets stuck behind a long one it happened to be grouped with, and the GPU sits partially idle padding through already-finished sequences. Continuous (in-flight) batching slots a new waiting request in the instant any sequence in the current batch finishes, keeping the GPU's batch dimension continuously full and avoiding that head-of-line blocking.

Why does KV-cache memory matter as much as GPU compute?

Every token generated for every in-flight sequence keeps an attention cache (the KV cache) resident in GPU memory, and it grows with both context length and how many sequences are being served concurrently. In practice, GPU memory capacity for KV cache — not raw compute — is often the actual ceiling on how many concurrent requests a GPU pool can serve, which is why cache eviction and admission policy are core design decisions, not implementation details.

What's the detail that separates a senior answer from a mid-level one here?

Distinguishing time-to-first-token from inter-token latency as two genuinely different problems with two different bottlenecks — the first dominated by queueing and the compute-heavy prompt 'prefill' pass, the second by the memory-bandwidth-bound per-token 'decode' pass — rather than treating 'latency' as one undifferentiated number to optimize.

LLM Inference Serving Platform Design

Every LLM API call looks the same from the outside — send a prompt, stream back tokens — but underneath, a serving platform is solving an unusually tight resource-allocation problem: GPUs are extremely expensive and mostly idle unless kept continuously busy, and different requests are wildly different sizes. This post walks through that design the way an interview conversation actually unfolds.

Requirements and Goals of the System

Functional Requirements

Accept a prompt and stream generated tokens back to the caller
Serve multiple models and model versions behind one platform
Support many tenants sharing the same underlying GPU fleet
Track token usage per request for downstream cost accounting

Below the line (out of scope)

Model training or fine-tuning — this system only serves already-trained models
Prompt orchestration, agents, or tool-calling built on top — that's a separate system (a natural next post in this series)
The billing/invoicing system itself — we only need to emit accurate token counts, not generate an invoice

Non-Functional Requirements

Scale: ~50,000 requests/second platform-wide
Latency: low time-to-first-token (the delay before the first token streams back) and low inter-token latency (the delay between subsequent tokens, since that's what determines how "fast" a response feels while streaming)
Efficiency: GPU-hours are the dominant cost — utilization matters as much as raw latency
Fairness: no single tenant should be able to monopolize shared GPU capacity and degrade everyone else's latency

Capacity Estimation and Constraints

Token throughput needed: at 50,000 requests/second, each producing on the order of 300 output tokens, the platform needs to generate roughly 15 million tokens/second in aggregate. If a single GPU, with batching, can sustain on the order of 2,000 tokens/second for a mid-sized model, that implies needing roughly 7,500 GPUs just for steady-state throughput — before headroom for peak load. This is the number that explains why every design decision below is really about extracting more useful work per GPU, not just adding more of them.

KV-cache memory: every token generated for every in-flight sequence keeps an attention cache resident in GPU memory, growing with both context length and how many sequences are being served concurrently. For long-context requests and large batch sizes, this cache — not raw compute — is frequently the actual limit on how many concurrent sequences a GPU can serve, which is why cache eviction policy (below) is a first-class design concern rather than an implementation detail.

The constraint that matters more than the others: GPUs are both extremely expensive per hour and slow to cold-start (loading model weights alone can take minutes) — so unlike a typical stateless web service, this system can't casually over-provision "just in case," and can't scale up instantly in response to a spike either.

System APIs

http

POST /v1/completions
{ "model_id", "prompt", "max_tokens", "stream": true }
→ streamed tokens over the connection, plus a request_id

GET /v1/requests/{id}/usage
→ { "input_tokens", "output_tokens" }

High-Level Design

The core requirement to design around: serve many concurrent generation requests with low latency, while keeping expensive GPUs as continuously utilized as possible.

Bad solution: process one request at a time per GPU — run the full forward pass for one sequence, token by token, until it's done, then move to the next.

This wastes almost all of a GPU's parallel compute capacity, which is built for doing many large matrix multiplications simultaneously, not one small sequence at a time. Hitting any meaningful throughput target this way requires far more GPUs than should be necessary, at a cost that doesn't scale.

Good solution: static batching — group several incoming requests into a fixed batch and run the forward pass for all of them together as one padded batch, using the GPU's parallelism far more effectively.

This multiplies effective throughput, but introduces head-of-line blocking: requests in a batch finish generating at different times (some hit their stop condition earlier than others), so the whole batch has to wait for its longest-running sequence before a new batch can start — a short request gets stuck waiting behind a long one it happened to be grouped with, and GPU cycles get spent padding through sequences that have already finished.

Great solution: continuous batching (also called in-flight batching). The instant any sequence in the currently-running batch finishes, a new waiting request is slotted into its place immediately — the batch never has to fully drain before accepting new work. This keeps the GPU's batch dimension continuously full, so utilization stays high and a short request is never stuck behind an unrelated long one. This pairs with a KV-cache manager that dynamically allocates and frees per-sequence cache memory as sequences enter and leave the batch, since — as the capacity estimate above showed — GPU memory, not just compute, is often the real ceiling on concurrency.

Database Design

Tenant — a customer of the platform, with a quota and priority tier
InferenceRequest — the prompt, model, and generation parameters for one call
Sequence — the in-flight generation state for a request while it's running: tokens produced so far, its KV-cache allocation
ModelDeployment — a specific model version and which GPU pool serves it
UsageRecord — the input/output token counts for a completed request, feeding cost accounting

The modeling choice worth explaining if asked: Sequence is intentionally separate from InferenceRequest and is the most ephemeral entity in the system — it exists only while a generation is actively running, holding live GPU-memory state (the KV cache) that has no meaning once the request completes or fails. UsageRecord, by contrast, is the durable, long-lived artifact — it's what billing and cost accounting depend on, so it's written once the sequence completes and never touches GPU state again.

Detailed Component Design

How does continuous batching actually interact with KV-cache memory?

Every sequence occupying a slot in the batch holds a growing KV cache for as long as it runs — longer context and more concurrent sequences both push memory usage up. When GPU memory fills, the scheduler needs an explicit eviction or admission policy: lower-priority sequences can be preempted (their cache freed, the sequence queued to restart later) to make room for higher-priority or already-further-along work, rather than the GPU simply refusing new requests outright. This is a genuine trade-off, not a solved problem — evicting too aggressively hurts fairness for the preempted tenant; evicting too conservatively risks the GPU running out of memory entirely.

How do we keep one tenant from monopolizing shared GPU capacity?

A tenant submitting a sudden burst of thousands of requests shouldn't be able to fill every batch slot on a shared GPU pool and starve everyone else. The scheduler needs per-tenant admission limits and fair-queuing into the batch — not simply first-come-first-served — with priority tiers (a paying tenant's traffic weighted above a free tier's) baked into how contested batch slots are allocated.

Why are time-to-first-token and inter-token latency different problems?

Time-to-first-token is dominated by queueing delay plus the "prefill" pass — processing the entire input prompt at once, which is compute-heavy and scales with prompt length. Inter-token latency, once generation has started, is dominated by the "decode" pass — one token at a time, which is more memory-bandwidth-bound than compute-bound. Because these two phases have different bottlenecks, a increasingly common approach is disaggregating prefill and decode onto separate GPU pools, each tuned for its own bottleneck, rather than running both phases on the same hardware configuration and accepting a compromise on both.

How do we autoscale GPU capacity when GPUs are slow to start?

Unlike a stateless web server, a GPU node can take minutes to become useful (provisioning plus loading model weights into memory), so purely reactive autoscaling — add capacity once the queue is already backed up — arrives too late for a real spike. Autoscaling here needs to be predictive, triggered on a rising queue-depth trend rather than a breached threshold, and paired with a small warm standby pool that absorbs the first minutes of a spike while new capacity is still coming online.

Fault Tolerance and Replication

Failure	Impact	Response
GPU node failure mid-generation	In-flight sequences on that node are lost	The client's stream errors out cleanly; because generation isn't deterministic anyway, the expected recovery is a clean client-side retry, not an attempt to resume mid-sequence
Scheduler/router instance crash	Requests routed through it fail	Stateless design, multiple replicas behind a load balancer — a crash just removes one instance from rotation
New model version rollout	A bad model version could serve broken output to all traffic at once	Canary/blue-green rollout — new versions take a small traffic percentage first, monitored before a full cutover
GPU pool for one model exhausted	Requests for that model queue or reject while other models' pools sit idle	Pools sized per model's demand, with the option to burst a popular model onto shared/general capacity rather than being rigidly fixed

Extended Requirements

Observability & Monitoring

Signal	Why it matters	Alert on
Time-to-first-token (P99)	The first thing a user perceives as "slow"	Sustained P99 above target
Inter-token latency (P99)	Determines how "smooth" streaming feels	Sustained rise vs. baseline
GPU utilization and KV-cache memory usage	Directly reflects whether continuous batching is doing its job	Utilization dropping (wasted capacity) or memory pressure rising toward eviction
Per-tenant queue depth	Confirms fair-queuing is actually working under load	One tenant's queue growing much faster than others sharing the same pool

Security & Compliance

Multi-tenant isolation is the load-bearing requirement here: one tenant must never be able to see another tenant's prompts, outputs, or even aggregate usage patterns, which has direct implications for how logs, caches, and debugging tooling are scoped. Prompts and outputs may contain sensitive or regulated content depending on the tenant's use case, so log retention policy for raw prompt/output text needs to be an explicit, configurable decision — not a default "keep everything for debugging" choice — and token usage accounting needs to be tamper-evident, since it's what tenant billing is built on.

Leveled Expectations

Mid-level: Proposes batching requests together as an improvement over one-at-a-time processing, and can be prompted toward continuous batching once static batching's head-of-line blocking is pointed out.

Senior: Independently proposes continuous batching and explains why KV-cache memory management has to go alongside it; distinguishes time-to-first-token from inter-token latency without prompting.

Staff+: Designs explicit multi-tenant fair-queuing and admission control as a first-class scheduler concern; raises prefill/decode disaggregation as the state-of-the-art response to the two-different-bottlenecks problem; reasons about predictive autoscaling given GPU cold-start time, rather than assuming reactive autoscaling (which works fine for stateless web services) transfers directly to this domain.