Why can't an LLM serving platform just process one request at a time per GPU?

A GPU is built to do enormous numbers of matrix multiplications in parallel. Running one sequence at a time uses a tiny fraction of that parallelism, so far more GPUs are needed to hit a given throughput target than necessary — at genuinely high cost, since GPU-hours are the dominant expense of any inference platform.

What is continuous batching and why does it matter?

Static batching groups requests into a fixed batch and waits for the whole batch to finish before starting the next one — so a short request gets stuck behind a long one it happened to be grouped with, and the GPU sits partially idle padding through already-finished sequences. Continuous (in-flight) batching slots a new waiting request in the instant any sequence in the current batch finishes, keeping the GPU's batch dimension continuously full and avoiding that head-of-line blocking.

Why does KV-cache memory matter as much as GPU compute?

Every token generated for every in-flight sequence keeps an attention cache (the KV cache) resident in GPU memory, and it grows with both context length and how many sequences are being served concurrently. In practice, GPU memory capacity for KV cache — not raw compute — is often the actual ceiling on how many concurrent requests a GPU pool can serve, which is why cache eviction and admission policy are core design decisions, not implementation details.

What's the detail that separates a senior answer from a mid-level one here?

Distinguishing time-to-first-token from inter-token latency as two genuinely different problems with two different bottlenecks — the first dominated by queueing and the compute-heavy prompt 'prefill' pass, the second by the memory-bandwidth-bound per-token 'decode' pass — rather than treating 'latency' as one undifferentiated number to optimize.

LLM Inference Serving Platform Design

An idle GPU is one of the most expensive idle things in computing. A serving platform's entire design exists in service of one number: how much useful work gets extracted from each GPU-hour before the bill comes due. Everything below — batching, memory management, even how requests get scheduled — is an answer to that same economic pressure wearing a different technical hat.

The Real Cost Center: An Idle GPU

At 50,000 requests/second, each producing on the order of 300 output tokens, the platform needs to generate roughly 15 million tokens/second in aggregate. If a single GPU, with batching, can sustain on the order of 2,000 tokens/second for a mid-sized model, that implies needing roughly 7,500 GPUs just for steady-state throughput — before headroom for peak load. That number is why every design decision below is really about extracting more useful work per GPU, not just adding more of them: GPUs are both extremely expensive per hour and slow to cold-start (loading model weights alone can take minutes), so unlike a typical stateless web service, this system can't casually over-provision "just in case," and can't scale up instantly in response to a spike either.

The platform has to accept a prompt and stream generated tokens back, serve multiple models and versions behind one surface, support many tenants sharing the same GPU fleet, and track per-request token usage for cost accounting. It doesn't train or fine-tune models, doesn't build agent/tool-calling orchestration on top (a separate system), and doesn't generate invoices — only accurate token counts for something downstream to bill against.

http

POST /v1/completions
{ "model_id", "prompt", "max_tokens", "stream": true }
→ streamed tokens over the connection, plus a request_id

GET /v1/requests/{id}/usage
→ { "input_tokens", "output_tokens" }

Four components make that box diagram actually work at the economics above. Each is a genuinely separate hard problem — worth taking one at a time rather than flattening them into one "the architecture is X" paragraph.

Component 1: From Static to Continuous Batching

Processing one request at a time per GPU wastes almost all of a GPU's parallel compute capacity, which is built for many large matrix multiplications simultaneously, not one small sequence at a time — hitting any meaningful throughput target this way needs far more GPUs than should be necessary.

Static batching — group several incoming requests into a fixed batch, run the forward pass for all of them together — multiplies effective throughput, but introduces head-of-line blocking: requests in a batch finish generating at different times, so the whole batch waits for its longest-running sequence before a new batch can start. A short request gets stuck behind a long one it happened to be grouped with, and GPU cycles get spent padding through sequences that have already finished.

Continuous batching (in-flight batching) fixes this directly: the instant any sequence in the currently-running batch finishes, a new waiting request is slotted into its place immediately, so the batch never has to fully drain before accepting new work. The GPU's batch dimension stays continuously full, utilization stays high, and a short request is never stuck behind an unrelated long one.

Component 2: The KV-Cache Is the Real Ceiling, Not Compute

Every sequence occupying a batch slot holds a growing attention cache (the KV cache) resident in GPU memory for as long as it runs, and it grows with both context length and how many sequences are running concurrently. In practice, GPU memory capacity for this cache — not raw compute — is frequently the actual limit on how many concurrent sequences a GPU can serve.

That makes eviction and admission policy a core scheduling decision, not an implementation detail. When memory fills, lower-priority sequences can be preempted — their cache freed, the sequence queued to restart later — to make room for higher-priority or already-further-along work, rather than the GPU simply refusing new requests outright. This is a genuine trade-off, not a solved problem: evict too aggressively and the preempted tenant's fairness suffers; evict too conservatively and the GPU risks running out of memory entirely.

Component 3: Two Bottlenecks Wearing One Name — "Latency"

Time-to-first-token is dominated by queueing delay plus the "prefill" pass — processing the entire input prompt at once, compute-heavy, scaling with prompt length. Inter-token latency, once generation has started, is dominated by the "decode" pass — one token at a time, more memory-bandwidth-bound than compute-bound. Treating "latency" as one undifferentiated number to optimize misses that these are two different problems with two different bottlenecks.

Because the bottlenecks differ, an increasingly common approach is disaggregating prefill and decode onto separate GPU pools, each tuned for its own bottleneck, rather than running both phases on the same hardware configuration and accepting a compromise on both.

Component 4: Fair-Queuing So One Tenant Can't Starve Everyone Else

A tenant submitting a sudden burst of thousands of requests shouldn't be able to fill every batch slot on a shared GPU pool and starve everyone else. The scheduler needs per-tenant admission limits and fair-queuing into the batch — not simply first-come-first-served — with priority tiers (a paying tenant's traffic weighted above a free tier's) baked into how contested batch slots are allocated.

Component 5: Autoscaling When Startup Takes Minutes, Not Seconds

Unlike a stateless web server, a GPU node can take minutes to become useful — provisioning plus loading model weights into memory — so purely reactive autoscaling (add capacity once the queue is already backed up) arrives too late for a real spike. Autoscaling here needs to be predictive, triggered on a rising queue-depth trend rather than a breached threshold, and paired with a small warm standby pool that absorbs the first minutes of a spike while new capacity is still coming online.

The Data Model

Tenant — a customer of the platform, with a quota and priority tier
InferenceRequest — the prompt, model, and generation parameters for one call
Sequence — the in-flight generation state for a request while it's running: tokens produced so far, its KV-cache allocation
ModelDeployment — a specific model version and which GPU pool serves it
UsageRecord — the input/output token counts for a completed request, feeding cost accounting

The modeling choice worth explaining if asked: Sequence is intentionally separate from InferenceRequest and is the most ephemeral entity in the system — it exists only while a generation is actively running, holding live GPU-memory state (the KV cache) that has no meaning once the request completes or fails. UsageRecord, by contrast, is the durable, long-lived artifact — it's what billing and cost accounting depend on, so it's written once the sequence completes and never touches GPU state again.

Failure Modes

Failure	Impact	Response
GPU node failure mid-generation	In-flight sequences on that node are lost	The client's stream errors out cleanly; because generation isn't deterministic anyway, the expected recovery is a clean client-side retry, not an attempt to resume mid-sequence
Scheduler/router instance crash	Requests routed through it fail	Stateless design, multiple replicas behind a load balancer — a crash just removes one instance from rotation
New model version rollout	A bad model version could serve broken output to all traffic at once	Canary/blue-green rollout — new versions take a small traffic percentage first, monitored before a full cutover
GPU pool for one model exhausted	Requests for that model queue or reject while other models' pools sit idle	Pools sized per model's demand, with the option to burst a popular model onto shared/general capacity rather than being rigidly fixed

What This Looks Like in Production

Signal	Why it matters	Alert on
Time-to-first-token (P99)	The first thing a user perceives as "slow"	Sustained P99 above target
Inter-token latency (P99)	Determines how "smooth" streaming feels	Sustained rise vs. baseline
GPU utilization and KV-cache memory usage	Directly reflects whether continuous batching is doing its job	Utilization dropping (wasted capacity) or memory pressure rising toward eviction
Per-tenant queue depth	Confirms fair-queuing is actually working under load	One tenant's queue growing much faster than others sharing the same pool

Multi-tenant isolation is the load-bearing requirement underneath all of it: one tenant must never see another tenant's prompts, outputs, or even aggregate usage patterns, which has direct implications for how logs, caches, and debugging tooling are scoped. Prompts and outputs may contain sensitive or regulated content depending on the tenant's use case, so log retention policy for raw prompt/output text needs to be an explicit, configurable decision — not a default "keep everything for debugging" choice — and token usage accounting needs to be tamper-evident, since it's what tenant billing is built on.

Calibrating the Bar: Mid, Senior, Staff

Mid-level: Proposes batching requests together as an improvement over one-at-a-time processing, and can be prompted toward continuous batching once static batching's head-of-line blocking is pointed out.

Senior: Independently proposes continuous batching and explains why KV-cache memory management has to go alongside it; distinguishes time-to-first-token from inter-token latency without prompting.

Staff+: Designs explicit multi-tenant fair-queuing and admission control as a first-class scheduler concern; raises prefill/decode disaggregation as the state-of-the-art response to the two-different-bottlenecks problem; reasons about predictive autoscaling given GPU cold-start time, rather than assuming reactive autoscaling (which works fine for stateless web services) transfers directly to this domain.