Prefill-Decode Disaggregation in LLM Serving

Every team running large language models at scale pays the same hidden tax. A long prompt lands, the GPU spends hundreds of milliseconds ingesting it, and every other user's next token waits in line behind it. Prefill-decode disaggregation removes that tax by running the two phases of inference on separate GPU pools that scale on their own. The design went from a 2024 research paper to the default architecture behind Kimi, DeepSeek, Perplexity, and NVIDIA's serving stack in under two years, for one measurable reason: you can't hold a tight time-to-first-token and a tight per-token latency on the same shared hardware.

Key takeaways

DistServe, the OSDI 2024 paper that named the approach, served 7.4x more requests or met a 12.6x tighter latency target than the best colocated system of its day, while keeping over 90% of requests inside both latency budgets.
Prefill is compute-bound and decode is memory-bandwidth-bound, so the two phases want different machines; an H200 moves 4.8 TB/s of memory bandwidth against an H100's 3.35 TB/s.
Under colocation, one large prefill can inflate per-token latency for concurrent requests by 2x to 30x, per the DistServe authors' 18-month retrospective.
Mooncake, Moonshot AI's disaggregated architecture, runs in production at Kimi and processes more than 100 billion tokens a day across thousands of nodes.
Shipping the KV cache between pools is the hard part; NIXL now unifies NVLink, InfiniBand, PCIe, and SSD transfers, and an 8,000-token batch can push roughly 10 GB across the wire.
Disaggregation stops paying off below about 512-token prompts, fewer than 10 concurrent users, or models under 7B parameters, where colocation or chunked prefill wins.

What Does Prefill-Decode Disaggregation Actually Change?

It splits inference into two independently scaled services: a prefill pool that ingests the prompt and a decode pool that generates the response. Instead of one GPU carrying a request end to end, the prefill pool builds the KV cache for the prompt, ships that cache to a decode GPU, and the decode pool streams tokens back to the user.

The split follows the physics of the two phases. Prefill runs one dense forward pass over the whole prompt, so it saturates the GPU's math units and stays compute-bound. Decode emits one token at a time, and each step re-reads the KV cache for every prior token, so it stalls on memory bandwidth rather than compute. The device sits mostly idle during decode, waiting for weights and cache to stream out of HBM.

Those two profiles want different machines. A prefill node rewards raw FP8 throughput; a decode node rewards memory bandwidth, where an H200 delivers 4.8 TB/s against an H100's 3.35 TB/s. Colocation forces both jobs onto identical silicon and makes neither efficient. Disaggregation lets you buy the right inference hardware for each half and size the two pools to your real traffic mix.

Why Couldn't Colocated Serving Keep Up?

Because the two phases fight for the same GPU, and prefill wins. When a long prompt arrives on a shared device, its compute-heavy forward pass monopolizes the card, and every concurrent request's token generation stalls behind it. The DistServe authors measured that interference directly: a single large prefill inflated time-per-output-token for other requests by 2x to 30x.

That collision breaks the metric that matters. Interactive LLM products carry two separate latency budgets, time-to-first-token (TTFT) for responsiveness and time-per-output-token (TPOT) for streaming speed. On shared hardware, tuning batches to protect TTFT starves TPOT, and the reverse holds too. The number worth optimizing is goodput: the request rate you can serve while staying inside both budgets at once, not raw tokens per second.

"When businesses run competitively at full scale, system throughput is not the only most important metric any more. Taming latency has become increasingly critical to the growth (or even survival) of their businesses." Hao AI Lab, on why disaggregation won.

How Much Faster Is Disaggregated Serving in Practice?

DistServe, the OSDI 2024 paper that named the technique, served 7.4x more requests or met a 12.6x tighter latency target than the best colocated system of its day, while holding over 90% of requests inside both TTFT and TPOT constraints. Those gains held up in production, not just on a benchmark.

Mooncake, Moonshot AI's KV-cache-centric disaggregated architecture, took FAST 2025 best paper and runs behind Kimi, where it clears more than 100 billion tokens a day across thousands of nodes. On open models, SGLang serving DeepSeek-R1 across 96 H100 GPUs reports 52.3k input tokens per second and 22.3k output tokens per second per node, per the DistServe team's retrospective.

Newer hardware widens the gap because it lets each pool specialize further. Moving from an H100 setup to a GB200 NVL72 rack produced 3.8x higher prefill throughput and 4.8x higher decode throughput in that same retrospective, since the decode pool can spread wide expert parallelism across the rack's fast interconnect.

How Does the KV Cache Move Between the Two Pools?

Over the network, and that transfer is the whole engineering problem. Once prefill finishes, its KV cache has to reach the decode GPU before generation starts, so the link between pools becomes a first-class part of the design rather than an afterthought.

The volume is not small. For an 8,000-token context served at a batch of 8, the cache runs to roughly 10 GB that must cross the wire per batch. NIXL, NVIDIA's Inference Xfer Library, has become the standard mechanism in both vLLM and NVIDIA Dynamo; it hides NVLink, InfiniBand, PCIe, and SSD fabrics behind one abstraction and does zero-copy GPU-to-GPU transfer over RDMA. DeepSeek built its own store, 3FS, pooling thousands of SSDs for locality-agnostic access. The KV cache is also the single largest lever in inference cost, so how you move it and how large it grows both feed the bill.

Network sizing tracks context length. Practitioner guidance puts 10GbE as adequate below about 2,000 tokens, 100GbE as necessary at 8,000-plus, and RDMA or RoCE as the floor past 32,000 tokens. Underprovision the fabric and the transfer latency erases the gains disaggregation bought you, which is why the topology deserves as much attention as the GPU choice.

Which Serving Strategy Fits Your Workload?

Three strategies sit on a spectrum from simplest to most scalable, and the right one turns on prompt length, concurrency, and how tight your latency budgets run. Full disaggregation earns its complexity only at scale; smaller workloads do better on a single node.

Strategy	How it works	Best for	Main trade-off
Colocated serving	Prefill and decode share each GPU	Low traffic, short prompts, simple operations	Prefill stalls decode; can't hit both SLOs at once
Chunked prefill	Long prompts split into chunks and interleaved with decode on one node	Moderate concurrency, prompts under about 8,000 tokens	20%-40% throughput gain only; still one hardware profile
Full disaggregation	Separate prefill and decode pools with the KV cache shipped between them	High concurrency, long prompts, strict TTFT and TPOT	KV transfer cost, fast fabric required, more moving parts

Chunked prefill ships in vLLM and similar servers and makes a sensible first step: it recovers much of the interference loss on one node before you take on the operational weight of two pools.

When Should You Keep Prefill and Decode Together?

Below a clear size threshold, disaggregation costs more than it returns. The KV transfer adds 50 to 200 ms of network latency to a cold request, and that penalty dominates when the work itself is small. Practitioner guidance flags four cases where colocation or chunked prefill wins: prompts under 512 tokens, fewer than 10 concurrent users, models under 7B parameters, and tight latency budgets on cold traffic.

Operational cost is the quieter tax. Two pools mean two autoscaling targets, a fabric to provision and monitor, and a failure mode (a stranded or slow KV transfer) that colocated serving simply doesn't have. If your traffic doesn't fill both pools, you strand GPUs on one side while the other saturates, which sharpens the infrastructure cost math rather than easing it.

The technique also has a hard research edge. Pushing disaggregation further, by splitting attention from the feed-forward network (AFD), works today only for mixture-of-experts models, whose all-to-all communication already carries the extra traffic. For dense models the activation transfer overhead makes AFD impractical, and overlapping that communication with computation remains unsolved.

What to Watch: SLA-Driven Autoscaling and a Shrinking KV Cache

Two shifts will change the math on disaggregation over the next year. The first is autoscaling that reacts to service-level targets instead of raw utilization. NVIDIA Dynamo, announced at GTC 2025, adds a Planner that watches TTFT and inter-token latency in real time and shifts GPUs between the prefill and decode pools as the traffic mix moves, which attacks the stranded-capacity problem head on.

The second is a cheaper KV cache. Google Research's TurboQuant, presented at ICLR 2026, compresses the cache to about 3 bits per coordinate with no reported accuracy loss and no calibration, cutting KV memory by 6x and speeding attention by up to 8x on an H100. A smaller cache shrinks exactly the payload disaggregation ships between pools, so the two techniques compound. Teams already treating KV cache as their dominant inference cost now get quantization and disaggregation pulling in the same direction.

Watch the fabric next. Research on prefill-as-a-service already asks whether the KV cache should travel across datacenters, which would turn prefill into a shared utility rather than a pool you own. That work is early, but it points at where the line between your cluster and the network is heading.