Why KV Cache Is the Biggest Lever in LLM Inference Cost

At a million tokens of context, the key-value cache consumes 70%-90% of GPU memory and 60%-85% of wall-clock time on a serving node, according to a 2026 survey on system-aware KV cache optimization. That means once your contexts get long, the thing setting your marginal cost per request is not the model weights or the GPU's hourly rate. It's how you store and reuse attention state. Most infrastructure conversations still anchor on parameter count and instance pricing, and miss the lever that actually moves the bill.

Key takeaways

At roughly 1M tokens, the KV cache accounts for 60%-85% of wall-clock time and 70%-90% of GPU memory on a serving node, per a 2026 survey of KV cache optimization.
The cache grows linearly with context length and batch size. For a 70B model at 100K tokens it can exceed 40 GB, more than the footprint of a smaller model's weights.
PagedAttention, the technique behind vLLM, removes memory fragmentation and reports 2x-4x higher throughput than prior serving systems at the same latency.
Multi-head Latent Attention in DeepSeek-V2 cut the KV cache by 93.3% and raised maximum generation throughput to 5.76x its predecessor.
TurboQuant, a Google Research method at ICLR 2026, compresses the KV cache to about 3 bits with reported zero accuracy loss, roughly 6x memory reduction and up to 8x faster attention on H100. Google has not published official code.
The cheapest win for most teams is prefix caching on shared system prompts, which reclaims compute you are otherwise repaying on every single request.

Why does the KV cache dominate your inference bill?

Because at long context the cache, not the weights, becomes the scarce resource, and it turns decoding from a compute-bound problem into a memory-bandwidth-bound one. During autoregressive generation the model reads the entire cache for every new token it produces. As the context grows, that read dominates, and the GPU spends more time moving cached state than doing math.

The 2026 survey on system-aware KV cache optimization puts numbers on it: around a million tokens, the cache eats 60%-85% of wall-clock time and 70%-90% of GPU memory. Weights are a fixed cost you pay once when you load the model. The cache is a variable cost you pay on every token, for every concurrent request.

The practical consequence: two deployments running the identical model on identical hardware can differ 5x-10x in cost per request depending only on how they handle the cache. That gap is engineering, not procurement, and it's why the same model can look cheap on one team's dashboard and ruinous on another's. If you're already tracking spend at the infrastructure layer, this is the line item hiding inside the AI compute gap.

What is the KV cache actually storing?

It stores the key and value projections for every token the model has already processed, so it never has to recompute them. Attention works by comparing the current token's query against the keys of all prior tokens, then pulling a weighted sum of their values. Without a cache, generating token 10,000 would require recomputing keys and values for the preceding 9,999 tokens on every step.

So the cache is a speed-for-memory trade that every production system makes. The size follows a simple formula: two (key and value) times layers times attention heads times head dimension times sequence length times batch size times bytes per element. Every term multiplies, which is why the cache balloons faster than intuition suggests once any single factor grows.

The important structural point for a senior engineer: the cache scales with the model's depth and width, not just context. A deeper model with more heads pays more per cached token, so long-context serving costs interact with architecture choices you locked in long before deployment.

How much does context length change the math?

Linearly per request, and brutally in aggregate. A 70B parameter model holding a 100K token context can allocate 40 GB or more to the KV cache alone, which on many GPUs rivals or exceeds the memory the weights occupy. Double the context and you double the cache. Serve more concurrent users and you multiply it again.

This is why naive long-context serving falls over. You run out of memory not because the model is large but because the cache for a handful of long conversations saturates the card, forcing tiny batch sizes and destroying throughput. The GPU sits underused on compute while starved on memory.

The reframe that matters for planning: context length is a cost multiplier applied to your entire concurrency, not a per-feature toggle. A product decision to support 200K-token documents is also an infrastructure decision that can change your fleet size. Teams that treat the two separately tend to discover the coupling in a bad month. The same discipline that governs token budgeting at the prompt level applies at the memory level.

Which techniques cut KV cache cost, and what do they trade?

Five families do most of the work, and they stack: memory layout, sharing, attention architecture, and quantization. They are not interchangeable. Some you get by choosing a serving engine, some you get by choosing a model, and some you bolt on and must validate yourself. The table below lays out the trade you accept with each.

Technique	What it does	Typical gain	Main trade-off	Where it lives
PagedAttention	Manages KV memory in fixed blocks, like OS virtual memory paging	Near-zero fragmentation; 2x-4x throughput vs prior systems	Requires a paged serving engine; kernel complexity	vLLM, TGI, TensorRT-LLM
Prefix caching	Computes the KV for a shared prefix once and reuses it across requests	Skips recompute of a repeated system prompt for every request	Only helps when prefixes actually repeat; eviction policy matters	vLLM, SGLang
GQA / MQA	Shares key and value heads across multiple query heads	4x-8x smaller cache depending on group size	Must be trained into the model; small quality cost vs full attention	Llama, Mistral, most modern models
MLA (latent attention)	Compresses KV into a low-rank latent vector	93.3% smaller cache and 5.76x throughput in DeepSeek-V2	Architecture change; trained in, not retrofittable	DeepSeek-V2 and V3
KV quantization	Stores keys and values in 3 to 8 bits instead of 16	2x-6x memory reduction; TurboQuant at about 3 bits	Accuracy risk at low bit widths; needs kernel support	KVQuant, TurboQuant, llama.cpp

Read the table by ownership. PagedAttention and prefix caching are serving-engine decisions you can adopt this week without touching the model. GQA and MLA are baked into the weights, so you get them by picking the right model, as covered in serving an open LLM with vLLM. Quantization is the one you apply and own, which makes it the one you must benchmark against your own workload before trusting.

How do PagedAttention and prefix caching give the fastest payback?

They pay back fastest because you adopt them by switching serving engines, with no model change and no retraining. PagedAttention, introduced with vLLM, borrows paging from operating systems: instead of reserving one contiguous memory region per request (which fragments badly), it allocates the cache in fixed blocks mapped through an indirection layer. The vLLM paper reports near-zero KV memory waste and 2x-4x throughput over systems like FasterTransformer and Orca at matched latency.

Prefix caching attacks a different waste. When a thousand requests share the same 2,000-token system prompt, a naive server recomputes that prompt's keys and values a thousand times. Prefix caching computes it once and reuses the result, which is pure recovered compute for any application with a fixed instruction block, a long tool schema, or a shared document.

The trade-off worth noting: prefix caching only helps to the degree your prefixes actually repeat, and the eviction policy decides whether a hot prefix stays resident. For chat products with a stable system prompt the hit rate is near total. For workloads where every request is unique, the gain approaches zero, so measure your prefix reuse before assuming the win.

What does TurboQuant change about the quantization trade-off?

It pushes the accuracy-safe floor of KV quantization down to roughly 3 bits, a level that previously cost visible quality. TurboQuant, from Google Research and accepted at ICLR 2026, reports about 6x memory reduction and up to 8x faster attention on H100 with what the authors describe as zero accuracy loss.

The mechanism is two-stage. It multiplies each vector by a random orthogonal rotation (PolarQuant) so the coordinates spread evenly, then adds a 1-bit residual correction from the QJL line of work to fix the systematic bias that ordinary error-minimizing quantizers introduce into inner-product estimates. The method is data-oblivious and online, meaning it quantizes each key and value the instant it's produced, with no training pass and no learned codebook. That online property is what makes it usable for a cache whose vectors arrive one at a time during decoding.

Two caveats a senior engineer should hold onto. First, Google has not released official code, and the community implementations in PyTorch, Triton, and llama.cpp vary in fidelity, so the zero-accuracy-loss claim is one to reproduce on your own evaluation set, not a spec to inherit. Second, TurboQuant sits in a crowded field of 2026 KV compression work spanning eviction, low-rank, and quantization methods; the right choice depends on whether your bottleneck is memory capacity or attention bandwidth. Extreme quantization helps both, but only after you confirm your kernels actually realize the speedup on your hardware.

What should you deploy first?

Deploy in order of payback per unit of risk, from free wins to owned risk. The sequence below front-loads the changes that require no retraining and defers the ones you must validate.

Run a paged serving engine. Moving to vLLM, TGI, or TensorRT-LLM gets you PagedAttention and usually prefix caching with a config change, no model surgery.
Choose a model with GQA or MLA. The cache reduction is already trained in. A model using MLA carries a structurally smaller cache before you optimize anything, which compounds every later saving.
Turn on prefix caching for shared prompts. If your application has a stable system prompt or tool schema, this is recovered compute with near-zero downside. Confirm your prefix reuse rate first.
Benchmark KV quantization last. INT8 is low-risk and widely supported. Reach for aggressive 3-bit methods like TurboQuant only after you have measured accuracy on your own tasks and confirmed the kernel speedup is real on your GPUs.

The ordering matters because the early steps are reversible config choices and the late ones change output quality. A team that jumps straight to exotic quantization to save memory often spends more engineering time debugging accuracy regressions than a paged engine would have saved outright. For workloads that can tolerate it, the cheapest KV cache is the one you never allocate, which is part of the case for smaller models in agentic systems.

What are the trade-offs and what should you watch?

The central tension is that every KV saving shifts your bottleneck, and the new bottleneck may not be cheaper. Cut memory with quantization and you can become attention-bandwidth-bound; the memory you freed only helps if you convert it into larger batches. Cut the cache with sparse or eviction-based attention and you risk silently dropping context the model needed, which shows up as quality loss that no latency graph will catch.

Watch three things. Accuracy claims in KV research are workload-dependent, so a method that's lossless on a summarization benchmark can degrade on long-horizon reasoning or code. Kernel maturity gates real speedups, and a paper's H100 numbers may not survive on your fleet's cards or your framework version. And reproducing research kernels carries a real engineering cost that belongs in the ROI math, especially when the reference implementation, as with TurboQuant, is community-built rather than official.

The durable takeaway: KV cache optimization is now the primary cost lever in long-context serving, and the highest-leverage decisions (serving engine, model architecture) are the ones you make before you ever tune a quantizer. If your inference bill is growing faster than your traffic, the cache is where to look first.