Diffusion LLMs in Production: When Parallel Decoding Wins

Inception Labs' Mercury Coder generates code at 1,109 tokens per second on a single NVIDIA H100, roughly ten times the throughput of the speed-optimized autoregressive models it matches on quality, per the company's technical report. That number is why diffusion language models stopped being a research curiosity in 2026. They attack the one cost autoregressive decoding cannot escape: producing each token means reading the full model out of memory, one token at a time.

The question for anyone running latency-sensitive inference is no longer whether diffusion LLMs work. It's where the speed holds up, where the quality holds up, and what you trade away to get both.

What Makes a Diffusion LLM Different From an Autoregressive One?

A diffusion LLM produces text by refining a masked sequence in parallel, rather than generating it left to right one token at a time. During training, tokens in a sequence are replaced with a mask symbol at varying rates, and the model learns to recover the originals using bidirectional attention over the whole span. At inference it starts from a mostly masked block, predicts every masked position at once, keeps the high-confidence predictions, and repeats the pass until the block resolves.

That single change moves the bottleneck. Autoregressive decoding is memory-bandwidth bound: each new token forces a full read of the model weights, and you cannot start token N+1 until token N exists. A diffusion model fills many positions in one forward pass, so it uses the GPU's compute and bandwidth far more fully. Inception Labs parameterizes Mercury with a standard Transformer and trains it to predict multiple tokens in parallel, which is what lets a small model post frontier-level throughput.

The trade a senior engineer should note early: bidirectional attention over the generated span is fundamentally incompatible with the causal KV cache that makes autoregressive serving cheap. Solving that tension is the whole game in production, and it is the reason the naive version of this idea never shipped.

How Fast Are Diffusion LLMs Today, and on What Hardware?

Fast enough that the numbers read like a different class of system. The published figures cluster between roughly 1,100 and 2,150 tokens per second per user, on data-center GPUs, from three independent labs.

Two cautions on reading this table. These are output-tokens-per-second figures, which is a throughput measure, not time-to-first-token; a diffusion model can win decisively on the former while a well-tuned autoregressive stack still returns its first token quickly. And the hardware differs across rows, so the raw numbers are not directly comparable across vendors. The signal that matters is the consistency: three separate teams land an order of magnitude above sequential decoding on the same task family.

Why Does Block Diffusion Decide Whether the Speed Is Usable?

Because pure diffusion throws away the KV cache, and without a cache your serving economics collapse. Block diffusion is the compromise that made these models shippable. It splits the sequence into fixed blocks, applies full bidirectional denoising inside a block, and uses ordinary causal caching between blocks, so already-finished blocks are never recomputed.

Block size is the dial. A block of about 32 tokens has become the production default. Smaller blocks cut the number of denoising steps and keep latency down, but fragment generation and can cost coherence. Larger blocks improve quality and let attention see more context at once, but raise per-step compute and erode the speed advantage that justified the architecture.

This is where an infrastructure lead earns their keep. The tuning surface is unfamiliar: you are trading block size against denoising steps against quality, not the throughput-versus-latency curve you already know from batching autoregressive traffic. If your team's mental model of serving cost is built entirely around the [KV cache](/blog/kv-cache-biggest-lever-in-llm-inference-cost), expect to rebuild part of it.

How Does Parallel Decoding Change the Quality Profile?

It makes quality a function of how many refinement passes you are willing to pay for, which is both the appeal and the risk. Each denoising step unmasks the positions the model is most confident about and leaves the rest for the next pass. Fewer steps means faster and rougher output; more steps means slower and cleaner. That is a genuine quality-speed knob, and autoregressive models simply do not have it inside a single generation.

The upside beyond speed is structural. Because attention runs in both directions over the span, diffusion models sidestep the reversal curse and can revise an early token in light of a later one, which suits code, structured output, and fill-in-the-middle tasks. The reported benchmarks bear this out on code: Mercury and Seed Diffusion post competitive scores against similar-scale autoregressive models while running many times faster.

The downside is that the parallel commitment can misfire. Deciding several tokens at once, then only partially revising them, is a weaker guarantee than deciding one token at a time with the full prefix fixed. That is why the honest positioning from Inception Labs on Mercury 2 is competitiveness with Haiku-scale models, not with the largest reasoning models. On long chains of dependent reasoning, sequential decoding still holds an edge.

Where Do Diffusion LLMs Fit, and Where Do They Not?

They fit anywhere throughput and latency dominate and the task tolerates a slightly different quality profile. Real-time code completion is the flagship case, which is why the first commercial diffusion models shipped as coders. Agent inner loops are the second: a planner or tool-router that fires dozens of short generations per task benefits far more from 1,000-plus tokens per second than from the last few points of benchmark quality.

High-volume batch generation is a third fit, where per-request wall-clock time compounds across millions of calls. In each of these, the diffusion model plays a role similar to the one a smaller specialist model plays in an [agentic system built around small language models](/blog/when-to-use-small-language-models-agentic-systems): it handles the hot path, and a stronger model handles the hard calls.

The poor fits are the mirror image. Long-form analytical writing, multi-step mathematical reasoning, and tasks where a single wrong early commitment cascades all favor a frontier autoregressive model. Streaming user experience is also harder, because a block must resolve before it can be shown, so the smooth token-by-token stream users expect from a chat interface takes extra engineering to reproduce.

How Do Diffusion LLMs Compare to Just Speeding Up an Autoregressive Model?

They are a different bet, and the distinction is the most important thing to get right before you commit budget. Speculative decoding, including production frameworks like EAGLE-3 and DeepSeek's DSpark, accelerates an existing autoregressive model while leaving its output distribution exactly intact, because the target model verifies every drafted token. You get the same answers, faster, on the model you already trust.

A diffusion LLM is a new model with its own distribution. The speedup is larger and it is architectural rather than a decoding trick, but you are not getting the same answers faster; you are getting a different model's answers. That means a real evaluation gate, not a swap-and-ship.

The practical reading: if you need your current model's exact behavior, faster, reach for [speculative decoding](/blog/speculative-decoding-eagle-3-production-when-it-pays-off) first. If you are choosing a model for a throughput-bound workload from scratch, a diffusion LLM belongs on the shortlist. The two are not competitors so much as tools for different problems, and both sit alongside serving-level wins like [prefill-decode disaggregation](/blog/prefill-decode-disaggregation-llm-serving).

What Should You Watch Before Committing?

Watch three things: tooling maturity, evaluation on your own traffic, and open-weight availability. The serving stack around diffusion models is younger than the autoregressive ecosystem, so expect gaps in the frameworks, quantization paths, and observability you take for granted. Budget engineering time for the block-size and denoising-step tuning that has no autoregressive equivalent.

Evaluate on your workload, not the code benchmarks. The published wins concentrate on code and structured tasks; your traffic may sit closer to the reasoning cases where the gap is real. Run a proper offline eval before you route production traffic, and treat time-to-first-token, not just tokens per second, as a first-class metric if you serve interactive users.

On availability, the field is moving toward open weights. Google DeepMind has signaled an open diffusion release in the Gemma line per reporting, which would let teams self-host rather than depend on a single hosted API. If diffusion becomes central to your latency strategy, an open-weight option is the difference between a bet you control and one you rent.

Sources

• Mercury: Ultra-Fast Language Models Based on Diffusion (Inception Labs technical report, arXiv)
• Inception Labs: Introducing Mercury, a commercial diffusion LLM
• Google DeepMind: Gemini Diffusion
• Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed Inference (arXiv)
• The Decoder: Inception launches Mercury 2, a diffusion reasoning model
• Daily Dose of Data Science: Diffusion LLMs from the ground up, training and inference engineering