Why MoE Models Change Your Inference Cost Math

A model card that reads "400B total, 17B active" is quoting you two different costs, and treating them as one is how a team ends up buying GPUs for a 400B model while budgeting for a 17B one. Mixture-of-experts (MoE) architecture now sits under nearly every frontier open model, and it deliberately decouples the compute you burn per token from the memory you have to keep resident. That split is a genuine cost lever, but it only pays off inside a fairly narrow band of serving conditions. This piece maps where that band sits and how to tell whether your workload lands inside it.

What Does "Active vs Total Parameters" Actually Change?

It breaks the assumption, true for every dense model, that one number governs both your compute and your memory. In a dense transformer, all parameters fire on every token, so a 70B model costs 70B worth of both. MoE routes each token through a small set of expert sub-networks and skips the rest, which means the arithmetic per token drops to the active count while the memory to hold the weights stays at the total.

The mechanism is a router in each MoE layer that scores the available experts for the incoming token and dispatches it to the top few. According to the DeepSeek-V3 technical report, each of its MoE layers holds one shared expert plus 256 routed experts, and the router activates 8 of those routed experts per token, sending each token to at most 4 nodes. The other 248 experts contribute nothing to that token's forward pass, yet their weights remain loaded because the next token may route to any of them.

So the model card's two numbers answer two separate questions. Active parameters set your floating-point work per token, which drives latency and the theoretical throughput ceiling. Total parameters set how much VRAM you must own before the model will load at all. A senior engineer sizing a cluster reads the total number first and the active number second.

How Big Are Today's MoE Models, and How Little Do They Activate?

The activation ratios have fallen sharply, and the newest flagships fire a smaller slice of themselves than the first generation did. Mistral's Mixtral 8x7B activates 2 of 8 experts and 12.9B of 46.7B parameters, so it "processes input and generates output at the same speed and for the same cost as a 12.9B model." That is a roughly 28% active share. Two years on, the frontier open models sit near or below 5%.

The figures come from each vendor's own materials: Alibaba's Qwen3 release for the two Qwen variants, Meta's Llama 4 Maverick model card for the 17B-active, 128-expert configuration, and Moonshot's Kimi K2 for the trillion-parameter class. The pattern is deliberate. Pushing more capacity into total parameters while holding active parameters low buys benchmark quality without raising the per-token compute bill, which is exactly why the ecosystem moved this way.

The trade the vendors are making lands on your infrastructure, not theirs. A 3% active share is a strong training and quality story and a hard serving story, because you still have to house 100% of the weights.

Why Does the Memory Bill Stay High When Compute Drops?

Because a router that skips an expert does not unload it. Every expert has to be resident in GPU memory before the first request arrives, since routing is decided per token at inference time and any token can land on any expert. Llama 4 Maverick's 400B parameters occupy roughly 800GB in BF16 or about 400GB in FP8 regardless of the fact that only 17B of them fire on a given token.

That resident footprint interacts badly with how decoding actually spends its time. Token-by-token generation is memory-bandwidth bound, not compute bound: each step reads a large slice of weights and the growing KV cache from HBM to produce a single token, and the matrix multiplies are too small to saturate the GPU's arithmetic units. Cutting active parameters cuts the arithmetic, but the arithmetic was rarely the bottleneck during decode in the first place.

This is the crux that catches teams out. MoE removes work from the part of inference that was already cheap and leaves the memory pressure, which was the expensive part, largely intact. If you want a fuller treatment of why memory traffic dominates the serving bill, the mechanics of the KV cache as the biggest lever in inference cost apply directly here.

When Does an MoE Model Actually Win on Cost?

When your GPUs stay saturated with concurrent requests, and not before. The active-parameter savings are real only if you convert them into throughput, which means keeping enough requests in flight that the expert matrix multiplies grow large enough to become compute bound. At that point the low active count lets one GPU pool serve far more tokens per second than a dense model of equal quality would.

At low utilization the math inverts. If you serve one request at a time, you pay to keep all 671B or 400B parameters powered and resident to produce tokens that only touch a sliver of them, while the sparse compute never gets busy enough to matter. A dense model of similar quality can be cheaper to run in that regime because its weights are smaller and its GPUs are used more fully per request.

There is also a batching subtlety specific to MoE. As you raise the batch size, tokens fan out across more experts, so the memory-bandwidth advantage of activating few parameters erodes: past a certain batch the union of experts touched by the batch approaches the full set, and you are effectively reading most of the model anyway. The sweet spot is a batch large enough to saturate the hardware but not so large that expert coverage saturates too. Finding it is empirical, per model and per GPU.

The practical decision rule: MoE rewards steady, high-concurrency traffic and punishes spiky, low-concurrency or single-tenant workloads. If your product cannot keep the batch full, a smaller dense model or a routing layer that reserves the big MoE for hard queries will usually cost less. That second option pairs naturally with model routing to cut inference cost without losing quality.

How Do You Fit an MoE Model Across GPUs Without Wasting VRAM?

You use expert parallelism, which places whole experts on separate GPUs rather than slicing every weight matrix the way tensor parallelism does. Each GPU owns a distinct subset of experts, and tokens are shipped to whichever GPU holds the expert they routed to. This keeps each device's memory proportional to its share of experts instead of forcing a full copy of the model onto every card.

The cost of that layout is communication. Every MoE layer now needs an all-to-all exchange, dispatching tokens to the GPUs that own their experts and combining the results back, twice per layer. This all-to-all traffic, not the matrix math, is what caps throughput at scale, which is why DeepSeek open-sourced DeepEP, a library of high-throughput, low-latency all-to-all GPU kernels for MoE dispatch and combine that scales to EP2048 while consuming few streaming multiprocessors.

Attention design complicates the parallelism choice further. DeepSeek-V3 uses multi-head latent attention with a single compressed KV head, and standard tensor parallelism cannot shard one head across ranks, so a TP=8 layout ends up duplicating the KV cache eight times. The fix is to combine data parallelism with expert parallelism: data parallelism partitions the KV cache by request while expert parallelism distributes the experts, so each GPU holds a fraction of both. The vLLM data-parallel deployment guide documents this DP-plus-EP pattern as the throughput-oriented default for large MoE models. Where the two goals split, teams increasingly separate the phases entirely, an approach the note on prefill-decode disaggregation in LLM serving covers in depth.

What Should You Watch Before Committing to an MoE Deployment?

Watch the operational tail, because the headline throughput number assumes conditions your production traffic may never hold. Four items decide whether the deployment behaves.

• Engine support lag. vLLM, SGLang, and TensorRT-LLM have matured fast, but native support for a brand-new MoE architecture often trails its release by days or weeks, and early kernels carry edge-case bugs. Do not pin a launch date to a model the day it drops.
• Expert load imbalance. Real traffic does not route uniformly. Popular experts land on a few GPUs and turn them into hot spots while others idle, dragging effective throughput below the theoretical figure. Auxiliary-loss-free balancing during training helps, but you still monitor per-expert load in production.
• A hard batch floor. The cost model only closes if you keep the batch full. A workload that cannot sustain concurrency will run the model at a fraction of its rated efficiency, which is the difference between a deployment that pays for itself and one that quietly does not.
• Quantization interactions. FP8 roughly halves the resident footprint and is how most teams make a 400B-class model fit on a single node, but MoE routing and low-precision experts can interact in ways that cost accuracy on your specific tasks. Measure quality on your evals, not the vendor's.

None of these is a reason to avoid MoE. They are the reasons a serving plan built on active parameters alone tends to miss its targets. Size the cluster to total parameters, prove you can keep the batch saturated, and budget for the all-to-all network before you count the compute savings.

Sources

• DeepSeek-V3 Technical Report (arXiv)
• Llama 4 Maverick model card (Meta, Hugging Face)
• Mixtral of Experts (Mistral AI)
• Qwen3 release (Alibaba)
• Kimi K2 (Moonshot AI)
• DeepEP: an efficient expert-parallel communication library (DeepSeek)
• vLLM data-parallel deployment guide