Hybrid Mamba-Transformer LLMs: When SSM Layers Pay Off

The most expensive part of serving a transformer is not the model weights. It's the key-value cache that grows with every token of context and every concurrent request, and it's the reason IBM reports its Granite 4.0 hybrid models need over 70% less RAM than comparable transformers on long inputs and concurrent batches. That single number is why hybrid Mamba-Transformer architectures moved from research curiosity to shipped enterprise models in late 2025 and 2026. This piece explains the mechanism, reads the production numbers against their primary sources, and gives a decision rule for when a hybrid earns its place over a pure transformer.

Key takeaways

A hybrid interleaves a majority of Mamba state-space layers with a minority of attention layers. Granite 4.0-H uses a 9:1 Mamba-2-to-transformer ratio.
State-space layers carry a fixed-size recurrent state instead of a per-token KV cache, so their memory stays constant as context and batch size grow, while a transformer's cost scales quadratically with sequence length.
IBM reports over 70% RAM reduction on long-context, multi-session workloads. NVIDIA reports Nemotron-H is up to 3x faster at inference than similarly sized transformers, and Nemotron Nano 2 up to 6x higher throughput in reasoning settings.
Pure state-space layers are measurably worse at exact copying and associative recall. Keeping a few attention layers is what preserves that capability, which is the entire argument for hybrids over pure SSMs.
The gains show up under long context and high concurrency. For short prompts at low batch size, a hybrid buys little, and the serving-stack support is still thinner than for transformers.

What is a hybrid Mamba-Transformer model?

It's a language model that runs most of its layers as Mamba state-space blocks and only a handful as standard self-attention. The design keeps the attention layers precisely because they do something the state-space layers do poorly, and it swaps everything else for a mechanism that scales better.

Attention compares every token against every other token. That's what gives transformers their recall, and it's also why their compute and memory grow quadratically: double the context and a transformer does four times the work and stores four times the intermediate state. A Mamba layer instead maintains a fixed-size recurrent state that it updates token by token. As IBM puts it, Mamba's compute scales linearly with sequence length and its memory stays constant regardless of how long the sequence gets.

The hybrid bet is that you don't need attention everywhere. You need it in a few places for exact lookups, and you can let efficient recurrent layers carry the rest. Granite 4.0-H interleaves the two sequentially in a 9:1 ratio, with the Mamba-2 blocks processing global context and periodically routing it through a transformer block. Nemotron-H takes the same shape, replacing the majority of self-attention layers with Mamba-2 layers that use constant compute and memory per generated token.

Why do the state-space layers cut memory and cost?

They remove the KV cache from most of the network, and the KV cache is what makes transformer serving expensive at scale. Every attention layer has to store a key and value vector for every token it has seen, for every sequence in the batch. That store grows linearly with context length and linearly with batch size, and it sits in GPU memory the whole time the request is alive.

A Mamba layer has no KV cache. It compresses history into a fixed-size state, so a 2,000-token conversation and a 200,000-token one cost the same memory in that layer. Cut nine out of ten attention layers and you cut most of the cache. That's the direct source of IBM's claim of over 70% less RAM for long inputs and concurrent batches, and it compounds exactly where transformer serving hurts most: long context and many simultaneous sessions.

The cost story follows the memory story. If KV-cache memory no longer limits how many requests fit on a GPU, you raise batch size and throughput on the same hardware, or serve the same load on cheaper hardware. IBM frames it as running on significantly cheaper GPUs at reduced cost. This is the same lever behind why KV cache is the biggest lever in LLM inference cost; hybrids attack it at the architecture level rather than the serving level.

What do the production numbers actually show?

Independent releases from IBM, NVIDIA, and AI21 report similar-or-better accuracy against comparable transformers while moving faster and using less memory, with the biggest gains on long context and high-concurrency workloads. The figures below come from each vendor's own announcement or technical report, so read them as vendor-reported and benchmark-specific rather than universal.

Model	Architecture	Reported efficiency claim	Context	Source
Granite 4.0-H (Small 32B/9B active, Tiny 7B/1B, Micro 3B)	Mamba-2 + transformer, 9:1 ratio, MoE in larger variants	Over 70% less RAM on long inputs and concurrent batches vs comparable transformers	Trained to 512K, validated to 128K	IBM
Nemotron-H (8B, 47B, 56B)	Majority Mamba-2, minority attention	Up to 3x faster inference at similar or better accuracy than Qwen-2.5 and Llama-3.1 peers	Long-context trained	NVIDIA
Nemotron Nano 2 (9B)	Nemotron-H-style hybrid, reasoning-tuned	Up to 6x higher throughput than Qwen3-8B at 8K input / 16K output, on-par accuracy	Long reasoning traces	NVIDIA
Jamba 1.5	Transformer + Mamba + MoE	First production-grade Mamba-based model; efficient long-context serving	256K	AI21

Two patterns are worth naming. First, the throughput multiplier grows with output length: Nemotron Nano 2's 6x shows up in a reasoning setting with 16K output tokens, because long generations are where a transformer's per-token KV growth bites hardest and a recurrent state doesn't. Second, the accuracy claims are consistently "on par or better," not "better across the board." The architecture is being sold on efficiency at equal quality, which is the honest framing.

Where do pure state-space layers fail, and why keep attention?

State-space layers are measurably worse at copying long spans verbatim and at associative recall, which is the whole reason a hybrid keeps attention instead of going pure Mamba. Jelassi and colleagues showed in "Repeat After Me" that transformers beat state-space models at copying strings and at retrieving specific items from context, and that the gap widens as the string gets longer. The mechanism is intuitive: a fixed-size recurrent state is a lossy summary, so when you ask it to reproduce an exact 40-token identifier from 100K tokens back, it can blur.

Attention doesn't blur, because it keeps every token addressable. That's expensive, but for exact lookup it's the right tool. So the hybrid keeps a few attention layers to preserve precise in-context retrieval, and hands the bulk of the sequence processing to Mamba. The 9:1 style ratio is an engineering answer to a real trade-off, not a rounding choice: enough attention to retrieve, few enough to keep memory flat.

For a senior engineer, the implication is that workload shape decides fit. A task dominated by exact retrieval from very long context, strict citation, or code that must be reproduced token for token, sits closest to the state-space weak spot, and you'll want to test recall directly rather than trust an aggregate benchmark.

When should you deploy a hybrid instead of a transformer?

Deploy a hybrid when your serving cost is dominated by KV-cache memory, which in practice means long context, high concurrency, or long generations, and reach for a transformer when none of those hold. The economics only tilt where the cache is the binding constraint.

Concretely, three profiles favor a hybrid. Long-context RAG and document ingestion, where each request carries tens of thousands of tokens. High-concurrency agents and customer-service systems, where hundreds of sessions share a GPU and each one's cache competes for memory. And reasoning workloads that emit long chains of thought, where the throughput multiplier is largest. These are the same pressure points behind prefill-decode disaggregation and the move toward small language models in agentic systems; a hybrid is another way to buy back the same memory.

Against that, a short-prompt, low-batch chat endpoint sees little benefit, because there's barely any cache to save. And if your task leans hard on exact recall from long context, validate that specific behavior before committing. Granite 4.0's small footprint also opens on-device and edge deployment that a comparably capable transformer can't reach, which matters if data residency or latency rules out a hosted call.

What breaks in the serving stack when you adopt one?

The tooling assumes transformers, so expect gaps in kernels, quantization paths, and anything built around a KV cache. This is the real adoption tax, and it's shrinking, but it's not zero in 2026.

Serving engines have added hybrid support unevenly. Check that your target runtime, whether vLLM, SGLang, or a vendor stack, has current Mamba-2 kernels for your GPU, since state-space scan kernels are hardware-specific and lag behind attention kernels on newer accelerators. Quantization is the second gap: recipes and calibration tooling are tuned for attention weights, and state-space layers can be more sensitive, so re-measure quality after quantizing rather than assuming transformer-grade results carry over.

The third gap is conceptual. A lot of serving infrastructure, from cache eviction to speculative decoding to prefix sharing, is designed around the KV cache. Some of it doesn't map cleanly onto a recurrent state, so features you take for granted may be absent or behave differently. Budget time to re-benchmark your own workload; the vendor's numbers are a starting hypothesis, not a substitute for measuring your traffic on your hardware.

What to watch next

The architectures are converging. Granite 4.0, Nemotron-H, and Jamba independently landed on the same recipe, a majority of Mamba-2 layers, a minority of attention, and Mixture-of-Experts on top, which suggests the hybrid plus MoE stack is becoming the default for efficiency-first open models rather than one vendor's bet.

Two things will decide how far it spreads. Whether serving stacks reach transformer-grade maturity for state-space kernels and quantization, which removes the adoption tax. And whether the recall gap stays contained to a few edge cases or shows up in mainstream long-context tasks as context windows push past 256K. If you run memory-bound inference at scale, the responsible move now is to benchmark one hybrid against your current transformer on your own long-context, high-concurrency traffic, and read the memory and recall numbers side by side before you commit.