The engine that wins a throughput benchmark is often the wrong one to deploy. Independent 2026 tests on a single H100 put TensorRT-LLM roughly 13% ahead of vLLM on a 70B model, yet that lead is workload-specific and evaporates the moment your model, quantization, or draft-decoding setup isn't the exact path the engine optimized for. The decision that actually blocks a launch is feature coverage, not tokens per second, and that is where these three engines genuinely diverge.
Key takeaways
- vLLM is the correct default for most teams: it runs on the widest hardware range and exposes an OpenAI-compatible server from a single install, so a throughput gap in the single digits rarely justifies giving that up.
- One 2026 H100 benchmark on Llama 3.3 70B in FP8 measured TensorRT-LLM at 2,100 tok/s, SGLang at 1,920, and vLLM at 1,850 at 50 concurrent requests, a 13% spread that costs a roughly 28-minute compile step.
- SGLang's RadixAttention pays off when traffic shares long prefixes, near 29% on prefix-heavy 8B workloads but only 3%-5% on unique-prompt 70B work, so the win depends entirely on your traffic shape.
- All three now enforce structured output through XGrammar, but TensorRT-LLM's C++ backend needs a separate tokenizer-info extraction step before it works.
- TensorRT-LLM engines are architecture-locked: change the GPU, the model, or the quantization format and you rebuild the engine.
- The real blocker is combination support. Speculative decoding, grammar-constrained output, and your model's quantization do not all compose cleanly on every engine.
Which Inference Server Should You Default To?
Default to vLLM unless you have a concrete reason not to. It installs with one pip command, exposes an OpenAI-compatible endpoint out of the box, and runs across NVIDIA CUDA, AMD ROCm, and other backends documented in the vLLM project documentation, so your client libraries and application code survive a hardware or model swap untouched.
That portability is the point. On a first production deployment the model is rarely final, and an engine you can move between GPU vendors without a rebuild is worth more than a single-digit throughput edge you would spend weeks re-benchmarking. If you want the fastest route to a running server, our walkthrough on serving an open LLM with vLLM for high throughput covers the setup end to end.
You reach for SGLang or TensorRT-LLM when a specific property of your workload or hardware makes the default leave real money on the table. The rest of this piece is about identifying when that is actually true.
Why Does Throughput Rarely Decide the Choice?
Because the winner changes with your workload and the exact versions you pin, so a headline number tells you little about your own traffic. A 2026 benchmark from Spheron's H100 comparison ran Llama 3.3 70B in FP8 on one H100 SXM5 80GB instance and measured TensorRT-LLM leading at every concurrency level once compiled, at a 13% advantage over vLLM at 50 concurrent requests.
The catch sits in the setup cost. That same test recorded a compile step of about 28 minutes for TensorRT-LLM against under 90 seconds for the other two. A 13% throughput gain is real, but you pay for it in deploy latency every time you change anything the engine compiles against.
Benchmark hygiene matters more than the numbers. Comparing a compiled TensorRT-LLM result against a different framework version of vLLM and calling the delta an engine effect is a common and misleading error. Reproduce any comparison on your own model, your own prompt distribution, and pinned versions before you weight it at all.
What Does Each Engine Support for Structured Output?
All three can constrain generation to a JSON schema or grammar, but they get there through different backends with different setup costs. vLLM defaults to an auto mode that picks between XGrammar and the guidance backend per request, and its unified structured_outputs parameter replaced the older guided_json and guided_regex fields as of v0.12.0, per the vLLM structured outputs documentation.
TensorRT-LLM supports guided decoding only through the XGrammar backend, covering JSON, JSON Schema, Regex, and EBNF grammar. Its C++ backend adds a preprocessing step: you run generate_xgrammar_tokenizer_info.py to extract the tokenizer info into a JSON file before guided decoding works, a requirement spelled out in the NVIDIA guided decoding guide. SGLang accepts XGrammar, Outlines, or llguidance, and keeps a grammar cache so repeated schemas cost almost nothing after warmup.
The practical implication for a senior engineer: if strict schema conformance is a hard requirement, verify it on your model before you commit, because a shared XGrammar dependency does not mean an identical developer experience or an identical set of edge cases across the three.
| Capability | vLLM | SGLang | TensorRT-LLM |
|---|---|---|---|
| Structured output backend | XGrammar or guidance (auto) | XGrammar, Outlines, llguidance | XGrammar only, with tokenizer-info prep on C++ backend |
| Prefix reuse | Automatic prefix caching | RadixAttention, on by default | KV cache reuse, engine dependent |
| Speculative decoding | n-gram, EAGLE, draft models | EAGLE-2, EAGLE-3, MTP, draft, n-gram | Draft-model modes set at build time |
| Multi-LoRA serving | Yes, dense and MoE layers | Yes | Yes, rank set at engine build |
| Hardware | CUDA, ROCm, and more | Primarily NVIDIA, ROCm support | NVIDIA only, architecture-locked |
| Config change cost | Restart, seconds | Restart, seconds | Recompile, about 28 minutes |
When Does SGLang's Prefix Reuse Actually Pay Off?
SGLang wins when your requests share long opening prefixes, and it does close to nothing extra for you when they don't. Its RadixAttention stores the KV cache in a radix tree keyed on the token prefix, so requests that begin with the same system prompt, retrieved corpus, or conversation history reuse that computation instead of recomputing it.
The size of the win tracks the workload directly. Independent tests put SGLang around 29% ahead of vLLM on prefix-heavy 8B traffic, but the gap narrows to 3%-5% on a 70B model with unique prompts, where the work turns compute-bound rather than memory-bound. RadixAttention runs by default and reports its cache hit rate at the /metrics endpoint, so you can confirm reuse against your real traffic rather than trusting a generic claim.
This maps cleanly to a decision rule. Fixed-corpus RAG, multi-turn agents, and structured decoding with a stable schema all repeat prefixes and favor SGLang; a stream of short, unrelated one-shot prompts does not, and the default engine is fine. If SGLang fits, the mechanics of getting it running are in our guide on serving an LLM with SGLang for faster inference.
What Does TensorRT-LLM's Compile Step Cost You Operationally?
It turns every configuration change into a rebuild, which reshapes how you run the fleet. A TensorRT-LLM engine is compiled and kernel-fused for a specific model, precision, and GPU architecture, so switching from an H100 to a different card, or from FP8 to INT4-AWQ, means generating a new engine before you can serve.
That has three operational consequences worth naming. Model updates stop being a config push and become a build-and-ship pipeline. You need spare cycles and infrastructure to own engine builds, not just a serving container. And a fleet upgrade cannot be a fast rolling restart, because each node needs the matching compiled engine staged first.
None of this is disqualifying at steady state. If your model is stable, your hardware is fixed NVIDIA silicon, and you serve enough volume that a 13% throughput gain covers real cost, TensorRT-LLM earns its place. The friction only bites teams that iterate on models weekly, and for them it can quietly dominate the total cost of the choice.
How Do Speculative Decoding and LoRA Differ Across the Three?
All three ship speculative decoding and multi-LoRA serving, but the maturity and the way you configure each differs. vLLM supports n-gram, EAGLE, and draft-model methods and provides multi-LoRA serving across dense and MoE layers; SGLang offers one of the broader menus, with EAGLE-2, EAGLE-3, Multi-Token Prediction, standalone draft models, and n-gram documented in its speculative decoding guide.
TensorRT-LLM sets its draft-model speculative mode at engine build time through trtllm-build, which folds the acceleration choice back into that same recompile constraint. For a deeper read on when the acceleration is worth the added complexity at all, see our analysis of when EAGLE-3 speculative decoding pays off in production.
The trap is combinations. Enabling speculative decoding and grammar-constrained output together has produced real conflicts, and a specific quantization format may not support your chosen draft method. Serving several fine-tunes as adapters has its own scaling limits, covered in our guide to serving multiple LoRA adapters on one vLLM server. Each feature can work in isolation and still fail when stacked.
What Should You Watch Before Committing?
Treat "supports the architecture" as the start of due diligence, not the answer. An engine that lists your model on its support matrix can still lack a fast path for your exact quantization, your speculative method, or the two combined, and you find that out in a load test, not a docs page.
Three checks save the most pain. Pin and record every version in any benchmark you trust, because an untracked upgrade invalidates the comparison. Confirm the specific feature stack you need, quantization plus structured output plus speculative decoding, actually runs together on a single node before you design around it. And if you move to TensorRT-LLM, canary it on a slice of traffic with the compiled engine staged, rather than cutting the whole fleet at once.
The honest summary is that these engines converge on capability and diverge on operational cost. vLLM buys flexibility, SGLang buys prefix efficiency for the right traffic, and TensorRT-LLM buys peak throughput in exchange for a compile-time contract. Pick against your workload and your iteration speed, not against a leaderboard.