How to Serve an LLM With SGLang for Faster Inference

Inference is where an LLM project quietly bleeds money. A model is trained once but served millions of times, and a serving stack that batches poorly can cost several times more per token than one that batches well. SGLang, the serving engine from the LMSYS team, exists to raise that batching efficiency, and its defining feature is RadixAttention, which reuses the KV cache across requests that share a prefix. The short version of how to run it: install the package, launch python -m sglang.launch_server --model-path <your-model>, then send OpenAI-format requests to port 30000.

Key takeaways

Short answer: install with uv pip install --prerelease=allow sglang, run python -m sglang.launch_server --model-path Qwen/Qwen3-8B, and call the OpenAI-compatible API at http://localhost:30000/v1.
SGLang needs Python 3.10 or higher and an NVIDIA GPU. The current install targets CUDA 13 by default and ships a separate CUDA 12 path.
The single most important tuning knob is --mem-fraction-static. Lower it when you hit out-of-memory errors, raise it for more KV cache and higher concurrency.
Use --tp-size to shard one model across several GPUs and --quantization (awq, fp8, gptq) to fit larger models on less memory.
SGLang and vLLM solve the same problem. SGLang's edge shows up on workloads with shared context, such as agents and few-shot prompts, where prefix reuse pays off.

What You Need Before You Start

This is a GPU serving stack, so the prerequisites are mostly hardware and access, not libraries.

An NVIDIA GPU with current drivers. An 8B model in bf16 carries roughly 16 GB of weights before any KV cache, so a 24 GB card is a comfortable floor. Quantize to fit smaller cards.
Python 3.10 or newer, which the SGLang installation guide lists as the minimum.
A Hugging Face account and an access token if the model is gated, which Llama and several others are. Create one on the Hugging Face access tokens page.
Optional: Docker, if you would rather run the official image than manage a Python environment.

How to Serve a Model With SGLang in Four Steps

These four steps take you from an empty environment to a running server your application can call. The reader can finish the task from this section alone.

Install SGLang. The docs recommend uv for a faster, cleaner resolve:
```
pip install --upgrade pip
pip install uv
uv pip install --prerelease=allow sglang
```
On a CUDA 12 host, reinstall the matching Torch and kernel wheels afterward, following the CUDA 12 section of the same install guide.
Launch the server. Point it at any Hugging Face model ID and SGLang downloads the weights on first run:
```
python -m sglang.launch_server \
  --model-path Qwen/Qwen3-8B \
  --host 0.0.0.0 \
  --port 30000
```
Wait for the log line that says the server is ready. The default bind is 127.0.0.1:30000, so add --host 0.0.0.0 only when you need clients on other machines to reach it, as noted in the request guide.

Send a request. The endpoint speaks the OpenAI chat format, so a plain curl works:

curl -s http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "Qwen/Qwen3-8B", "messages": [{"role": "user", "content": "What is the capital of France?"}]}'

Point your app at it. Because the API is OpenAI-compatible, the official OpenAI SDK works after you change the base URL:

import openai

client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="None")

resp = client.chat.completions.create(
    model="Qwen/Qwen3-8B",
    messages=[{"role": "user", "content": "Summarize SGLang in one line."}],
)
print(resp.choices[0].message.content)

That is a working server. Everything below makes it fit your hardware and run faster.

Which Server Flags Actually Matter?

A handful of flags decide whether your GPU runs hot or half-idle. Every one below is documented in the SGLang server arguments reference, and these are the ones you reach for first.

Flag	Default	What it controls
`--tp-size`	1	Tensor parallel degree. Set it to the number of GPUs to shard one model across them.
`--mem-fraction-static`	about 0.88	Share of GPU memory reserved for weights plus KV cache. Lower it to stop OOM, raise it for more concurrency.
`--quantization`	None	Weight compression: awq, fp8, gptq, or marlin. Cuts memory at some quality cost.
`--context-length`	model config	Maximum sequence length. Lower it to reclaim KV cache memory when your prompts are short.
`--chunked-prefill-size`	engine default	Tokens processed per prefill chunk. Reduce it to survive long-prompt OOM.
`--max-running-requests`	auto	Concurrency cap. Lower it if you run out of memory during decoding.

Serving a Model Across Multiple GPUs

When a model is too large for one card, shard it with tensor parallelism instead of reaching for a bigger GPU.

python -m sglang.launch_server \
  --model-path meta-llama/Llama-3.1-70B-Instruct \
  --tp-size 4

Weights split evenly across the GPUs, but the KV cache pool is usually what dictates how many concurrent requests you can hold. If you have not sized that before, the reasons the KV cache dominates inference cost are worth reading before you pick a card count.

Fitting a Bigger Model With Quantization

Quantization trades a little accuracy for a large drop in memory, which often decides whether a model fits at all. Pass a pre-quantized checkpoint and name the method:

python -m sglang.launch_server \
  --model-path <awq-checkpoint> \
  --quantization awq

SGLang also loads LoRA adapters at serve time, so you can host several fine-tunes of one base model without duplicating the weights. The mechanics mirror the approach in serving multiple LoRA adapters on one server.

How Do You Run SGLang in Docker?

The official image removes the CUDA and Python setup entirely, which is the fastest path to a reproducible deployment. Map the port, mount your Hugging Face cache, and pass a token for gated models:

docker run --gpus all --shm-size 32g -p 30000:30000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  --env "HF_TOKEN=<your-token>" --ipc=host \
  lmsysorg/sglang:latest \
  python3 -m sglang.launch_server \
    --model-path meta-llama/Llama-3.1-8B-Instruct \
    --host 0.0.0.0 --port 30000

The --ipc=host and --shm-size flags matter because tensor parallel workers communicate through shared memory, and a small default will crash multi-GPU runs.

Troubleshooting Common Errors

Most first-run failures are memory related and follow a short decision tree from the SGLang tuning guide.

CUDA out of memory at startup or during prefill. Lower --mem-fraction-static to 0.8 or 0.7, and reduce --chunked-prefill-size to 4096 or 2048 for long prompts. The tuning guide suggests reserving 5 to 8 GB for activations.
Out of memory during decoding. Lower --max-running-requests so fewer sequences share the KV cache at once.
Requests pile up and latency climbs. Watch the #queue-req field in the server logs. A healthy range sits between 100 and 2000. Values that stay far above that mean you are memory-bound, so add GPUs or quantize.
401 or access-denied on model download. Export HF_TOKEN and accept the model license on its Hugging Face page before launching.
Connection refused from another machine. The server binds to 127.0.0.1 by default. Relaunch with --host 0.0.0.0 and open the port.

What to Do Next

You have a server. The next moves turn it into a deployment you can trust under load.

Load-test it with the built-in benchmark, python -m sglang.bench_serving, described in the tuning guide, and tune --mem-fraction-static until token usage sits above 0.9 without OOM.
Decide whether SGLang or an alternative fits your traffic. If your prompts do not share much prefix, compare it against the setup in serving an open LLM with vLLM for high-throughput inference.
Read the SGLang project README for the current model support matrix and the RadixAttention design before you commit a production workload.