How to Serve an Open LLM With vLLM for High Throughput

A single NVIDIA A100 can serve a 13B model to dozens of concurrent users or to exactly one, and the difference is entirely how you run it. The short version of this guide: install vLLM, point vllm serve at a Hugging Face model id, and you get an OpenAI-compatible endpoint on port 8000 that batches incoming requests for you. vLLM's PagedAttention algorithm holds memory waste under 4% and, in the project's launch benchmarks, delivered up to 24x the throughput of raw Hugging Face Transformers. What follows takes you from an idle GPU box to a production-shaped inference server, and names the handful of flags that actually decide whether it stays up.

Check These Prerequisites First

If your goal is a model on a laptop for local tinkering, Ollama is the lighter tool and I have covered it separately in running large language models locally with Ollama. Reach for vLLM when you need concurrency, request batching, and throughput on a real GPU.

Before you start, confirm you have:

• A Linux host with an NVIDIA GPU and a current CUDA driver. vLLM also ships wheels for AMD ROCm.
• Python 3.10 through 3.13.
• Enough VRAM. A rough rule for 16-bit weights is about 2 GB per billion parameters, so a 7B model needs roughly 14 GB for weights alone before the KV cache. A 16 GB card handles small models; anything larger wants a 40 GB card or several GPUs. If GPU spend is the real constraint, I worked through that math in measuring AI infrastructure costs.
• A Hugging Face token if the checkpoint is gated. Set it up first with the steps in accessing gated models on Hugging Face.

Serve Your First Model in Four Steps

This is the whole procedure. Finish these four steps and you have a working, queryable server; everything after is tuning.

1. Create an isolated environment and install vLLM. The project recommends uv, which resolves the right Torch build for your hardware.

   uv venv --python 3.12 --seed
   source .venv/bin/activate
   uv pip install vllm --torch-backend=auto

2. Start the server against a Hugging Face model id. Begin with a small model to confirm the box works before you load anything heavy. It binds to localhost:8000 by default.

   vllm serve Qwen/Qwen2.5-1.5B-Instruct

3. Confirm the server is up and reporting the model.

   curl http://localhost:8000/v1/models

4. Send a chat completion. The model field has to match the id you served.

   curl http://localhost:8000/v1/chat/completions \
     -H "Content-Type: application/json" \
     -d '{
       "model": "Qwen/Qwen2.5-1.5B-Instruct",
       "messages": [
         {"role": "user", "content": "Explain KV cache paging in two sentences."}
       ]
     }'

If that last call returns JSON with a completion, you are done with the hard part. The server exposes the standard OpenAI routes, including /v1/completions, /v1/chat/completions, /v1/embeddings, and /v1/models.

Point Existing OpenAI Code at Your Server

Because the API matches OpenAI's, you swap the base URL and reuse the client you already have. When you have not set a server key, any non-empty string works as api_key.

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

resp = client.chat.completions.create(
    model="Qwen/Qwen2.5-1.5B-Instruct",
    messages=[{"role": "user", "content": "Give me one tip for load testing an inference server."}],
)
print(resp.choices[0].message.content)

Tune the Flags That Control Memory and Throughput

vLLM defaults run, but they rarely fit your GPU or your traffic. These are the flags worth knowing before you serve anything real, drawn from the engine arguments reference.

A memory-safe starting command for a 7B model on a single 24 GB card looks like this:

vllm serve Qwen/Qwen2.5-7B-Instruct \
  --gpu-memory-utilization 0.90 \
  --max-model-len 8192 \
  --api-key sk-local-your-secret

Scale a Model Past a Single GPU

When the weights don't fit on one card, tensor parallelism splits the model across several GPUs on the same node. Set --tensor-parallel-size to the number of GPUs you want to use, and vLLM handles the sharding.

vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --tensor-parallel-size 4

The parallelism and scaling guide covers the case where a model is too large for even one node: you combine tensor parallel with --pipeline-parallel-size across machines. For a single box, tensor parallel is the one you want.

Troubleshoot the Errors You'll Actually Hit

Almost every early failure traces back to memory or to a mismatched request. Here are the ones you will meet and what fixes each.

The single most useful habit: size --max-model-len to the traffic you actually serve. vLLM reserves KV cache blocks for the full window, so every token of headroom you never use is VRAM you paid for and wasted.

Move the Server Toward Production

A working localhost server is the start, not the finish. From here:

• Put the server behind a reverse proxy or gateway, and keep --api-key set. Never expose the raw port to a network you do not control.
• Fit a larger model on the same hardware with quantization. The quantization guide lists the supported methods and their trade-offs.
• Load test before you promise anyone a latency number. Measure tokens per second and time to first token at your real concurrency, not at one request.
• Right-size the model itself. A smaller model that clears your quality bar serves more users per GPU, which is the argument I made in when to use small language models in agentic systems.

If you are weighing a serving stack for a system you own and want a second opinion, my contact details are on the homepage.

Sources

• vLLM Quickstart
• vLLM OpenAI-Compatible Server
• vLLM Engine Arguments
• vLLM serve CLI Reference
• vLLM Parallelism and Scaling
• vLLM launch announcement and PagedAttention benchmarks