How to Serve Multiple LoRA Adapters on One vLLM Server

Teams that fine-tune one base model into a dozen task-specific variants hit the same wall: standing up a separate GPU server for every adapter burns money and leaves hardware idle. vLLM avoids that by loading many LoRA adapters on top of a single base model in one process, then routing each request to the right adapter by name. The short version: launch with --enable-lora, register adapters with --lora-modules, and set the model field of each API call to the adapter's name. That one server can then answer as any of your fine-tunes.

What You Need Before You Start

Three things, and the first is non-negotiable. A working single-model vLLM install is the foundation, so if you have not stood one up yet, start with the basics of serving an open model with vLLM for high throughput and come back.

• vLLM installed (pip install vllm) on a CUDA GPU with enough VRAM to hold the base model plus KV cache.
• A base model on Hugging Face, for example meta-llama/Llama-3.2-3B-Instruct.
• One or more LoRA adapters trained against that exact base checkpoint. If you still need to produce them, the workflow to fine-tune a model with LoRA and QLoRA on a single GPU outputs the adapter directories you will load here. An adapter trained on a different base will fail to load.

How Do You Serve Multiple LoRA Adapters on One vLLM Server?

Follow these six steps and one server will answer as any of your adapters. Everything you need to finish the task lives in this section.

1. Install vLLM and confirm the GPU is visible.

   pip install vllm
   python -c "import torch; print(torch.cuda.is_available())"

2. Collect your adapter directories. Each holds an adapter_config.json and the weight file, and each must have been trained against the base model you are about to serve.
3. Start the server with LoRA enabled and register the adapters. The --enable-lora flag turns on adapter support and each --lora-modules entry maps a request name to a path, as shown in the vLLM LoRA serving guide.

   vllm serve meta-llama/Llama-3.2-3B-Instruct \
       --enable-lora \
       --lora-modules sql-lora=/models/sql-lora math-lora=/models/math-lora \
       --max-loras 4 \
       --max-lora-rank 32 \
       --max-cpu-loras 8

4. Match the caps to your adapters. Set --max-loras to the number of adapters you expect in one batch and --max-lora-rank to the largest rank any adapter uses. Leave --max-cpu-loras higher than --max-loras so extra adapters can wait in CPU memory instead of being rejected.
5. Confirm the adapters registered. They appear in the model list next to the base model.

   curl http://localhost:8000/v1/models

6. Send a request to a specific adapter by name. Put the adapter name, not the base model name, in the model field.

   curl http://localhost:8000/v1/completions \
       -H "Content-Type: application/json" \
       -d '{
           "model": "sql-lora",
           "prompt": "List every customer in San Francisco:",
           "max_tokens": 64,
           "temperature": 0
       }'

   Chat completions work the same way: point model at math-lora and call /v1/chat/completions. Omit the adapter name and the base model answers.

What Does Each LoRA Flag Control?

Four flags decide how many adapters you can hold and how large they can be. The defaults below come from the vLLM LoRAConfig reference, and the two rank and count knobs are the ones that cause most first-run failures.

Higher --max-lora-rank reserves more memory whether or not you use it, so set it to your real maximum rather than padding it. If every adapter is rank 16, leave the default alone.

How Do You Add or Remove Adapters Without Restarting?

vLLM can register and drop adapters on a live server, which matters when you ship new fine-tunes without a maintenance window. Set the environment variable first, since the runtime endpoints stay disabled otherwise.

export VLLM_ALLOW_RUNTIME_LORA_UPDATING=True
# then start vllm serve as usual

With that set, POST to load a new adapter and POST again to unload it, both documented in the vLLM LoRA serving guide:

curl -X POST http://localhost:8000/v1/load_lora_adapter \
  -H "Content-Type: application/json" \
  -d '{"lora_name": "sql-lora-v2", "lora_path": "/models/sql-lora-v2"}'

curl -X POST http://localhost:8000/v1/unload_lora_adapter \
  -H "Content-Type: application/json" \
  -d '{"lora_name": "sql-lora-v2"}'

The same guide warns that enabling runtime updating in production is risky, because anyone who can reach the endpoint can load arbitrary adapter weights. Keep these endpoints behind an internal network or an authenticating proxy.

Prefer to drive it from Python instead of the HTTP API? Offline, build the engine with LLM(model=..., enable_lora=True) and pass a LoRARequest(name, unique_id, path) straight into generate:

from vllm import LLM, SamplingParams
from vllm.lora.request import LoRARequest

llm = LLM(model="meta-llama/Llama-3.2-3B-Instruct", enable_lora=True)
out = llm.generate(
    "List every customer in San Francisco:",
    SamplingParams(temperature=0, max_tokens=64),
    lora_request=LoRARequest("sql-lora", 1, "/models/sql-lora"),
)

Should You Use Multi-LoRA, Merged Weights, or Separate Servers?

Multi-LoRA is the right default for many small fine-tunes of one base, but it is not the only option. Merging an adapter back into the base gives the fastest single-variant path, and fully separate servers make sense when variants use different base models.

For autoscaling many adapters across replicas, a serving layer such as Ray Serve's multi-LoRA deployment sits on top of vLLM and handles replica placement.

Troubleshooting Common vLLM LoRA Errors

Most first-run failures trace to three mismatches: rank, name, and memory. Here is how each shows up and what fixes it.

• The server refuses to load an adapter and mentions rank. The adapter's rank exceeds --max-lora-rank. Read the rank from the adapter's adapter_config.json and raise the flag to match.
• A request returns a model-not-found error. The model value does not match a registered adapter name. Call /v1/models and copy the exact string, including case.
• You get out-of-memory errors after raising --max-loras. Each concurrent adapter adds activation memory on top of the base model. Lower --max-loras, or keep extra adapters in CPU memory with a higher --max-cpu-loras so they swap in rather than co-resident.
• The load and unload endpoints return 404. VLLM_ALLOW_RUNTIME_LORA_UPDATING was not set before the server started. Export it and restart.
• Loading fails with a shape or key mismatch. The adapter was trained against a different base checkpoint than the one you are serving. Retrain or serve the matching base.

What to Do Next

Once a single adapter answers correctly, harden the deployment. Watch per-request latency and throughput with vLLM's engine arguments and metrics so you know when --max-loras is starving a hot adapter. If VRAM is tight, quantize the base model to fit more KV cache, then scale out replicas behind a load balancer once one server saturates. Load-test each adapter separately, since a low-traffic adapter can still block a batch if --max-loras is set too low.

Sources

• vLLM: LoRA Adapters serving guide
• vLLM: LoRAConfig API reference
• vLLM: Engine Arguments
• Hugging Face PEFT: LoRA conceptual guide
• Ray Serve: Multi-LoRA deployment