How to Serve Embeddings Locally With Hugging Face TEI

Every RAG pipeline runs an embedding model on the hot path. You embed documents at ingest and you embed the user's query on every single request, so that model's latency and throughput set the ceiling for your whole retrieval layer. Calling a hosted embedding API works until the per-token bill or the data-residency rule stops you, and a raw sentence-transformers loop pins one request to one GPU with no batching. The clean answer is to run Text Embeddings Inference (TEI), Hugging Face's Rust server for embedding and reranker models: docker run --gpus all -p 8080:80 -v $PWD/data:/data ghcr.io/huggingface/text-embeddings-inference:cuda-1.9 --model-id BAAI/bge-large-en-v1.5 gives you a batching, OpenAI-compatible endpoint in one line.

What You Need Before You Start

• Docker installed, plus the NVIDIA Container Toolkit if you want GPU acceleration. TEI runs on CPU too, which is fine for small models and low traffic.
• An NVIDIA driver compatible with CUDA 12.2 or higher for the GPU images.
• The Hugging Face model ID you want to serve. Good starting points: BAAI/bge-large-en-v1.5, Alibaba-NLP/gte-base-en-v1.5, or Qwen/Qwen3-Embedding-0.6B.
• A Hugging Face access token only if the model is gated or private.

How to Deploy an Embedding Model With TEI in Five Steps

This section is the whole task. Run these and you have a working embedding server; the rest of the article explains the choices and fixes the errors.

1. Launch the server on a GPU. Set a shared volume so weights download once, then start the container.

   model=BAAI/bge-large-en-v1.5
   volume=$PWD/data
   
   docker run --gpus all -p 8080:80 -v $volume:/data \
     --pull always \
     ghcr.io/huggingface/text-embeddings-inference:cuda-1.9 \
     --model-id $model

2. Or launch on CPU if you have no GPU. Drop --gpus all and switch to the CPU image tag.

   docker run -p 8080:80 -v $PWD/data:/data \
     --pull always \
     ghcr.io/huggingface/text-embeddings-inference:cpu-1.9 \
     --model-id BAAI/bge-large-en-v1.5

3. Confirm the server is up. The /health route returns 200 once weights are loaded, and /info reports the model, its embedding dimension, and max input length.

   curl 127.0.0.1:8080/health
   curl 127.0.0.1:8080/info

4. Get an embedding. POST text to /embed and you get back the raw vector. Send an array to embed a batch in one call.

   curl 127.0.0.1:8080/embed \
     -X POST \
     -H 'Content-Type: application/json' \
     -d '{"inputs":["What is deep learning?","Embeddings map text to vectors."]}'

5. Call it as an OpenAI endpoint. The /v1/embeddings route matches the OpenAI schema, so existing client code needs only a new base URL.

   from openai import OpenAI
   
   client = OpenAI(base_url="http://localhost:8080/v1", api_key="-")
   resp = client.embeddings.create(
       model="text-embeddings-inference",
       input="What is deep learning?",
   )
   print(len(resp.data[0].embedding))

That is the full procedure. Once /health returns 200 and /embed returns a vector, wire the endpoint into your vector store. If you index into Postgres, the output drops straight into a column as shown in the pgvector setup guide.

Which Docker Tag Should You Run?

TEI ships a separate image per GPU architecture because the compute kernels are built ahead of time. Running the wrong one is the most common startup failure, so match the tag to your card before anything else.

The generic cuda-1.9 tag in the quickstart works across recent NVIDIA cards and is the safe default when you are not sure. Move to the architecture-specific tag once you know your hardware and want the fastest kernels for it. If you deploy to a mix of machines, pin the tag per host rather than sharing one image.

How Does TEI Compare to the Alternatives?

Serving embeddings is not the only option, and the right call depends on traffic, data rules, and how much you want to operate. Here is the honest trade-off across the four common approaches.

The reason TEI wins for a live retrieval service is the batching. When ten queries arrive at once, TEI merges them into a single GPU forward pass by token count instead of running them one after another, which is where in-process loops fall down under concurrency. For an ingest job that embeds a million rows once, a plain sentence-transformers script is simpler and there is no server to keep alive.

How Do You Add a Reranker?

A reranker sharpens retrieval by scoring each candidate against the query with a cross-encoder, and it fixes the recall-versus-precision gap that pure vector search leaves behind. TEI serves these models on the same binary, so you run a second container and hit /rerank.

docker run --gpus all -p 8081:80 -v $PWD/data:/data \
  --pull always \
  ghcr.io/huggingface/text-embeddings-inference:cuda-1.9 \
  --model-id BAAI/bge-reranker-large

Send the query with the shortlist of texts and TEI returns each text's index and relevance score, sorted best first.

curl 127.0.0.1:8081/rerank \
  -X POST \
  -H 'Content-Type: application/json' \
  -d '{"query":"What is deep learning?","texts":["Deep learning uses neural networks.","Bananas are yellow."]}'

Run the embedding model and the reranker as two ports, then retrieve with the first and reorder with the second. For how the reranking stage fits a full pipeline, see how to add a reranker to your RAG pipeline.

How Do You Tune Throughput and Long Inputs?

The defaults are conservative. Three flags matter once real traffic arrives, and you pass them after --model-id.

• --auto-truncate silently cuts inputs longer than the model's max length instead of rejecting them with a 413. Turn it on if your documents vary in size.
• --max-batch-tokens caps the tokens TEI packs into one forward pass. Raise it to push more work per batch on a large GPU; lower it if you hit out-of-memory errors.
• --max-client-batch-size limits how many inputs a single client request may carry, which protects the server from one caller sending an enormous array.

docker run --gpus all -p 8080:80 -v $PWD/data:/data \
  --pull always \
  ghcr.io/huggingface/text-embeddings-inference:cuda-1.9 \
  --model-id BAAI/bge-large-en-v1.5 \
  --auto-truncate \
  --max-batch-tokens 16384

For gated or private models, pass a read token as an environment variable so TEI can pull the weights.

docker run --gpus all -e HF_TOKEN=$token -p 8080:80 \
  -v $PWD/data:/data --pull always \
  ghcr.io/huggingface/text-embeddings-inference:cuda-1.9 \
  --model-id your-org/private-embedder

In an air-gapped environment, clone the model with git lfs first, mount the folder, and point --model-id at the local path such as /data/bge-large-en-v1.5.

Troubleshooting Common Errors

• Container exits immediately with a CUDA or kernel error. The image tag does not match your GPU. Recheck the table above and switch tags, or fall back to cuda-1.9.
• 401 or a gated-repo message on startup. The model needs authentication. Accept the license on the model page, then pass -e HF_TOKEN=$token.
• 413 payload too large. An input exceeds the model's max sequence length. Add --auto-truncate, or chunk the text before sending. See the local RAG pipeline guide for a chunking pattern.
• CUDA out of memory under load. Lower --max-batch-tokens, or move to a card with more memory. Embedding models are small, so this usually points to an oversized batch setting rather than the model.
• Unsupported model architecture. TEI covers BERT, RoBERTa, XLM-RoBERTa, Nomic, GTE, Qwen2 and Qwen3, ModernBERT, MPNet, and similar encoders. A decoder-only chat model will not load; pick a model from the supported list.

What to Do Next

You have a batching embedding server and, optionally, a reranker on a second port. From here:

• Wire the /embed output into a vector store. The Qdrant hybrid-search guide shows dense and sparse vectors working together.
• Put the endpoint behind your generation model so retrieval and answering share one stack; the vLLM serving guide is the matching piece for the LLM side.
• Scrape TEI's /metrics route into Prometheus to watch batch size, queue time, and throughput before you size the GPU fleet.
• Read the supported models page to confirm your chosen embedder is covered and to see its max input length.

Sources

• Text Embeddings Inference: Quick Tour
• Text Embeddings Inference on GitHub (Docker image tags and supported models)
• Text Embeddings Inference documentation
• NVIDIA Container Toolkit installation guide