How to Run a LiteLLM Proxy to Unify Every LLM Behind One API

If you run more than one model, you already feel the tax: an OpenAI SDK here, an Anthropic client there, a raw HTTP call to your own vLLM box, and a different API key format for each. A LiteLLM proxy collapses all of that into one OpenAI-compatible endpoint. Install it, write a short config.yaml that lists your models, run litellm --config config.yaml, and every app now points at http://0.0.0.0:4000 and speaks the OpenAI Chat Completions format no matter what runs behind it.

Key Takeaways

The short answer: pip install 'litellm[proxy]', list your models in config.yaml, run litellm --config config.yaml, and call the proxy at http://0.0.0.0:4000 with any OpenAI client.
One gateway fronts 100+ providers (OpenAI, Azure, Anthropic, Bedrock, Ollama, vLLM) in a single format, so your application code never imports a provider SDK.
Give two entries the same model_name and the proxy load-balances across them; add a fallbacks block and it retries a dead provider on a healthy one.
Never hardcode keys in the YAML. Use os.environ/VAR_NAME so credentials load from the environment at startup.
Set a master_key and attach a Postgres database to mint per-team virtual keys with their own budgets and model access.

What You Need Before You Start

This is a light setup. You need Python 3.10 or newer, at least one working provider key (or a local model endpoint), and a terminal.

Python 3.10+ and pip.
One or more upstream targets: a hosted key (OpenAI, Azure, Anthropic), a Bedrock role, or a local server such as Ollama or vLLM.
Optional: a Postgres database, only if you want virtual keys, budgets, and the admin UI. The core proxy runs fine without one.

Run a LiteLLM Proxy in Five Steps

Follow these in order and you will have a working gateway in a few minutes. Everything the reader needs to finish sits in this section.

Install the proxy extra, not just the library:
```
pip install 'litellm[proxy]'
```
Export your provider keys as environment variables so they never live in a file:
```
export OPENAI_API_KEY=sk-...
export ANTHROPIC_API_KEY=sk-ant-...
```

Write a config.yaml. Each entry maps a public model_name your apps will call to a real provider under litellm_params. The os.environ/ prefix reads the variable at startup:

model_list:
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY

  - model_name: claude-sonnet
    litellm_params:
      model: anthropic/claude-sonnet-4-20250514
      api_key: os.environ/ANTHROPIC_API_KEY

  - model_name: local-llama
    litellm_params:
      model: ollama/llama3.1
      api_base: http://localhost:11434

Start the server. It binds to port 4000 by default:
```
litellm --config config.yaml
```

Send a request. Any OpenAI-compatible client works; just point base_url at the proxy and switch the model string to swap providers:

curl http://0.0.0.0:4000/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "claude-sonnet",
    "messages": [{"role": "user", "content": "what model are you?"}]
  }'

Or from the OpenAI Python SDK, unchanged except for two lines:

import openai

client = openai.OpenAI(api_key="anything", base_url="http://0.0.0.0:4000")

resp = client.chat.completions.create(
    model="local-llama",
    messages=[{"role": "user", "content": "test request"}],
)
print(resp.choices[0].message.content)

That is the whole loop. Change one string in the request and the same code hits GPT-4o, Claude, or a model on your own hardware.

How Do You Add Load Balancing and Fallbacks?

This is where a gateway earns its keep. Two features cover most reliability needs, and both live in the same config file.

Balance Load Across Identical Deployments

List the same model_name more than once. The router spreads traffic across every entry, and per-deployment rpm and tpm caps weight the split so a bigger box takes a bigger share:

model_list:
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: os.environ/OPENAI_API_KEY
      rpm: 200
  - model_name: gpt-4o
    litellm_params:
      model: azure/gpt-4o
      api_base: https://my-azure.openai.azure.com/
      api_key: os.environ/AZURE_API_KEY
      rpm: 100

Fail Over When a Provider Breaks

A fallbacks block reroutes to a healthy model after a call fails its retries. Pair it with context_window_fallbacks to catch prompts that overflow a model's limit:

litellm_settings:
  num_retries: 2
  fallbacks:
    - "gpt-4o": ["claude-sonnet"]

router_settings:
  routing_strategy: "simple-shuffle"
  timeout: 30

The routing strategy controls how the proxy picks among equal deployments. Pick by what you optimize for:

Strategy	How It Picks	Best For
`simple-shuffle`	Weighted random by `rpm`/`tpm`	Default; spreading load with almost no overhead
`least-busy`	Fewest active requests in flight	Bursty traffic and uneven request cost
`usage-based-routing`	Deployment with the most remaining TPM headroom	Staying under provider rate limits
`latency-based-routing`	Lowest measured response time	Latency-sensitive, user-facing calls

How Do You Lock It Down With Keys and Budgets?

An open proxy on a shared network is a liability. Two moves fix that.

First, require a key on every call by setting a master key. Put it in the config or, better, in the LITELLM_MASTER_KEY environment variable:

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY

Now every request must carry Authorization: Bearer <master-key>. Second, attach a Postgres database and mint scoped virtual keys so each team or app gets its own budget and model allowlist without ever seeing your provider keys:

export DATABASE_URL=postgresql://user:pass@host:5432/litellm

curl http://0.0.0.0:4000/key/generate \
  -H 'Authorization: Bearer sk-your-master-key' \
  -H 'Content-Type: application/json' \
  -d '{"models": ["gpt-4o"], "max_budget": 100, "duration": "30d"}'

The response returns a sk-... key scoped to those models and that spend. When the budget runs out, the proxy rejects the key. This is how you hand access to a team without handing out the raw OpenAI credential.

Troubleshooting Common Errors

"No module named litellm.proxy" or the --config flag is unknown. You installed the base package. Reinstall with the extra: pip install 'litellm[proxy]'.
Requests return a 401 you did not expect. A master key is set, so the call needs Authorization: Bearer <key>. Add the header, or unset the key for local testing.
A provider rejects an unsupported parameter (for example a reasoning or logprobs field). Add drop_params: true under litellm_settings so the proxy strips arguments the target model does not accept.
Ollama or vLLM calls time out. Confirm the api_base points at the running server and that the model name after the provider prefix matches what that server actually serves.
Virtual key generation returns an error about the database. The /key/generate route needs a reachable Postgres. Check DATABASE_URL and that the proxy logged a successful DB connection at startup.

What to Do Next

Once the gateway is live, wire it into the rest of your stack:

Ship it in a container for anything beyond local use. The official image and Helm chart are in the LiteLLM deployment guide.
Turn on observability by adding success_callback: ["langfuse"] under litellm_settings, then follow the same tracing playbook from how to trace and debug LLM apps with Langfuse.
Point the proxy at your own hardware. Stand up a backend first with vLLM for high-throughput inference or Ollama for local models, then register it as one more entry in model_list.
Move routing logic out of your app. The gateway makes the cost and quality trade-offs in LLM model routing a matter of config, not code.