How to Run Large Language Models Locally with Ollama

To run a large language model locally with Ollama, install Ollama, then run one command: ollama run llama3.1:8b. That downloads the model on first use, loads it into memory, and drops you into an interactive chat, with no API key and no data leaving your machine. Everything below expands on that single line: how to install on each platform, how to pick a model that fits your hardware, and how to drive the model from your own code through the local REST API or the OpenAI-compatible endpoint.

Running models on your own hardware is increasingly the default for routine workloads, both for cost and for control. It removes per-token API bills for high-volume tasks and keeps sensitive prompts in-house. For the economic case, see how to shrink the token budget without shrinking the team; for the strategic one, see open source AI's rising impact on enterprises.

What you need

• A supported OS: macOS, Windows 10 22H2 or newer, or Linux.
• Memory that matches the model. As a rule of thumb, a model needs roughly its on-disk size plus a few GB of overhead resident in memory. For the common 4-bit quantization, budget about 0.5 to 0.7 GB per billion parameters, so an 8B model is a 5 to 6 GB download and needs about 8 GB free.
• Optional GPU for speed. NVIDIA (compute capability 5.0+ with a recent driver), AMD (ROCm on Linux, more limited on Windows), or Apple Silicon (Metal). With no supported GPU, Ollama runs on CPU and RAM, which works but is slower.
• Disk space for the weights, typically several GB per model.
• Hugging Face access for gated models. Some popular families require you to request access to the weights before you can use them. See how to access and use gated models on Hugging Face.

Steps: install, pull, run, chat, and call the API

1. Install Ollama

Pick your platform. Each installer sets Ollama up to run as a background service, so the local API is available immediately after install.

macOS: download the .dmg from ollama.com/download and drag the app to Applications. On first launch it offers to create the /usr/local/bin/ollama command-line symlink.

Windows: download OllamaSetup.exe from ollama.com/download and run it. It installs per user and needs no admin rights. To install elsewhere:

OllamaSetup.exe /DIR="d:\some\location"

Linux: use the official install script, which also configures the systemd service:

curl -fsSL https://ollama.com/install.sh | sh

To pin a specific version:

curl -fsSL https://ollama.com/install.sh | OLLAMA_VERSION=0.5.7 sh

On Linux you can confirm the service and read logs with:

sudo systemctl status ollama
journalctl -e -u ollama

Verify the install on any platform:

ollama --version

2. Pull a model that fits your hardware

Models are named model:tag, where the tag selects a variant, usually the parameter size. Omitting the tag defaults to :latest. Browse the full catalog at ollama.com/library and confirm the exact tag and size before you download.

Pull a model without running it:

ollama pull llama3.1:8b

Match the size tag to your memory. The 8b, 70b, and 405b suffixes are parameter counts; larger means more capable and more memory-hungry:

Quantization tags such as q4_K_M (4-bit, the common default), q8_0, or fp16 trade quality for a smaller footprint. If a model does not fit in VRAM, Ollama offloads layers to CPU and RAM and runs slower rather than failing.

3. Run the model and chat

This single command downloads the model if it is not already present, loads it, and opens an interactive session:

ollama run llama3.1:8b

Type your prompt at the >>> cursor. To leave the session, type:

/bye

For a single, non-interactive answer, pass the prompt inline:

ollama run llama3.1:8b "Summarize the CAP theorem in three sentences."

Vision-capable models accept an image path in the prompt:

ollama run llama3.2-vision "What is in this image? /home/you/desktop/chart.png"

4. Manage your local models

These commands cover the day-to-day lifecycle:

ollama ls              # list installed models
ollama ps              # list models currently loaded in memory
ollama stop llama3.1:8b   # unload a running model from memory
ollama rm llama3.1:8b     # delete a model from disk

5. Call the local API

Once Ollama is running, a REST server listens on http://localhost:11434. This is what turns a local chat toy into a building block for applications. Send a chat request with curl:

curl http://localhost:11434/api/chat -d '{
  "model": "llama3.1:8b",
  "messages": [
    { "role": "user", "content": "Why is the sky blue?" }
  ],
  "stream": false
}'

The response is a single JSON object whose answer lives at message.content. Note the "stream": false: both /api/chat and /api/generate stream NDJSON by default, so set this when you want one complete object instead of a token stream.

The local API in depth

Native endpoints under /api

The native API exposes two generation endpoints plus model-management routes. Use /api/generate for single-prompt completion and /api/chat for multi-turn conversations with a messages array.

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.1:8b",
  "prompt": "Why is the sky blue?",
  "stream": false
}'

Useful body fields include system (a system prompt), format ("json" or a JSON schema for structured output), keep_alive (how long to keep the model in memory, for example "5m", or 0 to unload immediately), and options (temperature, top_k, top_p, seed, num_ctx, and more). Management routes let you script model operations:

curl http://localhost:11434/api/tags                        # list installed models
curl http://localhost:11434/api/ps                          # list loaded models
curl http://localhost:11434/api/pull   -d '{"model":"llama3.1:8b"}'
curl http://localhost:11434/api/embed  -d '{"model":"llama3.1:8b","input":"hello"}'

The OpenAI-compatible endpoint at /v1

Ollama also serves an OpenAI-compatible API at http://localhost:11434/v1, covering /v1/chat/completions, /v1/completions, /v1/embeddings, and /v1/models. Point any OpenAI SDK at that base URL. The client requires an API key, but Ollama ignores its value, so pass any string. This makes existing OpenAI code a drop-in local replacement. See the OpenAI compatibility docs for the full surface.

Python usage

Reuse OpenAI SDK code by changing only the base URL and key:

from openai import OpenAI

client = OpenAI(
    base_url='http://localhost:11434/v1/',
    api_key='ollama',  # required by the client, ignored by Ollama
)

chat_completion = client.chat.completions.create(
    messages=[{'role': 'user', 'content': 'Say this is a test'}],
    model='llama3.1:8b',
)
print(chat_completion.choices[0].message.content)

Or hit the native API with nothing but requests:

import requests

r = requests.post("http://localhost:11434/api/chat", json={
    "model": "llama3.1:8b",
    "messages": [{"role": "user", "content": "why is the sky blue?"}],
    "stream": False,
})
print(r.json()["message"]["content"])

There is also an official ollama-python library if you prefer a native client: pip install ollama, then call chat(model=..., messages=[...]).

Customize a model with a Modelfile

A Modelfile bakes a base model, a system prompt, and default parameters into a named, reusable model. Create a file called Modelfile:

FROM llama3.1:8b
SYSTEM "You are a terse senior engineer. Answer in at most three sentences."
PARAMETER temperature 0.4
PARAMETER num_ctx 8192

Build and run it:

ollama create -f Modelfile terse-engineer
ollama run terse-engineer

The full directive reference is in the Modelfile documentation.

Troubleshooting

• Connection refused on port 11434. The server is not running. Start it with ollama serve, or on Linux confirm the service with systemctl status ollama.
• Model runs slowly. It likely does not fit in VRAM and is offloading to CPU. Check ollama ps, and either pick a smaller size tag or a lower quantization such as q4_K_M.
• Out of memory on load. The model exceeds available RAM plus VRAM. Move down a size tier per the memory table above.
• GPU not used on Linux (NVIDIA). Verify the driver with nvidia-smi and confirm it meets the minimum version Ollama requires.
• Model name or tag not found. Tags change often. Confirm the exact model:tag on its library page before scripting against it.
• Verify versions when a feature is missing. Newer parameters and endpoints depend on your build. Check ollama --version and GET /api/version.

What to do next

• Add a browser UI. Put a self-hosted, ChatGPT-style interface in front of Ollama with Open WebUI, which auto-connects to Ollama on port 11434 so non-technical users can chat and pull models from a browser. See the Open WebUI setup guide.
• Build structured output into your app using the format field with a JSON schema for reliable parsing.
• Create task-specific models with Modelfiles so each workload has a named, preconfigured assistant.
• Try larger or specialized variants as your hardware allows, including tool-calling and embedding models from the library.

Sources

• Ollama download
• Ollama model library
• Ollama OpenAI compatibility documentation
• Ollama Modelfile documentation
• Official ollama-python library
• Open WebUI: connecting to Ollama