How to Build a Local RAG Pipeline With LlamaIndex and Ollama

You want an LLM to answer questions from your own PDFs, notes, or docs, and you want that to happen on your machine with nothing leaving it. The short version: run ollama pull llama3.1, install three LlamaIndex packages, point SimpleDirectoryReader at a folder, build a VectorStoreIndex, and call query_engine.query(). That is a working retrieval-augmented generation (RAG) pipeline, fully local, in about a dozen lines.

The rest of this guide gets you from zero to a running pipeline, then explains the parts you will actually tune: the embedding model, persistence, and the errors that show up on the first run.

Key takeaways

The short answer: pull a model with Ollama, install llama-index-llms-ollama and llama-index-embeddings-huggingface, then index a folder and query it.
Ollama serves the generation model over a local HTTP endpoint at http://localhost:11434; nothing touches an external API.
Embeddings and generation are separate models. A small local embedding model like bge-base-en-v1.5 handles retrieval; the Ollama model only writes the final answer.
Persist the index with storage_context.persist() so you build it once and reload in milliseconds instead of re-embedding every run.
The first errors are almost always a stopped Ollama server, an unpulled model, or the default 30-second timeout. All three have one-line fixes.

What You Need Before You Start

Python 3.9 or newer, and a terminal.
Ollama installed and running. On macOS and Windows the desktop app starts the server; on Linux you run ollama serve.
Roughly 6 GB to 8 GB of free RAM for an 8B model. If you have less, use a 3B model (covered in troubleshooting).
A folder of documents to query. Plain text, Markdown, and PDF all work out of the box.

If you have never run a model through Ollama, my walkthrough on running large language models locally with Ollama covers install and first run in detail.

Build the Pipeline in Six Steps

Do these in order. By the end of step 5 you can ask questions against your own files; step 6 makes it fast on every run after the first.

Pull the generation model. This downloads the weights Ollama will serve locally.
```
ollama pull llama3.1
```
Create and enter a project folder, then add your documents. LlamaIndex reads everything in the target directory.
```
mkdir local-rag && cd local-rag
mkdir data
# copy your .txt, .md, or .pdf files into ./data
```
Install the LlamaIndex packages. You need the core library, the Ollama LLM binding, a local embedding model, and the file reader for PDFs.
```
pip install llama-index-core llama-index-llms-ollama \
  llama-index-embeddings-huggingface llama-index-readers-file
```

Configure the models globally. Set the embedding model and the LLM once through Settings so every component uses them. The request_timeout matters: Ollama's default is 30 seconds and a cold model load blows past it.

from llama_index.core import Settings
from llama_index.llms.ollama import Ollama
from llama_index.embeddings.huggingface import HuggingFaceEmbedding

Settings.embed_model = HuggingFaceEmbedding(
    model_name="BAAI/bge-base-en-v1.5"
)
Settings.llm = Ollama(
    model="llama3.1",
    request_timeout=360.0,
    context_window=8000,
)

Load the documents, build the index, and query. VectorStoreIndex.from_documents chunks each file, embeds every chunk with the local embedding model, and stores the vectors in memory.
```
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader

documents = SimpleDirectoryReader("data").load_data()
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine()

response = query_engine.query("What does the onboarding doc say about refunds?")
print(response)
```
Run the file. The first run downloads the embedding model from Hugging Face, so give it a minute. After that, the answer prints with the retrieved context stitched in.

Persist the index so you build it once. Re-embedding on every run wastes time. Save to disk, then reload from disk on the next run.

# after building the index once
index.storage_context.persist("storage")

Reload it later instead of rebuilding:

from llama_index.core import StorageContext, load_index_from_storage

storage_context = StorageContext.from_defaults(persist_dir="storage")
index = load_index_from_storage(storage_context)
query_engine = index.as_query_engine()

That is the whole pipeline. Everything below is depth on the choices inside it.

Why the Embedding Model Is a Separate Choice

A RAG pipeline runs two different models. The embedding model turns text into vectors so retrieval can find the chunks that match a question. The generation model reads those chunks and writes the answer. They are chosen independently, and confusing them is the most common early mistake.

The example above runs bge-base-en-v1.5 in-process through Hugging Face. It is small, fast on CPU, and strong for its size, which is why the official local starter uses it. You can also serve embeddings through Ollama itself if you would rather keep everything behind one server.

Serving Embeddings Through Ollama Instead

Pull an embedding model and swap the two lines in step 4. This keeps the whole stack inside Ollama, at the cost of one more model download.

ollama pull nomic-embed-text

from llama_index.embeddings.ollama import OllamaEmbedding

Settings.embed_model = OllamaEmbedding(model_name="nomic-embed-text")

Install the binding first with pip install llama-index-embeddings-ollama. Here is how the two common local generation models and the two embedding paths compare.

Component	Option	Runs where	Best for
Generation	`llama3.1` (8B)	Ollama server	Balanced quality on 8 GB+ RAM
Generation	`llama3.2:3b`	Ollama server	Low-RAM machines, faster replies
Embedding	`bge-base-en-v1.5`	In Python process	Strong retrieval, no extra server
Embedding	`nomic-embed-text`	Ollama server	One unified local endpoint

What Actually Happens When You Query

Calling query_engine.query() runs four steps in order. Knowing them tells you where to look when an answer is wrong.

Your question is embedded with the same model that embedded the documents.
LlamaIndex compares that vector against every chunk vector and pulls the top matches, two by default.
Those chunks plus your question go into a prompt.
The Ollama model reads the prompt and generates the answer.

If the answer is confidently wrong, the retrieval step is usually the culprit, not the LLM. The right chunk never made it into the prompt. Raise the number of retrieved chunks with index.as_query_engine(similarity_top_k=5), and if quality still lags, reorder results with a reranker. My guide on adding a reranker to a RAG pipeline covers that exact fix.

Troubleshooting the First-Run Errors

httpx.ConnectError or connection refused. The Ollama server is not running. Start the desktop app, or run ollama serve, then confirm it answers: curl http://localhost:11434 should return Ollama is running.
model "llama3.1" not found. You skipped the pull. Run ollama pull llama3.1 and check ollama list.
The query hangs, then times out. A cold model load exceeds Ollama's 30-second default. Set request_timeout=360.0 on the Ollama(...) call, as shown in step 4.
The process is killed or the machine swaps hard. The 8B model does not fit in RAM. Switch to ollama pull llama3.2:3b and set model="llama3.2:3b".
PDFs load as empty documents. Install the file reader: pip install llama-index-readers-file. Scanned PDFs with no text layer need OCR first; LlamaIndex reads text, not images.
Answers ignore obvious content in a file. The chunk was not retrieved. Raise similarity_top_k, and check that the file actually landed in data/ before you built the index.

What to Do Next

The in-memory index is perfect for prototyping and fine for a few thousand chunks. Past that, or once you want the index to survive across services, move the vectors into a real store.

Swap the in-memory store for a database. My walkthrough on setting up pgvector for semantic search in Postgres gives you a durable vector store you likely already run.
Add a reranker to lift answer quality before you touch anything else, using the reranker guide.
Read the LlamaIndex local starter for streaming, chat engines, and metadata filters.
Browse the Ollama model library to trade size for quality once the pipeline is stable.