How to Add Contextual Retrieval to Your RAG Pipeline

If your RAG answers go wrong on the retrieval step, the usual culprit is context loss at chunk time: a passage that reads "revenue grew 3%" no longer says which company or which quarter, so your embedding and keyword index have nothing to match a real question against. Contextual retrieval fixes this by prepending a short, LLM-written context line to every chunk before you embed and index it. Anthropic measured this cutting the top-20 retrieval failure rate by 35% with contextual embeddings, 49% once you add a contextual BM25 index, and 67% with a reranker on top.

What You Need Before You Start

This assumes you already have documents split into chunks and an embedding model wired up. Contextual retrieval is a layer you add on top of an existing pipeline, not a rewrite.

• A chunked document set. If you have not settled on a chunking strategy, decide that first; see how to chunk documents for RAG.
• A small, cheap generative model to write the per-chunk context (Claude Haiku works well; a local 7B-8B model also does the job).
• An embedding model for the dense side (Voyage, OpenAI, or a local model).
• A vector store for dense search and a BM25 index for lexical search.
• Optional but recommended: a cross-encoder reranker for the final pass.

How to Contextualize and Index Your Chunks

Work through these seven steps in order. Finish them and your retriever will match on the facts a chunk carries even when the chunk text alone dropped them.

1. Split each document into chunks and keep the parent document text on hand, because you need the whole document as context for each chunk.

   chunks = chunk_document(doc_text)  # your existing splitter

2. Generate a context line for each chunk using Anthropic's prompt, and cache the document so you pay for it once. The cache_control marker on the document block is what makes this affordable.

   import anthropic
   client = anthropic.Anthropic()
   
   DOC = "<document>\n{doc}\n</document>"
   CHUNK = """Here is the chunk we want to situate within the whole document
   <chunk>
   {chunk}
   </chunk>
   Please give a short succinct context to situate this chunk within the
   overall document for the purposes of improving search retrieval of the
   chunk. Answer only with the succinct context and nothing else."""
   
   def contextualize(doc, chunk):
       r = client.messages.create(
           model="claude-haiku-4-5-20251001",
           max_tokens=150,
           messages=[{"role": "user", "content": [
               {"type": "text", "text": DOC.format(doc=doc),
                "cache_control": {"type": "ephemeral"}},
               {"type": "text", "text": CHUNK.format(chunk=chunk)},
           ]}],
       )
       return r.content[0].text.strip()

3. Prepend the context to the chunk so the stored text carries both the situating line and the original passage.

   contextualized = [f"{contextualize(doc_text, c)}\n\n{c}" for c in chunks]

4. Embed the contextualized text (not the raw chunk). This is the contextual embeddings half of the method.

   import voyageai
   vo = voyageai.Client()
   vectors = vo.embed(contextualized, model="voyage-3",
                      input_type="document").embeddings

5. Build a BM25 index over the same contextualized text. This is the contextual BM25 half, and it is what catches exact identifiers, error codes, and names that embeddings blur.

   from rank_bm25 import BM25Okapi
   tokenized = [t.lower().split() for t in contextualized]
   bm25 = BM25Okapi(tokenized)

6. At query time, run both retrievers and fuse the ranks. Reciprocal rank fusion needs no score calibration between the two systems, which is why it holds up across corpora.

   def rrf(dense_ids, sparse_ids, k=60, top_k=20):
       score = {}
       for rank, i in enumerate(dense_ids):
           score[i] = score.get(i, 0) + 1 / (k + rank)
       for rank, i in enumerate(sparse_ids):
           score[i] = score.get(i, 0) + 1 / (k + rank)
       return sorted(score, key=score.get, reverse=True)[:top_k]

7. Rerank the fused top-20 and pass them to the answer model. Send the raw chunk text (not the context line) to the model once retrieval is done, so the answer is grounded in the source passage.

Why Prepend Context Instead of Storing Bigger Chunks?

Bigger chunks dilute the embedding, so a large passage matches everything weakly and nothing sharply. Contextual retrieval keeps the chunk small and targeted while adding just enough surrounding fact to make it findable. The context line answers the questions the chunk cannot: which document, which entity, which period, what came before.

The dense and sparse sides fail in opposite ways, which is the reason to run both. Embeddings understand meaning but miss exact strings; BM25 nails exact strings but misses paraphrase. Contextualizing both indexes, then fusing them, covers each one's blind spot. If you want the fusion side in depth, see how to build hybrid search with dense and sparse vectors.

Keeping the Indexing Cost Down

The cost trap is re-sending the full document for every chunk without caching. With prompt caching, the document tokens are written to cache on the first chunk and read cheaply for the rest, which is how Anthropic lands at about $1.02 per million document tokens. Two rules make the cache hit: keep the document block first with its cache_control marker, and process all chunks from one document back to back so you stay inside the cache lifetime. The mechanics are covered in how to cut LLM API costs with prompt caching.

Troubleshooting Common Failures

• Recall did not improve at all. The usual bug is embedding the raw chunk while only the BM25 side got the context. Confirm both indexes were built from the same contextualized list.
• Indexing cost is far above $1 per million tokens. The cache is not hitting. The document block must come first and carry cache_control, the model and parameters must stay identical across calls, and chunks for one document must run consecutively.
• Context lines invent facts not in the chunk. Cap max_tokens near 150 and keep the "answer only with the succinct context" instruction. A larger budget invites the model to summarize the whole document.
• The document is larger than the model context window. Contextualize each chunk against its own section or parent segment rather than the entire file, so the cached block stays within limits.
• Dense and sparse return different id spaces. Index one canonical chunk list by position and have both retrievers return those positions, so fusion compares like for like.

What to Do Next

Prove the gain before you trust it. Build a small question-to-chunk eval set and measure recall at 20 with and without the context line; a swing you can see in numbers is the only signal that matters. Start with how to evaluate a RAG pipeline with Ragas.

Then add the reranker, which is where the largest single jump in Anthropic's numbers came from. See how to add a reranker to your RAG pipeline. For a runnable reference, Anthropic's cookbook and Together AI's open-model version below both implement the full flow end to end.

Sources

• Anthropic, "Contextual Retrieval in AI Systems" (method, prompt, and failure-rate figures)
• Anthropic cookbook: retrieval-augmented generation guide
• Anthropic docs: prompt caching
• Together AI: implement contextual RAG with open models
• Voyage AI embeddings documentation
• rank_bm25 library (BM25Okapi)