How to Add a Reranker to Your RAG Pipeline

If your retrieval-augmented generation app pulls the wrong passages, no amount of prompt tuning saves the answer. The fix is a reranker: a second-stage model that reads each candidate passage together with the query and scores how well it actually answers it. In one line, over-fetch a wide set of candidates from your vector store, score them with a cross-encoder, then pass only the top few to the model.

What Do You Need Before You Start?

You need a working first-stage retriever already returning candidates by embedding similarity, such as a pgvector table in Postgres or any vector store. You also need Python 3.9 or later and a machine with either a GPU or a CPU that can tolerate a few hundred milliseconds of extra latency per query.

Install one package to follow the local path:

pip install -U sentence-transformers

That pulls in PyTorch and Hugging Face Transformers. For the managed path, install cohere instead and set your API key.

How Do You Add Reranking to Retrieval in Five Steps?

Follow these five steps and you can finish the task from this section alone. The pattern is the same whether you rerank locally or through an API.

1. Load the reranker once at startup. Loading the model per request is the most common performance mistake. Instantiate it as a module-level singleton.

   from sentence_transformers import CrossEncoder
   
   # 0.6B params, multilingual, 512-token input limit
   reranker = CrossEncoder("BAAI/bge-reranker-v2-m3", max_length=512)

2. Over-retrieve from your vector store. Ask for far more candidates than you plan to keep. Where you used to fetch the top 5, fetch 50 to 100. The reranker only helps if the right passage is somewhere in this candidate set.

   query = "How do I rotate a signing key without downtime?"
   candidates = vector_store.search(query, top_k=50)  # your existing retriever
   passages = [c.text for c in candidates]

3. Score and rank the candidates. The rank() method scores every query-passage pair and returns them sorted, with the original index in corpus_id. Set top_k to how many you want to keep.

   ranked = reranker.rank(query, passages, top_k=5, return_documents=True)
   # ranked -> [{'corpus_id': 12, 'score': 7.9, 'text': '...'}, ...]

4. Map back to your source records. Use corpus_id to recover the full candidate object, so you keep metadata like document IDs and URLs for citations.

   top_hits = [candidates[r["corpus_id"]] for r in ranked]

5. Send only the reranked passages to the model. Build your context block from top_hits, not from the raw vector-search output. This is the whole point: fewer, sharper passages give the LLM less room to go wrong.

If you prefer a managed reranker, replace steps 1 and 3 with a single API call. The response returns each candidate's original index and a relevance_score in the 0 to 1 range.

import cohere
co = cohere.ClientV2()  # reads CO_API_KEY

resp = co.rerank(
    model="rerank-v3.5",   # check Cohere's model list for the current version
    query=query,
    documents=passages,
    top_n=5,
)
top_hits = [candidates[r.index] for r in resp.results]

Why Does a Cross-Encoder Beat Raw Vector Search?

Vector search compares two vectors that were computed separately, one for the query and one for each chunk, and ranks by cosine similarity. That is fast because embeddings are precomputed, but similarity and relevance are not the same thing.

A passage can sit close to your query in embedding space and still fail to answer it. Rerankers exist to catch exactly that gap.

A cross-encoder feeds the query and one passage through the network together, so attention can compare their tokens directly and output a single relevance score. That joint pass is why it ranks better, and also why it costs more: you run the model once for every candidate instead of once for the query.

Which Reranker Should You Pick?

The right choice depends on your languages, your latency budget, and whether you want to host a model at all. The table compares four common options.

Start with bge-reranker-v2-m3 if you self-host and your content spans languages. Drop to the MiniLM cross-encoder when every millisecond counts and your corpus is English. Reach for Cohere when you would rather call an endpoint than manage a GPU.

How Many Candidates Should You Rerank?

Retrieve enough that the correct passage is almost always in the set, but few enough to stay fast. Fetching 50 candidates and keeping 5 is a sound starting point. Push the fetch higher only if you measure that the right answer keeps landing outside the candidate window; every extra candidate is one more forward pass.

Remember the 512-token input limit on bge-reranker-v2-m3. If your chunks run longer than that, the model truncates them, so keep chunk sizes modest during ingestion.

How Do You Troubleshoot the Common Failures?

• Latency spikes on every query. You are almost certainly loading the model inside the request handler. Move the CrossEncoder(...) call to startup and reuse the instance.
• Scores look meaningless or unbounded. Raw cross-encoder scores are logits, not probabilities, so a score of 7 is simply higher than a score of 2. Use them for ranking as is; only apply a sigmoid if you need a 0 to 1 value for a threshold.
• Reranking did not improve answers. The right passage was never retrieved in stage one. Reranking cannot recover what vector search missed, so widen the candidate count or fix your chunking and embeddings first.
• CUDA out of memory. Lower the internal batch size with reranker.predict(pairs, batch_size=16), or run the smaller MiniLM model.
• Truncated long passages score poorly. Passages beyond 512 tokens get cut off. Re-chunk to shorter spans or switch to a long-context reranker like the Jina model.

What Should You Do Next?

Measure the change before you trust it. Build a small set of real queries with known correct passages, then compare hit rate at 5 with and without the reranker; an evaluation harness such as the one in this guide to running LLM evals with Promptfoo makes that repeatable.

Once reranking earns its place, tighten the context you build from the top hits. Passing fewer, higher-quality passages is the same discipline that keeps agents reliable, covered in context engineering for long-horizon agents. From there, read the model cards below to tune batch size and precision for your hardware.

Sources

• BAAI/bge-reranker-v2-m3 model card, Hugging Face
• CrossEncoder usage, Sentence Transformers docs
• CrossEncoder API reference, Sentence Transformers docs
• Rerank API reference, Cohere docs
• Best practices for using Rerank, Cohere docs
• FlagEmbedding repository, BAAI