How to Evaluate a RAG Pipeline With Ragas

A wrong RAG answer tells you nothing about why it's wrong. The retriever might have missed the passage that held the answer, or the model might have ignored a passage it did retrieve. Those are opposite problems with opposite fixes, and from the outside they look identical. Ragas scores retrieval and generation separately so you can tell them apart, and you run it with pip install ragas plus a judge model. This guide covers the install, the dataset shape, the judge setup, and how to read the numbers on your own pipeline.

Key takeaways

The short answer: build an EvaluationDataset from your query, retrieved contexts, and answer, then call evaluate() with a wrapped judge LLM and a list of metrics.
Ragas splits quality into a retrieval layer (context precision, context recall) and a generation layer (faithfulness, response relevancy), so a bad score points at the stage that caused it.
Faithfulness and response relevancy need no ground-truth answer; context recall, context precision with reference, and factual correctness do.
Response relevancy needs an embeddings model in addition to the judge LLM. The others need only the LLM.
Every metric is at least one LLM call per row, so scoring is an API cost and a latency cost, not a free unit test.
Scores drift between runs because the judge is an LLM. Fix the judge temperature at 0 and average over a fixed test set.

What You Need Before You Start

Ragas judges your pipeline's output; it doesn't run the pipeline for you. So you bring the outputs, and Ragas brings the scoring.

Python 3.9 or newer and pip.
An API key for a judge model. The examples use OpenAI, so export OPENAI_API_KEY="sk-...". Any LangChain-compatible chat model works, including a local one served through an OpenAI-compatible endpoint.
For each test question: the question, the contexts your retriever returned, and the answer your pipeline generated. Add a written reference answer for any question where you want retrieval recall or factual correctness.
A test set of 20 to 50 questions to start. Small enough to iterate on, large enough that a single odd row doesn't swing the mean.

How Do You Score a RAG Pipeline With Ragas?

Five steps: install, assemble the dataset, wrap the judge, choose metrics, run. The code below is a complete run you can paste and adapt.

Install Ragas and set your key.

pip install ragas langchain-openai
export OPENAI_API_KEY="sk-..."

Assemble one record per question. Each record carries the four fields Ragas reads: the user's question, the contexts your retriever returned, your pipeline's answer, and an optional reference.

from ragas import EvaluationDataset

dataset = EvaluationDataset.from_list([
    {
        "user_input": "How long is the hardware warranty?",
        "retrieved_contexts": [
            "All hardware ships with a 24-month limited warranty.",
            "Returns are accepted within 30 days of delivery.",
        ],
        "response": "The hardware carries a 24-month limited warranty.",
        "reference": "24 months.",
    },
    # one dict per question in your test set
])

Wrap the judge model and an embeddings model. Ragas talks to models through thin wrappers so the same metric code runs against any provider.

from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from ragas.llms import LangchainLLMWrapper
from ragas.embeddings import LangchainEmbeddingsWrapper

evaluator_llm = LangchainLLMWrapper(
    ChatOpenAI(model="gpt-4o-mini", temperature=0)
)
evaluator_embeddings = LangchainEmbeddingsWrapper(OpenAIEmbeddings())

Choose the metrics that match the data you have. Pick at least one retrieval metric and one generation metric so a low overall score tells you which stage to open.

from ragas.metrics import (
    Faithfulness,
    ResponseRelevancy,
    LLMContextPrecisionWithReference,
    LLMContextRecall,
)

metrics = [
    Faithfulness(),                     # generation, no reference
    ResponseRelevancy(),                # generation, needs embeddings
    LLMContextPrecisionWithReference(), # retrieval, needs reference
    LLMContextRecall(),                 # retrieval, needs reference
]

Run the evaluation and read the per-metric means.

from ragas import evaluate

result = evaluate(
    dataset=dataset,
    metrics=metrics,
    llm=evaluator_llm,
    embeddings=evaluator_embeddings,
)

print(result)
# {'faithfulness': 0.94, 'answer_relevancy': 0.88,
#  'llm_context_precision_with_reference': 0.79, 'context_recall': 0.71}

df = result.to_pandas()   # per-row scores for drilling in

That result object is the whole point. If context recall sits at 0.71 while faithfulness sits at 0.94, your retriever is dropping relevant passages and your generator is behaving. You go fix chunking or add a reranker, not the prompt.

What Do the Metrics Actually Measure?

Each metric answers one narrow question, and the four core ones split cleanly across the two stages of a RAG system. Read them together, never as a single average.

Metric (class)	Question it answers	Layer	Needs reference?	Needs embeddings?
`Faithfulness`	Is every claim in the answer supported by the retrieved contexts?	Generation	No	No
`ResponseRelevancy`	Does the answer address the question that was asked?	Generation	No	Yes
`LLMContextPrecisionWithReference`	Are the useful contexts ranked above the noise?	Retrieval	Yes	No
`LLMContextRecall`	Did retrieval surface everything the answer needed?	Retrieval	Yes	No
`FactualCorrectness`	Does the answer agree with the reference answer?	Generation	Yes	No

Faithfulness Catches Hallucination; Relevancy Catches Evasion

Faithfulness breaks the answer into individual claims and checks each one against the retrieved contexts. A high faithfulness score with a low relevancy score is the classic evasive answer: technically grounded, but not actually responsive to the question. Response relevancy catches that by generating candidate questions from the answer and measuring, through embeddings, how close they land to the real one.

Context Precision and Recall Grade the Retriever, Not the Model

Context recall asks whether the passages needed to support the reference answer were retrieved at all. Low recall means the right chunk never made it into the context window, so no prompt change will save you. Context precision asks whether those useful passages ranked near the top rather than buried under filler, which is exactly the number a reranker moves. If you don't have written references, use LLMContextPrecisionWithoutReference, which grades precision against your generated answer instead of a golden one.

Common Errors and How to Fix Them

Troubleshooting

A metric returns NaN for every row. The row is missing a field that metric needs. Context recall and the reference variants need reference set; response relevancy needs the embeddings argument passed to evaluate(). Fill the field or drop the metric.

Rate-limit or timeout errors on a large set. Ragas fires many judge calls in parallel. Throttle them with a run config:

from ragas import RunConfig
result = evaluate(dataset=dataset, metrics=metrics, llm=evaluator_llm,
                  run_config=RunConfig(max_workers=4, timeout=180, max_retries=5))

Scores move every time you run the same data. The judge is a probabilistic model. Set temperature=0 on the judge, keep the test set fixed, and treat differences smaller than a few points as noise rather than signal.
Faithfulness looks too harsh. It penalizes any claim not literally in the contexts, including correct outside knowledge. That's intended. If your product is meant to answer from general knowledge as well as retrieval, faithfulness is the wrong metric for those questions.
The bill is higher than expected. Each metric is one or more judge calls per row, so four metrics over 50 rows is 200-plus calls. Use a small judge like gpt-4o-mini for routine runs and reserve a larger model for release gates.

What to Do Next

A first score is a baseline, not a verdict. Turn it into a loop:

If context recall is your weak layer, revisit how you split documents. See how to chunk documents for RAG.
If context precision lags while recall holds, the right passages are being retrieved but ranked low. Add a reranker to the pipeline and re-score.
Wire the whole thing into a reproducible harness so scores gate deploys. Pair Ragas with Promptfoo for LLM app evals, and if you're building the retrieval stack locally, start from a local RAG pipeline with LlamaIndex and Ollama.
Read the official RAG evaluation quickstart for the current metric set, and the Langfuse cookbook for logging every score to a trace.