How to Evaluate Your RAG System in 2026: A Practical Guide With Ragas and DeepEval

If you have built a RAG system, you already know the quiet fear. It answers confidently, the demo looks great, and then someone asks the one question that exposes a hallucination. The fix is not another prompt tweak. It is evaluation. On Reddit, the question comes up constantly: what are the best ways to actually measure whether my RAG works? This tutorial answers that with a working setup.

We will use two mature open source tools, Ragas and DeepEval, to score your retrieval and generation on the four metrics that matter. By the end you will be able to run an evaluation on your own pipeline and read the numbers like a pro.

Why is RAG evaluation so hard?

The short answer is that there is no single correct answer. Unlike a classifier, a RAG response can be right in ten different wordings. Traditional unit tests do not fit. That is why the field moved to LLM based metrics: you use a model to judge the model. It sounds circular, but in practice these judges correlate well with human ratings when you pick the right metrics.

In my experience, the teams that struggle with RAG are not bad engineers. They are measuring the wrong thing. They track token cost and latency, then wonder why users do not trust the answers. Start with faithfulness and retrieval quality, and the trust problem usually shrinks.

What are the four core RAG metrics?

Both Ragas and DeepEval organize around the same foundation. Here is what each one tells you.

Ragas documents these under Retrieval Augmented Generation metrics: Context Precision, Context Recall, Response Relevancy, and Faithfulness, plus extras like Noise Sensitivity and Context Entities Recall. DeepEval wraps the four core ones into a single RagasMetric for convenience.

Step 1: Install the tools

You need Python 3.9 or newer. Install both libraries in a clean environment:

pip install ragas deepeval

Ragas does the metric math. DeepEval gives you a test runner, thresholds, and nice reports. You can use Ragas alone, but DeepEval makes batch evaluation and CI integration far easier.

Step 2: Build a small evaluation dataset

Evaluation lives or dies on the dataset. You need real questions and the context your system retrieved for each one. A minimal record looks like this:

dataset = [
  {
    "question": "What is the refund policy?",
    "answer": "You get a full refund within 30 days.",
    "contexts": [
      "All customers are eligible for a 30 day full refund at no extra cost."
    ],
    "ground_truth": "You are eligible for a 30 day full refund at no extra cost."
  }
]

The ground_truth field is your reference answer. Faithfulness and context metrics do not strictly need it, but answer relevancy and any similarity check benefit from having one. Start with 20 to 50 real questions pulled from your logs. Quality beats quantity here.

Step 3: Run a Ragas evaluation

Here is a complete, runnable example using Ragas directly. It computes faithfulness, answer relevancy, context precision, and context recall for each row.

from ragas import evaluate
from ragas.metrics import (
    faithfulness,
    answer_relevancy,
    context_precision,
    context_recall,
)
from datasets import Dataset

records = [
  {
    "question": q["question"],
    "answer": q["answer"],
    "contexts": [q["contexts"]],
    "ground_truth": q["ground_truth"],
  }
  for q in dataset
]

data = Dataset.from_list(records)
result = evaluate(
  data,
  metrics=[faithfulness, answer_relevancy, context_precision, context_recall],
)
print(result)

Each metric returns a score from 0 to 1. Ragas uses an LLM as a judge under the hood, so set your provider key (for example OPENAI_API_KEY) before running. The output shows the average per metric, which is exactly what you track over time.

Step 4: Run the same check with DeepEval

DeepEval expresses the same idea as test cases. This is handy when you want thresholds and CI gates.

from deepeval import evaluate
from deepeval.metrics.ragas import RagasMetric
from deepeval.test_case import LLMTestCase

test_case = LLMTestCase(
  input="What is the refund policy?",
  actual_output="You get a full refund within 30 days.",
  expected_output="You are eligible for a 30 day full refund at no extra cost.",
  retrieval_context=["All customers are eligible for a 30 day full refund at no extra cost."],
)

metric = RagasMetric(threshold=0.5, model="gpt-4o-mini")
evaluate([test_case], [metric])

The RagasMetric averages Answer Relevancy, Faithfulness, Contextual Precision, and Contextual Recall into one score. If it drops below your threshold, the test fails. Wire that into your pipeline and you get a red alert the moment retrieval regresses.

Step 5: Read the results and act

A number alone changes nothing. Map each low metric to a fix:

• Low faithfulness: the generator is inventing details. Tighten the prompt to “only use the provided context” and shorten the answer.
• Low context precision: relevant chunks are buried. Improve chunk size, add a reranker, or switch the embedding model.
• Low context recall: the facts are not in your index. Fix parsing, add sources, or broaden retrieval.
• Low answer relevancy: the model is not answering the question. Check query rewriting and the system prompt.

The point is to turn a vague “it feels off” into a specific, fixable cause.

Step 6: Make evaluation a habit, not a one off

Run this suite on every change to your index, prompt, or model. Keep a spreadsheet of scores per version. When a new embedding model promises better retrieval, you will know in minutes whether it actually helps, instead of trusting a blog post. For agentic workflows, Ragas also offers Topic Adherence, Tool Call Accuracy, and Tool Call F1, so the same habit scales as you move from RAG to agents.

Common pitfalls and how to avoid them

Frequently asked questions

Do I need ground truth to evaluate RAG?
Not for faithfulness, context precision, or context recall. Those judge the answer against the retrieved context. Ground truth helps with answer relevancy and similarity.

Which is better, Ragas or DeepEval?
They overlap. Ragas is the metric library. DeepEval adds a test runner, thresholds, and reporting. Many people use Ragas metrics inside DeepEval.

Can I evaluate without paying for an LLM judge?
Partly. Some metrics need an LLM judge. You can use a local model via Ollama as the judge to keep it free, though scores may vary from a hosted model.

How many questions do I need?
Start with 20 to 50 real questions. More is better, but real coverage of your use cases matters more than raw count.

When should I move to agent evaluation?
Once your system calls tools or plans steps, add Topic Adherence, Tool Call Accuracy, and Tool Call F1 from Ragas agent metrics.

Step 7: A full end to end example

If you want to copy one block and run it, here is the whole flow with Ragas. It assumes your RAG app exposes a function ask(question) that returns the answer and the retrieved contexts.

from ragas import evaluate
from ragas.metrics import (
    faithfulness, answer_relevancy,
    context_precision, context_recall,
)
from datasets import Dataset

questions = [
    "What is the refund policy?",
    "Do you ship internationally?",
    "How do I reset my password?",
]

rows = []
for q in questions:
    answer, contexts = ask(q)   # your RAG app
    rows.append({
        "question": q,
        "answer": answer,
        "contexts": contexts,
        "ground_truth": "",      # fill if you have it
    })

data = Dataset.from_list(rows)
result = evaluate(
    data,
    metrics=[faithfulness, answer_relevancy, context_precision, context_recall],
)
print(result)

Run this against your production questions and you get four scores per question plus an average. Save the averages to a CSV each week. After a month you will see exactly which change helped and which quietly broke retrieval.

What good scores look like

People ask for a magic number. There is not one, but as a rule of thumb: faithfulness above 0.9 means the generator rarely invents facts, context recall above 0.8 means most needed facts are present, and answer relevancy above 0.8 means the reply stays on the question. Context precision is the noisiest, so treat a drop there as a signal to tune chunking before you panic. The real goal is a stable trend, not a single high score on launch day.

Using a local judge to keep evaluation free

If you do not want to pay per evaluation call, point Ragas at a local model through Ollama. Set the judge in your Ragas evaluator to a hosted Ollama endpoint and the LLM judge runs on your machine. Scores will differ slightly from a hosted model, so pick one judge and stick with it for trend tracking. The comparison only breaks if you swap judges between versions.

References: Ragas available metrics →   DeepEval RAGAS metric →

Updated July 11, 2026.