How to Evaluate a RAG Pipeline: Metrics, Test Sets and CI

Updated 2026-10-26 · 12 min read

Measure retrieval and generation separately, build a small golden test set, use a self-hosted LLM judge carefully, and run RAG evaluations on every change.

To evaluate a RAG pipeline you measure two things separately: whether retrieval finds the right passages, and whether the generated answer is faithful to those passages and answers the question. This guide covers the RAG evaluation metrics that matter (recall@k, MRR, nDCG, faithfulness, answer relevance, context precision), how to build a small golden test set, how to use a self-hosted model as an LLM judge without fooling yourself, and how to run all of it in CI.

How to evaluate a RAG pipeline: the short answer

Build a golden set of 50 to 100 real questions, each labelled with the document passages that answer it and a short reference answer. Score retrieval with recall@k and MRR, which are deterministic and cheap. Score answers with faithfulness (is every claim supported by the retrieved context) and answer relevance, using an LLM judge. Run retrieval metrics on every pull request and judge-based metrics nightly or before releases, and compare against a stored baseline instead of an absolute target.

Why retrieval and generation must be measured separately

A RAG answer can fail in two different places, and the fix is different for each:

| Symptom | Likely cause | Where to look | |---|---|---| | Correct answer missing, model says "I don't know" | Retrieval missed the passage | recall@k, chunking, embedding model | | Confident answer with facts not in your documents | Generation ignored or embellished context | faithfulness, prompt, model | | Right passage retrieved but ranked 8th, answer wrong | Ranking or context budget | MRR, nDCG, top-k, reranker | | Answer is accurate but does not address the question | Prompt or model | answer relevance |

If you only score the final answer, a drop could come from either stage and you will not know which change caused it. Evaluate retrieval on its own first. If the pipeline exposes a search endpoint separate from the answer endpoint, as the one in our self-hosted RAG pipeline with Docker Compose guide does, this is easy.

RAG evaluation metrics for retrieval: recall@k, MRR and nDCG

For each question you have a set of relevant passages and the ranked list your retriever returned. Let k be the number of chunks you actually send to the LLM.

| Metric | Formula | What it tells you | |---|---|---| | Hit rate@k | 1 if any relevant passage is in the top k, else 0 | Did retrieval succeed at all | | Recall@k | relevant passages in top k / total relevant passages | How much of the needed evidence reached the LLM | | Precision@k | relevant passages in top k / k | How much noise you send along | | MRR | mean over questions of 1 / rank of first relevant passage | How high the first useful passage ranks | | nDCG@k | DCG@k / ideal DCG@k, with DCG = sum of rel_i / log2(i + 1) | Ranking quality, with graded relevance |

Some practical notes:

• Recall@k is the metric to watch first. If the evidence is not in the context, the generator cannot use it. Measure it at the k you actually use in production.
• MRR matters when k is small or when your prompt puts the most relevant passage first. It only looks at the first hit.
• nDCG is worth it when relevance is graded, for example 2 for "directly answers", 1 for "related background". With binary labels it carries similar information to MRR and recall combined.
• Precision@k is often low by design. With k=5 and one relevant passage per question, the maximum is 0.2. Do not optimize it in isolation.

Label passages, not chunk IDs

The most common mistake in retrieval evaluation is labelling relevant chunks by their ID. The moment you change chunk size, every ID changes and your golden set becomes useless. Label by source document plus a short quote that must appear in the retrieved text. A chunk counts as relevant if it comes from that document and contains the quote. This survives re-chunking, re-embedding and switching vector databases.

A dependency-free retrieval evaluator

# eval_retrieval.py: run against a pipeline that exposes POST /search
import json
import math
import sys

import httpx

K = 5
API = "http://127.0.0.1:8000"


def is_relevant(hit, label):
    return hit["source"] == label["source"] and label["quote"].lower() in hit["text"].lower()


def score(question):
    hits = httpx.post(f"{API}/search", json={"question": question["question"], "top_k": K},
                      timeout=60).json()
    labels = question["relevant"]
    # relevance per rank position: highest grade of any label the hit matches
    rels = [max((l.get("grade", 1) for l in labels if is_relevant(h, l)), default=0) for h in hits]
    found = {i for i, l in enumerate(labels) for h in hits if is_relevant(h, l)}
    first = next((rank for rank, r in enumerate(rels, start=1) if r > 0), None)
    dcg = sum(r / math.log2(i + 2) for i, r in enumerate(rels))
    ideal = sorted((l.get("grade", 1) for l in labels), reverse=True)[:K]
    idcg = sum(r / math.log2(i + 2) for i, r in enumerate(ideal))
    return {
        "hit": 1.0 if first else 0.0,
        "recall": len(found) / len(labels),
        "mrr": 1.0 / first if first else 0.0,
        "ndcg": dcg / idcg if idcg else 0.0,
    }


rows = [json.loads(line) for line in open(sys.argv[1]) if line.strip()]
scores = [score(q) for q in rows]
summary = {m: round(sum(s[m] for s in scores) / len(scores), 3) for m in scores[0]}
print(json.dumps(summary))

The loop uses log2(i + 2) because i is zero-based; rank 1 gets a discount of log2(2) = 1. Run it with python eval_retrieval.py golden.jsonl and store the JSON output per commit.

Answer metrics: faithfulness, answer relevance, context precision

Retrieval metrics need no model. Answer metrics do, because judging whether a sentence is supported by a passage is a language task. The definitions below follow how Ragas, DeepEval and promptfoo describe them; the exact prompts differ by tool.

| Metric | Question it answers | Needs reference answer | How it is usually computed | |---|---|---|---| | Faithfulness (groundedness) | Is every claim in the answer supported by the retrieved context | No | Judge splits answer into claims, checks each against the context, score = supported / total | | Answer relevance | Does the answer address the question | No | Judge or embedding similarity between question and answer (or questions generated from it) | | Context precision | Are the relevant chunks ranked above irrelevant ones | Usually | Judge labels each chunk relevant or not, then a rank-weighted precision | | Context recall | Does the context contain what the reference answer needs | Yes | Judge checks each reference statement against the context | | Answer correctness | Does the answer match the reference | Yes | Judge or claim-level comparison with the reference answer |

Faithfulness is the one to prioritize. It is the closest automatic proxy for hallucination, and it needs no reference answer, so you can also run it on sampled production traffic. Note what it does not catch: an answer can be perfectly faithful to an irrelevant or outdated chunk. That is why retrieval metrics and a reference answer still matter.

Building a small golden test set

A golden set is a list of questions with known good evidence and answers. It does not need to be large to be useful, but it does need to be honest.

Where questions come from. In order of preference: real user questions from logs, questions from support tickets or internal chat, questions written by people who know the documents, and finally LLM-generated questions. Synthetic questions tend to reuse the wording of the source passage, which makes retrieval look better than it is. If you generate them, rewrite them in the vocabulary a user would actually use.

What to cover. Include a spread on purpose:

• Single-fact lookups ("What port does the admin API use?")
• Questions that need two passages combined
• Questions whose answer is in a table or list
• Questions that use different words than the documents
• Questions the documents cannot answer, where the right output is "I don't know"
• Questions where an older document contradicts a newer one

Format. One JSON object per line keeps diffs readable:

{"id": "q017", "question": "How do I rotate the Qdrant API key without downtime?", "relevant": [{"source": "ops/qdrant.md", "quote": "rolling restart", "grade": 2}], "reference": "Set the new key alongside the old one, restart nodes one at a time, then remove the old key.", "tags": ["ops", "multi-step"]}

How big. Treat any single score as an estimate with noise. For a pass/fail metric with true rate p over n questions, the standard error is sqrt(p * (1 - p) / n). At p = 0.8 and n = 50 that is about 0.057, so a difference of a few points between two runs is within noise. Use 50 to 100 questions to catch clear regressions, and grow the set when you need to detect smaller differences. Tag questions by category so you can see which kind regressed.

Keep it versioned in the same repository as the pipeline config, review changes to it like code, and never tune prompts on the same questions you report scores on. Split off a held-out portion if you iterate heavily.

LLM-as-judge with a self-hosted judge model

Using a model you host yourself as the judge keeps documents and questions inside your network and makes evaluation cost a matter of GPU time. It also brings some pitfalls you need to design around.

Known pitfalls

• Self-preference. A model judging its own outputs tends to rate them favourably. Use a different model family for the judge than for generation where you can.
• Small judges are noisy. Claim extraction and verification are harder than they look. Small models miss claims, merge them, or mark obvious contradictions as supported. Use the largest model you can afford for judging, even if generation runs on something smaller.
• Verbosity and position bias. Judges tend to prefer longer answers, and in pairwise comparisons the first or second option depending on the model. Randomize order in pairwise setups and prefer per-claim scoring over holistic 1-to-10 ratings.
• Format failures. Evaluation frameworks expect structured output. Local models sometimes return malformed JSON, which tools may count as a failure or silently skip. Check how many samples were actually scored.
• Non-determinism. Set temperature to 0, pin the judge model tag, and pin the evaluation library version. A judge upgrade changes your baseline.
• Long contexts get truncated. If the judge runs with a small context window, the retrieved context it is judging against may be cut off, and faithfulness drops for reasons unrelated to your pipeline. Size the judge's context for question, context and answer together.

Calibrate against humans once

Before trusting a judge, label 30 to 50 answers yourself as faithful or not, run the judge on the same answers and compute simple agreement (or Cohen's kappa if classes are unbalanced). If the judge disagrees with you on a large share of cases, change the judge model or prompt before relying on its numbers. Repeat this when you change the judge.

Tools: Ragas, DeepEval, promptfoo

All three work with self-hosted models. Pick by where you want the evaluation to live.

| Tool | Language | Strength | Self-hosted judge | |---|---|---|---| | Ragas | Python | RAG-specific metrics, dataset-oriented | Any OpenAI-compatible endpoint, such as Ollama's /v1 | | DeepEval | Python (pytest style) | Test-case assertions, CI integration | Built-in OllamaModel and deepeval set-ollama | | promptfoo | YAML + CLI (Node.js) | Config-driven, compares prompts and providers | ollama:chat: providers as grader |

The APIs of these projects have changed significantly between versions. Pin the version you use and check the documentation for that version; the snippets below follow the current documentation at the time of writing.

Ragas

Recent Ragas versions provide metrics in ragas.metrics.collections that take an LLM created with llm_factory. Pointing it at Ollama's OpenAI-compatible endpoint keeps the judge local:

import asyncio

from openai import AsyncOpenAI
from ragas.llms import llm_factory
from ragas.metrics.collections import Faithfulness

client = AsyncOpenAI(base_url="http://127.0.0.1:11434/v1", api_key="ollama")
judge = llm_factory("qwen2.5:14b", client=client)
faithfulness = Faithfulness(llm=judge)


async def main():
    result = await faithfulness.ascore(
        user_input="How do I rotate the Qdrant API key?",
        response="Restart the nodes one at a time after adding the new key.",
        retrieved_contexts=["... rolling restart ..."],
    )
    print(result.value)

asyncio.run(main())

The older SingleTurnSample and evaluate() API is deprecated in recent versions, so expect to see both styles in examples online.

DeepEval

DeepEval feels like pytest. Configure Ollama as the default judge once with deepeval set-ollama --model=qwen2.5:14b --base-url="http://localhost:11434", or pass a model object explicitly:

# test_rag.py
import json

import httpx
import pytest
from deepeval import assert_test
from deepeval.metrics import AnswerRelevancyMetric, FaithfulnessMetric
from deepeval.models import OllamaModel
from deepeval.test_case import LLMTestCase

judge = OllamaModel(model="qwen2.5:14b", base_url="http://localhost:11434", temperature=0)
golden = [json.loads(line) for line in open("golden.jsonl") if line.strip()]


@pytest.mark.parametrize("item", golden, ids=[g["id"] for g in golden])
def test_answer(item):
    out = httpx.post("http://127.0.0.1:8000/query", json={"question": item["question"]},
                     timeout=300).json()
    case = LLMTestCase(
        input=item["question"],
        actual_output=out["answer"],
        retrieval_context=out["contexts"],
        expected_output=item["reference"],
    )
    assert_test(case, [FaithfulnessMetric(threshold=0.7, model=judge),
                       AnswerRelevancyMetric(threshold=0.7, model=judge)])

Run it with deepeval test run testrag.py. ContextualPrecisionMetric, ContextualRecallMetric and ContextualRelevancyMetric cover the retrieval-side judge metrics and use the expectedoutput field.

promptfoo

promptfoo is the most convenient when you want to compare configurations side by side without writing Python. Point an HTTP provider at your RAG endpoint and use a local model as grader:

# promptfooconfig.yaml
prompts:
  - "{{query}}"

providers:
  - id: http
    config:
      url: http://127.0.0.1:8000/query
      method: POST
      headers:
        Content-Type: application/json
      body:
        question: "{{prompt}}"
      transformResponse: json

defaultTest:
  options:
    provider: ollama:chat:qwen2.5:14b
    transform: output.answer
  assert:
    - type: context-faithfulness
      contextTransform: output.contexts.join("\n\n")
      threshold: 0.7
    - type: context-relevance
      contextTransform: output.contexts.join("\n\n")
      threshold: 0.5

tests:
  - vars:
      query: How do I rotate the Qdrant API key without downtime?
    assert:
      - type: context-recall
        value: Add the new key, restart nodes one at a time, then remove the old key.
        contextTransform: output.contexts.join("\n\n")

Run with npx promptfoo@latest eval locally, or pin a version in CI. Set OLLAMABASEURL if Ollama is not on localhost:11434.

Running evals in CI

Evaluation is only useful if it runs every time something changes: chunk size, embedding model, prompt, model tag, library version. A setup that works well in practice has two tiers:

| Tier | When | What | Gate | |---|---|---|---| | Fast | Every pull request | Retrieval metrics on the full golden set, a few judge checks on a small subset | Fail if recall@k or MRR drops beyond a set tolerance from baseline | | Full | Nightly or before release | All judge metrics on the full set, larger judge model | Report and alert, review manually |

Retrieval metrics are deterministic for a fixed index, so you can gate on them strictly. Judge metrics have noise, so gate on a tolerance below baseline rather than an exact value, and look at per-question diffs before blocking a merge.

A GitHub Actions job for the fast tier can bring up the same Docker Compose stack you run in production, with a small model so it fits a CPU runner:

# .github/workflows/rag-eval.yml
name: rag-eval
on: [pull_request]

jobs:
  retrieval:
    runs-on: ubuntu-latest
    timeout-minutes: 45
    env:
      QDRANT_API_KEY: ci-only-key
      EMBED_MODEL: nomic-embed-text
      CHAT_MODEL: llama3.2:1b
    steps:
      - uses: actions/checkout@v4
      - name: Start stack
        run: docker compose up -d --build --wait
      - name: Ingest fixtures
        run: docker compose exec -T rag python app.py ingest /docs
      - name: Retrieval metrics
        run: |
          pip install httpx
          python eval/eval_retrieval.py eval/golden.jsonl | tee scores.json
          python eval/compare.py scores.json eval/baseline.json --max-drop 0.03

compare.py is a few lines that load both JSON files and exit non-zero if any metric fell by more than the tolerance. Commit baseline.json and update it in the same pull request when a drop is intentional, so the change is reviewed. Model downloads dominate CI time; cache the Ollama models directory between runs or run the job on a self-hosted runner that keeps the volume.

For the nightly tier, run on a machine with a GPU and the same judge model each time, and store the per-question results, not just averages. When a score moves, the per-question diff tells you whether it is one category of questions or noise spread across all of them.

Next steps

• If you do not have a pipeline to evaluate yet, start with the self-hosted RAG pipeline guide or the packaged RAG pipeline stack.
• If recall@k is the weak spot, revisit your embedding model choice before tuning prompts.
• Browse other RAG stacks and the use case library for evaluation ideas in your domain.