Evaluating RAG Without Losing Your Mind

Most RAG projects I've audited have the same shape: a slick demo, a production deployment, and no idea whether a prompt tweak last Tuesday made things better or worse. The moment you can't answer that question, you've lost control of the system. Evaluation is how you take it back.

You are evaluating two systems, not one

The first mistake is treating RAG as a single black box. It isn't. It's a retriever and a generator bolted together, and they fail in completely different ways. A metric that mashes them together will tell you something is wrong but never where.

1. Retrieval — did the right context reach the model at all?
2. Generation — given that context, did the model answer faithfully?

Score them separately. If retrieval is broken, no amount of prompt engineering will save you.

Start with retrieval

Retrieval is the part you can measure honestly because ground truth is cheap. For each question in your golden set, annotate the documents (or chunks) that should be retrieved, then compute the usual suspects:

• Context precision — of the chunks you retrieved, how many were relevant?
• Context recall — of the relevant chunks, how many did you retrieve?

Recall is the one that hurts. A retriever with 60% recall is silently dropping the answer 40% of the time, and the generator will paper over it with a confident guess.

Track these as you change chunk size, embedding model, or top-k. They're the fastest signal you'll get, and they don't depend on a language model to grade.

The golden dataset is the project

The metrics are easy. The dataset isn't. A golden set of question, expected answer, and relevant-documents triples is the single highest-leverage artifact in a RAG project — and nobody wants to build it because it's slow, manual work.

• Start small. Fifty hand-written, high-quality examples beat five thousand synthetic ones.
• Cover the failure modes, not just the happy path. Add the ambiguous queries, the ones with no good answer, and the multi-hop ones.
• Version it like code. Review changes, log who added what and why.

If your eval dataset is a pile of LLM-generated questions graded by an LLM, you have built a machine that flatters itself.

Generation metrics, and the faithfulness trap

Once the context is good, grade the answer. The two I actually trust:

1. Faithfulness — every claim in the answer is supported by the retrieved context. This is your hallucination detector.
2. Answer relevance — the answer actually addresses the question.

Faithfulness is the one that matters in production. A fluent, well-structured answer that invents a single detail is worse than a hedge, because it looks correct.

Use LLM-as-judge, but don't trust it blind

A language model grading another language model is fast, cheap, and biased in ways that are hard to see. It's a useful instrument, not an oracle.

• Calibrate it against human labels on a sample before you believe a single number out of it.
• Watch for self-preference: the judge systematically favours answers in its own style.
• Prefer pairwise comparisons over absolute scores. "A is better than B" is far more reliable than "A is 7.4 out of 10".

A single aggregate score is a vanity metric. Optimise the components; the headline number will follow.

Don't build the dataset before you check what exists

Hand-curating a golden set is the right answer for your domain, but it is not where you start. There is a real, growing catalogue of public RAG benchmarks, and reusing one buys you three things at once: a baseline number, a sanity check on your harness, and a rough idea of where you stand against published results.

The key distinction is what each benchmark measures. Some stress the retriever in isolation; others grade the full retrieve-and-generate loop.

Retriever-only benchmarks

These give you documents, queries, and relevance labels — but no generated answers. They're the right tool for measuring embeddings, rerankers, and chunking in isolation.

Start with BEIR. Its zero-shot design mirrors the most common real situation: you're searching a corpus the embedding model never trained on.

End-to-end RAG benchmarks

These include the generated answer, so you can grade the whole pipeline — not just whether retrieval found the passage, but whether the model used it faithfully.

• RGB — stress-tests the generator with noisy, counterfactual, and negative context. Great for finding out whether your model gets confused when the retrieved docs disagree.
• RAGBench — 100k examples across five domains, explicitly built to address the scale and coverage gaps in earlier benchmarks. The one I reach for when I want a broad, realistic signal.
• ARES — less a dataset than an automated scoring system: it trains tiny classifiers on a small set of human labels to predict context relevance, faithfulness, and answer relevance at scale.

A pragmatic split: use BEIR to pick your retriever, then RAGBench (or a slice of it) to grade the assembled system. Resist the urge to report a single number across all of them — they measure different things, and mashing them together produces a leaderboard entry, not an insight.

Frameworks: pick one and stop shopping

You do not need to build the grading pipeline by hand, and you should not write your own before trying the existing ones. The three I see in production:

• RAGAS — reference-free metrics (faithfulness, answer relevancy, context precision/recall). It grades what you have without requiring a labelled gold answer, which is why it's the most common starting point.
• TruLens — implements the "RAG Triad" (context relevance, groundedness, answer relevance) and ships with a dashboard for tracing runs. Good for observability during development.
• DeepEval — pytest-style assertions and CI integration. The natural fit if you want evals to live inside your existing test suite.

A minimal RAGAS run looks like this — enough to get a number on the board in an afternoon:

from datasets import Dataset
from ragas import evaluate
from ragas.metrics import (
    faithfulness,
    answer_relevancy,
    context_precision,
    context_recall,
)

eval_dataset = Dataset.from_dict({
    "question":     questions,
    "answer":       answers,        # your RAG pipeline's outputs
    "contexts":     contexts,       # the chunks actually retrieved
    "ground_truth": references,     # gold answers, for context_recall only
})

results = evaluate(
    eval_dataset,
    metrics=[faithfulness, answer_relevancy,
             context_precision, context_recall],
)
print(results)

Note which metrics need a gold answer and which don't. context_recall needs ground_truth; the others run reference-free. That matters when you're grading real user questions that nobody has a reference answer for yet.

Make it a regression gate

An eval harness that lives in a notebook is a hobby. The version that matters runs on every pull request, diffs the metrics against main, and blocks the merge when retrieval recall drops two points.

golden set -> retriever + generator -> grade -> diff vs. main -> gate

That loop is the difference between a RAG system you maintain and a RAG system that maintains a quiet vendetta against you.