A retrieval-augmented generation system can sound confident and still be wrong in three different places. The retriever may fetch the wrong passages, the generator may ignore the passages it was given, or the answer may be accurate but fail to address the question that was actually asked. A single overall score cannot tell you which of those happened. That is why practical RAG evaluation splits quality into separate relationships: question to context, context to answer, and question to answer.
Two vocabularies describe much of that split. TruLens popularised a three-part view often called the RAG triad: context relevance, groundedness, and answer relevance. The open-source RAGAS framework, introduced in a 2023 paper by Shahul Es and co-authors (arXiv:2309.15217, revised in 2025), operationalises a related set as faithfulness, answer relevancy, context precision, and context recall. The names differ slightly, but the diagnostic idea is the same. Measure each edge separately, then read the scores together before you change anything.
Start with what each score actually measures
All of the core scores discussed here run from 0 to 1, with higher being better. None of them is a direct measure of truth in the world. Each one compares two artefacts produced by your system, or compares one artefact with a reference you supply. Keeping those inputs straight is what makes the numbers interpretable.
Faithfulness: is the answer supported by the context?
Faithfulness, called groundedness in the triad vocabulary, asks whether the claims in the generated answer can be supported by the retrieved context. In the RAGAS formulation the answer is first broken into individual claims. Each claim is then checked against the context, and the score is the number of supported claims divided by the total number of claims.
If an answer makes four claims and the context supports three of them, faithfulness is 0.75. That simple arithmetic is useful because you can inspect the unsupported claim directly. Low faithfulness with good retrieval usually points to generation: a prompt that invites the model to elaborate, a model that prefers its parametric knowledge, or instructions that never require the system to say it does not know.
Faithfulness does not prove the retrieved documents themselves are correct. A system can faithfully repeat a wrong or outdated source and score well. That is a data-quality problem, not a grounding problem, and no grounding metric can detect it on its own.
Answer relevancy: does the answer address the question?
Answer relevancy, or answer relevance, asks a different question: leaving the context aside for a moment, does this response actually answer what the user asked? An answer can be fully grounded and still be irrelevant if it summarises the wrong section, answers a neighbouring question, or pads the reply with background the user never requested.
RAGAS estimates this without a reference answer by working backwards. A judge model generates questions that the given answer could plausibly be answering, embeds those generated questions, and compares them with the original question using cosine similarity. The average similarity becomes the score. If the answer is really about the original question, the reconstructed questions should resemble it. If the answer drifted, they will not.
Context precision: was the useful context ranked well?
Context precision grades retrieval, not generation. It asks whether the chunks that are actually useful for answering the question appear in the retrieved set, and whether they are ranked near the top. Ranking matters because most generators weight early context more reliably, because context windows are finite, and because irrelevant chunks are not harmless: they compete for attention and can pull the answer off course.
Conceptually, precision at each rank rewards systems that put a relevant chunk first rather than third. If the only useful chunk is buried below two irrelevant ones, precision falls even though retrieval technically found the evidence. Low context precision therefore points to chunking, embedding choice, top-k settings, metadata filtering, or the absence of a reranking step.
Context recall: did retrieval miss anything needed?
Context recall asks whether retrieval found all of the information needed, not just some relevant material. Unlike faithfulness and answer relevancy, the classic RAGAS version needs a reference answer. The reference is broken into claims, and the score is the share of those claims that can be attributed to the retrieved context.
If the reference contains two essential facts and the context supports only one, recall is 0.5 even if precision is perfect. That distinction matters in production. A system can return a small, clean, highly precise context and still fail the user because a policy exception, a date, or a second document was never retrieved. Precision problems call for better ranking and less noise; recall problems call for broader retrieval, better indexing, query rewriting, or hybrid search.
The triad as a diagnostic table
The value of these metrics is in the pattern, not in any one number. Read them together:
- Low context relevance or precision, anything else: fix retrieval first. The generator never had a fair chance, so tuning the prompt at this stage is wasted effort.
- Good context scores, low faithfulness: fix generation. Tighten the instruction to answer only from the context, require citations, add an explicit abstention path, or test a model that follows grounding instructions more reliably.
- Good context scores, good faithfulness, low answer relevancy: the system is faithfully answering the wrong thing. Look at query understanding, routing, and the prompt template. The retrieved passage may answer a related question the system substituted for the real one.
- Good precision, low recall: retrieval is clean but incomplete. Increase top-k cautiously, improve chunk overlap, add a second retrieval pass, or check whether access controls or metadata filters are silently excluding documents.
- All scores low: do not tune everything at once. Fix the earliest failing stage, re-run the same test set, and only then move downstream.
Build a test set before you tune
Evaluation is only as good as the questions you run. A useful starter set is small enough to inspect by hand and representative enough to cover real use.
- Collect real questions. Use support tickets, search logs, and internal testers rather than inventing only tidy textbook queries. Keep awkward phrasing; users bring it anyway.
- Write a short reference answer for each question where recall or correctness matters. Keep it factual and limited to what the corpus should support.
- Record the expected source when you can: document and section. That makes manual audits of failed cases much faster.
- Mix difficulty deliberately. Include exact-keyword questions, paraphrases, multi-document questions, and a handful of questions the corpus cannot answer. For those negative cases, the correct behaviour is abstention, not a fluent guess.
- Freeze a version. Once a set is used for a comparison, do not quietly edit it between runs. Add new cases as a new version instead.
Fifty well-chosen cases will teach you more than five hundred unexamined ones. For each case the evaluation harness needs four fields: the user question, the generated response, the retrieved contexts in ranked order, and, where available, the reference answer.
Running an evaluation with RAGAS
RAGAS can score a recorded dataset after the fact, which is the simplest integration pattern. Run your pipeline once per question, save the question, response, and retrieved contexts, then score that file. A minimal modern usage follows the pattern shown in the RAGAS documentation, where each metric is constructed with a judge model and applied to a sample:
from ragas.metrics.collections import Faithfulness
scorer = Faithfulness(llm=judge_llm)
result = await scorer.ascore(user_input=question, response=answer, retrieved_contexts=contexts)
The exact class names and dataset helpers vary between RAGAS versions, so pin a version in your project and check its documentation for that release before copying examples from older tutorials. What should not vary is the discipline around the run: use the same judge model for every compared run, record the pipeline version alongside the scores, and keep the per-question results, not only the averages. Averages hide the cases you most need to read.
After scoring, sort by each metric and read the worst ten examples manually. Patterns there — one document that retrieves badly, or answers that add an unsupported final sentence — usually suggest a specific fix.
Limits, costs, and thresholds
Most of these metrics use a language model as a judge, and that brings real constraints. Judge scores can vary between runs and between judge models. A stronger judge is not automatically a fairer one for your domain, and a cheap judge may miss subtle unsupported claims. Validate the judge on a small hand-labelled sample before trusting it as a release gate.
A common cost pattern is offline scoring on a frozen test set for every change, plus sampled scoring on live traffic to catch drift.
Finally, treat published target numbers as examples, not laws. Teams often start by requiring faithfulness to be their highest, most tightly watched score, because unsupported claims are the riskiest failure in many applications. But the right threshold depends on the cost of a wrong answer in your setting. Set an initial baseline from your own first honest run, require new versions not to regress on it, and raise the bar only after you understand which failures remain.
Used that way, faithfulness and answer-relevance metrics stop being a report card and become something more useful: a way to locate a failure in a pipeline, fix the stage that actually failed, and prove the fix on the same questions that exposed it.
