Evaluating RAG: Faithfulness & Relevance

A company deployed RAG for twenty enterprise clients. Some report "great answers." Others report "garbage." With no actual metrics in place, there is no way to know which specific component is failing, or for which clients, or why. "The answers seem good" or "the answers seem bad" is not an evaluation strategy, it is a guess, and it gives you nothing actionable to fix.

The RAGAS framework (Retrieval-Augmented Generation Assessment) is one widely-used way to structure RAG evaluation, and its real value is not any specific number, it is the discipline of separating "is retrieval good?" from "is generation good?" into genuinely distinct, independently measurable questions, rather than one vague, unhelpful "does this feel right?" judgment.

• -Context Recall: of all the information actually needed to answer the question correctly, how much of it did retrieval actually surface? Low context recall means the retrieval stage is the problem, the right information may not even exist anywhere in what was retrieved, no matter how good generation is.
• -Context Precision: of what retrieval did surface, how much of it was actually relevant, as opposed to noise? Low context precision (with high recall) suggests retrieval is finding the right information but burying it among a lot of irrelevant chunks, exactly the assembly problem from earlier in this tier.
• -Faithfulness: does the generated answer only contain claims that are actually supported by the retrieved context, or is the model adding things beyond what the context says (hallucinating on top of real retrieved material)? Low faithfulness with good retrieval metrics means the problem is specifically in generation, not retrieval.
• -Answer Relevance: does the generated answer actually address the question that was asked, even if every individual claim in it is faithful to the retrieved context? A technically faithful answer can still be relevance-poor if it wanders off-topic or answers a slightly different question than the one asked.

The real power of separating these four signals is diagnostic. Low context recall points squarely at ingestion or chunking (the right information genuinely isn't there). Low context precision with good recall points at retrieval ranking or reranking (the information is there, but not surfaced clearly). Low faithfulness with good retrieval scores points at the generation prompt or the model itself (the ingredients were fine, the model still overreached). This is exactly how "some clients report garbage" gets turned from a vague complaint into an actionable root cause, by measuring each stage of the pipeline independently instead of only ever looking at the final answer.

A practical starting point does not need a large team or an elaborate framework: build a small, fixed set of representative question-answer pairs with known-correct answers and known-correct source documents (even 20 to 50 well-chosen examples is a genuinely useful start), run the full pipeline against them regularly, and score each of the four dimensions above, using either a separate LLM call as a judge (an increasingly common pattern) or, where possible, exact-match checks against the known-correct answers. Track these scores over time and treat any regression exactly like a failing test suite, something a change should never be allowed to silently break.