Inside this article
A retrieval-augmented generation (RAG) system searches a collection of documents, then gives relevant passages to a language model, which uses them to answer a question. Test whether search finds useful evidence, whether the answer uses it accurately, whether citations point to supporting passages, and how the system responds when evidence is missing or misleading.
One score can hide failures. Track search results, answer quality, support from the retrieved passages, citation support, and how the system handles failures separately. Review examples as well as automated scores.
Map the RAG pipeline
A RAG answer comes from several steps. The search component finds passages and ranks them by relevance. The system puts selected passages into the model’s context, the information the model can use while generating an answer. The model then produces a response, sometimes with citations.
For each test, save the question, the passages search returned and their ranks, the context sent to the model, the answer, and its citations. These records help you locate a failure. If the knowledge base contains the correct passage but search does not return it, retrieval failed. If the passage is in the context but the answer contradicts it, generation failed. If the passage supports the answer but the citation points elsewhere, citation handling failed.
These records also make it easier to see what changed over time. AI observability describes the related practice of recording and inspecting what AI services do.
Build a representative RAG test set
Start with questions that reflect the knowledge base and how people use it. For each question, note whether the collection has enough information to answer, which passages provide relevant evidence, and what a good answer must cover. When possible, label the evidence itself instead of relying only on a reference answer written by a model.
Include questions that require one passage and questions that require evidence from multiple documents. Add realistic wording variations, such as a precise technical question and a less specific version of the same request. If users ask about information that changes, include questions whose answers depend on document dates or versions.
Set aside some test questions for evaluation, rather than using them to tune prompts or retrieval. Repeatedly improving the system on the same examples makes its scores less useful for judging how it handles new questions. Compare versions using the same set-aside questions, and add cleaned-up examples of production failures to later test runs.

Measure retrieval quality
Retrieval tests check whether search returned the evidence needed to answer and ranked it high enough for the model to use. For each query, label the relevant passages so you can compare them with the search results.
Recall at k measures how much of the known relevant evidence appears among the first k search results, where k is the number of results considered. It helps reveal missed evidence. Precision at k measures how many of those results are relevant, which helps identify distracting passages. Mean reciprocal rank (MRR) gives a better score when the first relevant result is near the top. Normalized discounted cumulative gain (nDCG) accounts for both relevance and rank when several results matter. Choose measures that fit how the system uses search results, and inspect the passages instead of relying on scores alone.
If an answer needs two passages but search returns only one, recall will show that evidence is missing, even if the first result looks relevant. If search returns both passages but ranks an irrelevant passage above them, a measure that accounts for rank can show why the model received weaker context.
RAGAS offers context precision and context recall metrics for related checks. Its context precision evaluates whether relevant passages rank above irrelevant ones; its context recall evaluates how much of a reference answer’s information the retrieved context supports. These measures address different retrieval questions. Neither establishes overall answer quality.
External benchmarks can help compare search methods, but you still need to test them on your own data. BEIR evaluated information retrieval across 18 datasets and found BM25, a common keyword-search method, to be a robust baseline. In that benchmark, reranking and late-interaction methods had stronger average zero-shot results, but required more computing resources. These results do not show which method works best for your collection. Compare the options using your own questions and documents.
Check answers and citations
Score answer relevance, factual correctness, and support from the retrieved passages separately. Relevance asks whether the answer responds to the request. Correctness asks whether reliable evidence supports its claims. Grounding asks whether the retrieved passages support those claims. An answer can respond to the request but be wrong, or be correct by coincidence without support from the context.
To check grounding claim by claim, break the answer into factual claims and compare each one with the retrieved evidence. Mark claims the evidence does not support, missing qualifications, and contradictions. Then check whether each source is current and reliable. An outdated or inaccurate document does not prove that a claim is correct.
Check citations claim by claim. When a factual claim should have a citation, confirm that it does and that the cited passage supports the claim. Check that the citation points to the specific evidence, not just a generally related document. Track citations that are missing, irrelevant, broken, or too broad. If the test set labels which passages support each claim, use those labels to measure citation coverage and citation precision.
Citation support does not prove that the model relied on a source while generating its answer. A 2024 study on RAG attribution distinguished citation correctness, where the cited source supports a statement, from citation faithfulness, where the model actually relied on that source. In the study’s experiments, researchers found post-rationalized citations in up to 57% of cases. That result is specific to the study’s setup; it shows why checking support alone cannot reveal the model’s internal basis for a claim.
Test missing and misleading evidence
Include questions the system should answer, answer with qualifications, decline, or clarify. If every test question has an answer, the system gets credit for answering even when evidence is missing.
Pair similar questions that have different evidence conditions. For one, make the answer available and retrievable. For another, keep the answer in the collection but make search miss it. Also test cases where the answer is absent or the evidence is incomplete. Include cases where retrieved passages discuss the topic but do not support the requested conclusion, and cases where sources conflict or have different dates. These tests help distinguish a search failure from a genuine lack of evidence.
A practical test suite should also include:
- Ambiguous questions that need clarification.
- Questions with false assumptions or requests for information outside the collection.
- Multi-document questions where the answer depends on combining passages.
- Evidence the user is not authorized to see.
- Retrieved documents containing instructions that should be treated as untrusted text.
UAEval4RAG groups unanswerable requests into six types, including underspecified questions, false assumptions, nonsensical requests, modality limits, safety concerns, and questions about information outside the database. Use these categories to broaden your tests, and define an acceptable response for each type. For example, a response to missing evidence could state what information is missing, while an ambiguous request could prompt a clarifying question.
If the system passes document text to a model or connected tools, test documents that contain hostile instructions. The instructions may try to override other instructions or expose information. Check whether the system assesses retrieved text as evidence instead of following it as a command. The risks and defenses depend on how each RAG system connects search, answer generation, and tools.

Validate automated scores
Automated evaluators help compare many responses consistently, but their scores are estimates. In RAGAS’s original study, agreement with human judgments varied by measure: 0.95 for faithfulness, 0.78 for answer relevance, and 0.70 for context relevance in a WikiEval pairwise comparison experiment. These results apply to that study’s data and setup. They do not guarantee the same performance on another collection or with another evaluator.
ARES offers another approach: it uses automated judges for context relevance, answer faithfulness, and answer relevance, then uses a smaller human-labeled sample to account for judge errors. Its reported evaluations covered eight knowledge-intensive tasks and used a few hundred human annotations. Compare automated scores with human judgments on a sample, especially for dimensions where errors would be costly.
Review examples with both high and low scores, especially when the automated judge and a human reviewer disagree. A judge might reward plausible wording even when the answer misses an important qualification, or misread evidence in a specialized field. Use human review to check whether the measure catches the failures that matter for your use.
Track evaluation results by category
Report separate results for search, answer correctness and relevance, support from retrieved passages, citations, and cases where the system declines to answer. Break results down further when the consequences differ, such as by question type, document age, or whether an answer needs multiple sources. A strong average can hide poor results in a small but important category.
No single score proves that a RAG system is reliable. Set acceptance criteria based on its purpose and the consequences of errors. A system that answers questions about internal reference material has different requirements from one used to inform legal, financial, or safety decisions. State what each measure cannot show and which cases the test set does not cover.
When search settings, prompts, models, document collections, or citation formatting change, rerun the same tests and compare results by category. Save the settings used for each run so you can link a score change to a specific system version.
Created with AI assistance using published sources. Illustrations are AI-generated. Redbag does not claim hands-on testing or exclusive reporting. Check the linked sources for changing product details.
Our approach ↗

