The Direct Answer to RAG Evaluation
RAG evaluation metrics should measure three separate outcomes: whether the retriever returned useful evidence, whether the generator answered the user’s question using that evidence, and whether the resulting answer was safe and useful in operation. For retrieval, teams commonly use recall@k, precision@k, mean reciprocal rank, normalized discounted cumulative gain, and context precision or relevance. For generation, they use faithfulness or groundedness, answer correctness, relevance, completeness, and citation accuracy. Production evaluation should also add latency, token cost, refusal quality, toxicity, and task-specific measures such as resolution rate for a customer-support system.
Also worth reading: How Do You Build an AI Pilot Evaluation Framework That Can Survive Production? · Which AI Evaluation Metrics Matter Most for Reliable LLM and Agent Systems? · How Should You Measure AI Evaluation Metrics for Real-World Reliability?
There is no universally correct composite score. A RAG system with medical or legal retrieval requires strict evidence-grounding and human review, while an internal search assistant may be judged mainly by whether users can find the correct document quickly. A practical baseline is to set at least 20 representative evaluation questions, publish the retrieval and generation results separately, and require improvement in both before accepting a model or configuration change. Treat any single score below 80% as a warning rather than proof of failure; it becomes a release gate only when the business has established that risk level. The best metric set is therefore the smallest one that connects offline test performance to a visible production outcome.
How RAG Evaluation Metrics Work
A RAG evaluation begins by dividing a request into measurable stages. The retriever converts a query and knowledge base into candidate passages, the reranker may reorder those passages, and the generator produces an answer from the selected context. Retrieval metrics compare relevant documents or passages with the items returned at each rank, while generation metrics compare the answer with a trusted reference or expert judgment. This staged approach reveals failure causes that a final “correctness” score hides: a low score may originate from poor chunking, an unsuitable embedding model, an absent source document, an overloaded context window, or a prompt instruction that permits unsupported claims.
Human reviewers, deterministic string or database checks, and LLM-as-a-judge methods can all supply labels. LLM judges are convenient because they can score subjective qualities such as clarity or faithfulness, but they can vary with judge model, prompt, temperature, and context. They should therefore be calibrated against expert-rated examples and tested regularly for evaluator drift. MLflow introduced LLM-as-a-judge-style metrics in its 2.8 generation, illustrating that judge-based evaluation had become part of mainstream ML tooling by 2023, but the existence of tooling does not remove the need for validation. Metrics that use “LLM in the loop” still require ordinary software controls: versioning, reproducible settings, documented rubrics, and sampled human audits.
Retrieval Metrics and Evidence Quality
Retrieval metrics answer a narrow but important question: did the system obtain the evidence needed to answer? Precision@5 asks whether most of the first five results are relevant, whereas recall@5 asks how much of the known relevant evidence appeared among them. Neither should be reported alone because a retriever can achieve perfect recall by returning many passages and a misleading score by returning one highly ranked but irrelevant result. Mean reciprocal rank rewards relevant items appearing early, making it useful for search-style interactions. NDCG provides a ranked, graded alternative when some passages fully answer the query while others provide only partial support.
Context precision and context relevance are often evaluated after reranking because candidate generation can perform adequately even when the final prompt receives poor context. A useful production target is at least 90% recall@10 on a stable, expert-curated query set, followed by a context precision target of 70% or higher for a typical factual assistant. These figures are not universal standards; they are starting points that should be adjusted for corpus redundancy and query difficulty. If two passages contain identical approved language, a binary relevance label may understate useful retrieval diversity, while a broad relevance label may overstate precision. Teams should define what counts as relevant before comparing two retrieval systems.
The evaluation dataset must also resemble production traffic. A benchmark containing only clean, short questions can conceal failures involving ambiguous language, spelling errors, conflicting documents, recent events, and multi-turn references such as “that second policy.” Include at least 10% adversarial queries in the test set, then report metrics by category instead of averaging all cases into one number. Stratification exposes a system that achieves 92% overall recall while failing badly on safety-policy or numeric questions. Ground-truth relevance should be reviewed by domain experts, and newly discovered production failures should enter a controlled regression set rather than immediately changing the test set.
Generation, Groundedness, and Task Metrics
Faithfulness measures whether claims in the generated answer are supported by the retrieved context; correctness measures whether the answer agrees with a verified reference or accepted domain knowledge. Those are different tests. An answer can be factually correct but unsupported by the supplied documents, which matters when compliance requires traceability. Conversely, an answer can faithfully repeat misleading retrieved text and therefore be untrustworthy. Citation precision should record whether cited passages actually support each claim, while citation completeness should record whether material claims have citations. For systems expected to abstain, evaluate whether the model correctly refuses when the context lacks sufficient evidence.
Task metrics connect language quality to the application’s purpose. A support assistant can be measured by resolution rate, transfer rate, and the percentage of answers accepted by an agent. A knowledge worker may care more about time saved, successful document finding, and the proportion of decisions supported by approved evidence. A summarization system needs factual consistency, coverage of the source, concision, and omission rate rather than generic relevance. Clear scoring rubrics should define, for example, that 5 means fully correct with sufficient evidence, 3 means a minor omission or repairable error, and 1 means a major unsupported error. Defining 2 and 4 prevents judges from clustering at the extremes and makes results more stable.
Do not ask one judge prompt to produce every metric. Separate evaluation calls for retrieval relevance, claim support, answer correctness, and style reduce prompt interference and make failures traceable. Use a capable, fixed judge model for benchmarking, a low temperature where supported, and blind the judge to system identity whenever possible. Before deployment, have domain experts review 100 or 200 cases and compare judge agreement with human judgment; Cohen’s kappa may be used for categorical labels, while Spearman correlation can assess ranking agreement for numeric scores. A judge agreement target of 0.75 or higher is a reasonable starting point, but high agreement on easy examples is not evidence that the judge handles difficult production cases.
Practical Implementation in Six Controlled Stages
Begin with a representative dataset of 50 to 200 questions, covering frequent use, high-cost failures, and known edge cases. Each item should include the user query, relevant documents or passages, a reference answer where one is objective, metadata for difficulty and risk, and the date on which its ground truth was verified. The current system should then be evaluated without changes to establish a baseline. Record model, prompt, index, embedding, chunking, reranker, temperature, and judge versions, because otherwise a later score cannot be reproduced. A results table should show each metric, category slice, sample count, and confidence interval rather than only a headline score.
Next, diagnose whether retrieval or generation failed. Compare retrieved evidence with the ground truth, inspect reranked context, and only then review the prompt or generator output. Teams often modify a prompt when the actual defect is a missing document or an index built from an obsolete policy. After a proposed change, rerun the fixed suite and a separate held-out set to reduce the chance of optimizing directly to the test questions. A 5% or greater relative improvement on the primary metric, with no material regression in safety or latency, is stronger evidence than a larger gain on a tiny sample. Keep at least 20% of cases out of routine tuning and rotate a small set periodically to detect benchmark aging.
Finally, deploy through a staged process: offline validation, shadow traffic, internal pilot, and limited production exposure. Compare the new system against the incumbent on user acceptance, escalation, latency, token cost, and unresolved incidents, not merely an offline judge score. Set automatic rollback conditions such as a 10% decline in task completion, a 20% rise in unsupported claims, or a p95 latency increase above the service objective. Evaluate on a schedule—daily for fast-changing retrieval systems, weekly for stable ones, and after every material model or corpus change. This cadence turns evaluation into an operating control rather than a one-time benchmark exercise.
Comparison of Evaluation Approaches
No single evaluation method covers every requirement. Human review offers strong subject-matter judgment but is expensive and slow. Deterministic tests are inexpensive and reproducible but work only for objectively verifiable properties. LLM-as-a-judge scales well and handles open-ended text, yet introduces cost, variance, and possible bias. Online behavioral measures are valuable for actual outcomes, but they are confounded by user behavior, seasonality, and imperfect instrumentation. Most production programs need a combination, with automation used for broad monitoring and people used where stakes or ambiguity are high.
| Feature | Offline RAG benchmark | LLM-as-a-judge | Human review | Online experiment |
|---|---|---|---|---|
| Scale | 50 to thousands of cases | Thousands of cases | Usually dozens to hundreds | All eligible traffic |
| Cost | Low to medium | Low to medium per case, but recurring inference cost | Highest per case | Infrastructure and analysis cost |
| Reproducibility | High with frozen data and versions | Moderate to high with fixed model, prompt, and settings | Lower because reviewers vary | Moderate |
| Best use | Regression tests and model comparison | Semantic scoring of relevance, faithfulness, and clarity | Calibration, safety, and ambiguous cases | Task completion and user outcomes |
| Main weakness | Can become unrepresentative | Judge bias, drift, and prompt sensitivity | Slow and costly | Results are observational unless randomized |
Common Measurement Mistakes
The most frequent mistake is averaging all observations into one score. An overall 87% can hide a 45% result on regulated financial queries, and a 2% average change can be statistical noise when the sample is only 30 cases. A second error is treating context recall, answer faithfulness, and business success as interchangeable. A system can retrieve excellent passages but answer them incorrectly, or answer a common question well while failing rare safety cases. Report a metric matrix with slices by language, tenant, document age, query length, and risk level, accompanied by sample size and confidence intervals.
Teams also weaken evaluation by using synthetic questions generated by the same model family being tested. Synthetic data expands coverage but may reproduce training assumptions and miss real vocabulary. Another error is evaluating only the final answer, preventing engineers from locating the failed stage. Reference answers can become outdated after a policy revision, so dates and owners matter. LLM judges can be gamed by answer formatting, verbosity, or repeated claims, and verbose responses may appear more complete unless the rubric penalizes unsupported detail. Finally, a test set that is repeatedly used for prompt optimization becomes a development set rather than an independent estimate of generalization.
Mitigation requires governance rather than a larger tool purchase. Keep immutable test versions, review metric definitions quarterly, and require evidence for threshold changes. Maintain a small “evaluation completeness” review that checks whether major query and failure classes are represented; testing the tests is itself a distinct discipline. Track the share of production incidents covered by regression cases and the percentage of new failure categories added after review. A mature team may cover at least 80% of its known incident classes, but it should not present that number as proof of complete coverage because unknown failure modes remain.
When to Act and What Good Maturity Looks Like
A team should establish RAG evaluation before launch when incorrect information can affect customers, money, compliance, safety, or access to public services. Earlier is also appropriate when a system will handle more than 10,000 monthly requests, serve multiple business units, or rely on frequently updated private documents. Evaluation is less elaborate for a low-risk internal prototype, but even a prototype should record 30 representative cases, five known adversarial cases, and a basic retrieval-versus-generation diagnosis. Waiting for a visible hallucination often means users have already supplied the first evaluation set, though at the cost of avoidable trust and support expense.
Six months after adoption, a credible program has traceable data, versioned metrics, a calibrated judge or deterministic layer, human review, and production monitoring. It should be able to answer which change caused a 7-point regression, which customer segments it affected, and whether the increase was larger than expected sampling variation. Release criteria should be risk-based, with stricter thresholds for unsupported medical, legal, financial, or policy claims than for stylistic errors. Cost and latency belong beside accuracy; a system that improves recall from 76% to 89% but doubles p95 latency from 1.8 to 3.6 seconds may be a poor choice for interactive search.
Maturity does not mean maximizing every metric. Highly precise retrieval can reduce recall, abstention can lower apparent answer coverage, and strict citation rules can increase length. Instead, select trade-offs with accountable owners and validate them against actual user tasks. Publish a short decision record whenever a threshold or model changes, retain at least 90 days of metric history, and revisit the benchmark after major index migrations. By September 2026, organizations should expect more agentic RAG patterns, but the measurement principle remains stable: evaluate the evidence, the generated claims, the user outcome, and the operating cost as separate parts of one system.
A Recommended Metric Set for Most Teams
For a typical factual RAG assistant, begin with recall@5 and NDCG@10 for retrieval, context precision for the reranked passages, and faithfulness plus answer correctness for generation. Add claim-level citation precision and completeness when answers contain references, and measure abstention accuracy on questions intentionally outside the knowledge base. For operations, report task completion, user acceptance or correction rate, p50 and p95 latency, token cost per successful answer, and incident frequency. Evaluate each metric over at least 100 stable cases and 5% sampled live traffic where volume permits; if volume is lower, use periodic manual review rather than pretending a small sample is precise.
A workable initial release rule is retrieval recall@10 of at least 85%, context precision of at least 70%, and a risk-weighted faithfulness score of at least 90% on high-risk cases, while using looser task thresholds for low-risk queries. Compare results with the incumbent and examine slices before approval. These numbers are decision anchors, not industry constants: a precise enterprise policy system may demand 95% faithfulness, while a brainstorming assistant may accept 70%. The durable standard is transparency about thresholds, data, failures, cost, and uncertainty. A modest but reproducible evaluation that guides releases is more useful than an elaborate dashboard that nobody trusts.