Retrieval
Did the required document and passage appear within the context actually sent to the model?
RAG evaluation
Test retrieval and generation as separate release risks.
A retrieval-augmented system can return the wrong source, ignore the right source, or produce an answer the source does not support. One end-to-end score hides which control failed.
The direct answer
How do you test a RAG system before release?
Build a golden set that names the required source documents, record the retrieved passages, and score retrieval, evidence support, and answer quality independently before applying one release gate.
A correct-looking answer is not enough. It may have been produced from the wrong evidence or from model memory. A failed answer is not enough either: without the retrieval trace, the team cannot tell whether to change the corpus, search, chunking, reranking, prompt, or model.
Evidence support
Does each material claim follow from the cited passage, with no unsupported addition?
Answer quality
Does the response answer the question completely, obey the task constraints, and escalate when the evidence is insufficient?
Release decision
Do all blocking thresholds hold across representative cases and named high-consequence slices?
Golden set
Build questions around known-correct evidence.
-
Start with real decisions
Use questions from the workflow, including ordinary requests, ambiguous wording, exceptions, conflicting documents, stale documents, and questions the corpus cannot answer.
-
Name the expected source
For each answerable question, record the document version and passage that contain the required evidence. For an unanswerable question, record that retrieval should not manufacture support.
-
Capture the retrieval trace
Store document identifiers, ranks, passages, filters, index version, retrieval settings, prompt version, and model version with every result.
-
Score the stages independently
A retrieval miss blocks the case even if the final wording looks correct. Unsupported generation blocks the case even if the correct document was present.
-
Gate by consequence, not only by average
Set minimums for the whole set and zero-tolerance conditions for prohibited advice, missing escalations, or unsupported claims in high-consequence slices.
Engineering reasoning: a release gate should identify the owner of a regression. Retrieval failures belong to the data and search path; support and answer failures belong to the generation and control path. Combining them too early makes both harder to improve.
Public sources
Primary references for the evaluation boundary
Microsoft Foundry RAG evaluators
Microsoft separates process evaluation of document retrieval from system evaluation of groundedness, relevance, and response completeness. This supports measuring the stages independently. The source documents one vendor's evaluator definitions; it does not establish the thresholds or representative cases for another production system. Read the Microsoft reference (retrieved 2026-08-16).
NIST AI RMF Playbook: Measure
NIST's measurement resources frame assessment as contextual evidence for managing a specific system's risks. The playbook is guidance, not proof that any particular RAG system is reliable or ready to release. Read the NIST resource (retrieved 2026-08-16).
Published method
Agent frameworks for this decision
For agent-based workflows, these documents expand on the decision above. They describe our method, not client results.
Agent Reference Architectures
Inspect the retrieval-grounded pattern: ingestion trust, retrieval authorization, source pointers and separate answer evaluation.
Next decision
Scope the evaluation problem
Describe the system and the failure you need the evaluation harness to catch. Alpha AI will use this context to frame a bounded engineering conversation.
We review your enquiry and reply by email. No commitment to a paid project; scope and terms are agreed before work begins.
Prefer email? dev@alpha-ai.dev