Retrieval Evaluation Sets: How Infrastructure Teams Prove Their RAG Layer Returns the Right Passage

Can you name the exact passage your agent should have returned the last time it gave a wrong answer? If the honest answer is no, the incident review that closed with "the model hallucinated" closed on a guess.

Most teams running retrieval-augmented generation in production already have a model evaluation harness — a prompt set, a grader, and a score that moves when the model version changes. Far fewer have the artifact one layer down: a frozen list of questions paired with the specific passages that should have been retrieved to answer them.

That artifact is the retrieval evaluation set, and it turns an unfalsifiable complaint into a number a named engineer can move. Without it, every retrieval defect gets misfiled as a generation defect and routed to whoever owns the prompt.

Why "The Agent Hallucinated" Is Usually A Retrieval Defect

Generation sits downstream of the context window. If the passage that answers the question never entered the prompt, the model was never in a position to be faithful — it was asked to answer from documents that did not contain the answer.

Four distinct failures get reported as the same bug, and telling them apart is the whole operational value of the eval set. Here is what actually breaks:

• The passage does not exist in the corpus. Ingest failed silently three weeks ago, or the document was never in scope. No retriever tuning and no model upgrade will recover it.
• The passage exists but was not retrieved. The embedding placed it outside the top-k neighborhood, or a metadata filter excluded it before scoring. This is precisely the defect recall@k exists to catch.
• The passage was retrieved but ranked below the cut. It came back at rank 14 while your prompt assembler pastes the top 8. That is a reranker and k-budget problem, not a corpus problem.
• The passage was in the prompt and the model ignored it. Only this one is a model defect, and in a mature system it is the rarest of the four.

A faithfulness score on the final answer collapses all four into one failing number. That is how a team spends a quarter swapping model vendors and lands back at the same wrong-answer rate, because the retrieval blind spots underneath the agent were never the model's to fix.

What A Retrieval Evaluation Set Actually Contains

A list of questions in a spreadsheet is not an eval set. The set is the question plus everything required to reproduce the run six weeks later, when the number has moved and nobody remembers what changed:

• Question text and question class. Tag every question as head, long-tail, multi-hop, or deliberately unanswerable. The class is what lets you segment a failure later instead of staring at one average.
• Gold passage IDs. One to three stable chunk identifiers per question, each with a content hash, so you can detect when the underlying text changed out from under the label.
• Corpus snapshot version. The index build ID or ingest timestamp the labels were created against. A recall number without one is not comparable to anything.
• Embedding model and dimension. Pin the exact version — Titan Text Embeddings V2 at 1024 dimensions behaves differently from the same family at 512, and a silent provider-side update will move your numbers overnight.
• Chunker configuration. Chunk size, overlap, and splitter strategy, because a 400-token chunk with 50-token overlap and a 1,200-token chunk with none are two different retrievers wearing the same name.
• Retrieval parameters. The k value, active metadata filters, hybrid weighting between BM25 and dense scores, and the reranker version if one sits in the path.
• Observed result. The rank at which each gold passage appeared, the grader verdict per retrieved passage, and the run timestamp.

Every field is there so that a regression has exactly one suspect instead of six. Store the set in the same repository as the retriever configuration and treat an edit to it as a code change, with review.

How Many Golden Questions, And Where They Come From

The most common reason a retrieval eval never gets built is that teams imagine they need thousands of labeled questions. They need sixty to start.

Question sourcing matters considerably more than question count. Pull from four places, in this order:

• Production query logs. Sample the actual head of your traffic, not the questions your team imagines users ask. The gap between those two sets is usually embarrassing and always instructive.
• Escalations and deflection failures. Every ticket where the agent handed off to a human is a question the system already failed once. These are the highest-yield labels you will ever write.
• Subject-matter expert questions. Ask the two people who know the corpus best for the twenty questions they would use to test a new hire. They will reach straight for the multi-hop cases your logs underrepresent.
• Adversarial out-of-corpus questions. Plausible-sounding questions your documents genuinely cannot answer, which exist to measure whether the retriever returns confident garbage instead of nothing.

Label with two raters and adjudicate disagreements rather than averaging them. Expect 10% to 15% disagreement on long-tail questions — and treat that rate as a finding about corpus ambiguity, not as noise to be smoothed away.

Recall@k And The Metrics That Sit Around It

Retrieval has a well-worn metric vocabulary, and most of it is optional. One metric belongs on the release gate and the rest belong on the tuning dashboard:

Metric	What it measures	Right job for it	What it hides
Recall@k	Whether any gold passage appears in the top k	The release gate	Rank position — a gold passage at 8 of 8 scores the same as one at rank 1
Precision@k	Share of retrieved passages that are relevant	Context-budget and token-cost tuning	Can look healthy while recall collapses on the long tail
MRR	Reciprocal rank of the first gold passage	Reranker tuning	Multi-passage questions, where three chunks are all required
nDCG@k	Graded relevance across all positions	Synthesis questions spanning several documents	Little — it simply costs graded labels to compute
Abstention rate	Share of unanswerable questions that return nothing	Trust and safety review	Invisible to every metric above it

Set k to the number of chunks your prompt assembler actually pastes, not the number your index returns. Measuring recall@20 while the prompt receives 8 chunks produces a comfortable number that describes a system nobody is running.

Passage-Level Grading: Who Decides A Passage Is Right

Binary relevance falls apart on real corpora, where a passage often supports an answer without containing it. Grade on three levels instead: answer-bearing, supporting, and irrelevant.

Answer-bearing means a competent reader could produce the correct answer from that passage alone. Supporting means the passage contributes necessary context — a definition, a date range, a scope limitation — without carrying the answer itself.

Humans label the golden set once. After that, a judge model can grade repeat runs at a fraction of the cost, provided you control it the way you would control any other panel, because position bias and grader agreement have to be measured before a score means anything.

Randomize passage order on every judged run and re-measure agreement against the human labels quarterly. This is the same discipline you already apply to end-to-end agent evaluation, applied one layer earlier in the stack.

Wiring The Eval Set Into The Release Pipeline

An eval set that runs when someone remembers it is a document, not a control. It becomes an operational artifact the moment a specific change triggers it automatically.

Run the full set on any change to the chunker, the embedding model version, the index build, the reranker, the metadata filter logic, or the k value in the prompt assembler. Run it on any ingest batch that adds or replaces more than 5% of the corpus.

For rebuilds large enough to be scary, run the old and new indexes side by side in shadow mode for 48 hours and compare per-question ranks rather than the aggregate alone. A rebuild that holds recall@5 steady while shuffling ranks underneath is telling you something about your embedding pipeline that the average will never say.

Wire the run output into the same place your other signals land, so retrieval quality appears next to latency and error rate in agent observability instead of in a notebook one person owns. If you publish service level objectives for your agents, recall@k on the pinned set deserves a line in that document and a rollback path in agent release management.

What Retrieval Evaluation Catches That Model Evaluation Cannot

The defects below are all invisible to a prompt-level test suite, and all of them have taken down production systems whose model evaluation stayed green. This is the class of problem that the data and retrieval layer quietly owns:

• Chunk-boundary amputation. The answer spans a boundary, so half lives in chunk 41 and half in chunk 42, and neither one alone is answer-bearing. Recall@k reads it as a miss and the fix is overlap, not prompting.
• Metadata schema drift. An upstream system renames a field from doc_type to documentType, your filter silently matches nothing, and retrieval quietly returns a plausible but wrong neighborhood.
• Stale index and ingest lag. The document was updated on Tuesday and the index rebuilt on Friday, so for three days the agent confidently cited a superseded policy.
• Embedding model swaps. A new embedding version changes the neighbor geometry of the entire corpus, so questions that worked for a year now retrieve a different top five. This is the strongest practical argument for model-version pinning.
• Tenant filter over-restriction. An access-control predicate one degree too tight removes the answer-bearing passage before scoring, and the agent answers from whatever survived the filter.
• Near-duplicate crowding. Five revisions of the same template document occupy the whole top five, leaving no room for the one passage that carries the answer.

Each of these is diagnosable in minutes with a golden set and effectively undiagnosable without one. The pattern holds across the rest of AI agent operations: the failure that looks like intelligence is usually plumbing.

Reading The Numbers Without Fooling Yourself

Recall@5 of 0.72 is not a grade. It means that on 28% of your eval questions, the model was handed documents that did not contain the answer and asked to produce one anyway.

Read it segmented or do not read it at all. A blended 0.78 that decomposes into 0.96 on head questions and 0.41 on the long tail describes a system that demos beautifully and fails the specialists who matter most.

Resist the reflex to fix recall by raising k. Going from 5 to 20 chunks might buy three points of recall while quadrupling input tokens on every call — a tradeoff that belongs in the same conversation as the rest of your AI agent cost governance.

Watch the abstention number alongside recall, because the two trade against each other. A retriever tuned to always return its best five passages will score well on answerable questions and feed the model confident garbage on everything the corpus cannot answer.

The First Two Weeks

This is a two-week artifact, not a quarter-long program. Here is the sequence that gets a working gate in place:

1. Pull 60 questions from last month's production logs. Forty from the traffic head and twenty from escalations, sorted by frequency rather than by how interesting they are.
2. Label the gold passages by hand. Two raters, three-level grading, disagreements adjudicated out loud in a single sitting instead of resolved by averaging.
3. Run the set against today's retriever and record everything. Corpus version, embedding model, chunker config, k, and the rank of every gold passage.
4. Pin that run as the baseline. It does not need to be a good number — it needs to be a real number that later runs can be measured against.
5. Add twenty unanswerable questions and measure abstention. Most teams discover their retriever has never once returned nothing, which is itself the finding.
6. Put the run in CI behind the six trigger changes. Chunker, embedding version, index build, reranker, filter logic, and prompt-assembler k.

By the end of week two you will have converted at least one open "the agent hallucinated" ticket into a specific, assignable retrieval defect. That conversion is the entire point of the exercise.

Where To Take This Next

If you are running a RAG layer in production and cannot currently state what your recall@k is, that gap is worth two weeks and not much more. The team at iSimplifyMe builds and operates production retrieval and agent systems across CRM, ticketing, and data warehouse environments every week.

Reach out for a working session. We will pull a golden question set out of your actual query logs, stand up passage-level grading against your current index, and leave you with a baseline recall@k number and a CI gate that holds it.