Teams that are unhappy with a RAG system almost always start by changing the generation model. It is the most visible component and the easiest one to swap. It is also, in most systems of this kind, not the problem. The model is doing what it was told: synthesizing an answer from the passages it was handed. The passages were wrong.
Retrieval is where answer quality is decided, and it is measured far less often than it is discussed. That gets more expensive the moment the system stops being a chat box. Retrieved context is not only read by a person who can raise an eyebrow at it, it is passed into the next tool call, so a retrieval defect stops being a bad paragraph and becomes an action. What follows is three retrieval layer gaps we see repeatedly when reviewing enterprise data and AI platforms, what each one looks like when it fails, and the evaluation we add to catch it before a user does. They are ordered by how quietly they fail rather than by how hard they are to fix.
Gap one: the answer is split across a chunk boundary
The most common retrieval defect is not that the system retrieved the wrong document. It retrieved the right document and returned the wrong part of it. A chunker that splits on a fixed token count will eventually cut between a rule and its exception, between a table header and its rows, or between a defined term and the clause that qualifies it. Retrieval scores the fragment that mentions the subject, the model receives half the answer, and it completes the other half from its own priors.
In a policy corpus this shows up as answers that state the general rule and omit the carve-out. In product documentation it shows up as a supported configuration described without its version constraint. The output is fluent, specific, and wrong in a way that reads as authoritative. Nobody files it as a retrieval bug, because the cited source genuinely does discuss the topic.
Inside an agent the missing half is worse than a misleading sentence. A carve-out that never reaches the context window is a condition the agent never evaluates, so it does not pause, does not escalate, and proceeds to the tool call as though the exception did not exist. The behavior reads as decisive precisely because the evidence that should have stopped it was truncated two steps earlier.
The evaluation that catches this scores retrieval on its own, before generation is involved. Build a set of real questions with the gold passage marked in the source document, then measure whether the retrieved set fully contains that span. Span coverage is the metric that exposes boundary damage. A document level hit rate will report success, because the correct document was retrieved, and it will keep reporting success while the system quietly loses the qualifying sentence every time.
A cheap secondary signal: hold the question set constant and vary k. If answers change materially between a small k and a large one, the context window is compensating for a chunker that is losing information. Widening the window hides that, it does not fix it. The fix belongs upstream, in chunk boundaries that respect document structure, or in a retrieval strategy that pulls neighboring chunks along with the hit.
Gap two: your queries and your documents do not live in the same space
Embedding search assumes that a question and the passage answering it land near each other in vector space. Production queries break that assumption, because users type short elliptical fragments loaded with internal shorthand and dense embeddings smear exact tokens. Hybrid retrieval with a cross encoder reranker is the standard fix, and our enterprise RAG implementation guide covers that build in detail.
What is usually missing is the measurement that would have told the team it needed either, and that is a sourcing problem before it is a scoring problem. Most teams build the question set by asking a model to generate questions about their own documents, so the retriever passes an exam written in its own handwriting. Sample the real query log instead, then segment the result: recall for short queries, for queries carrying an identifier, and for queries naming an internal product or team. The aggregate hides the failing segment, and that segment is usually the one your most experienced users depend on.
An agent inherits whichever segment nobody is measuring. It will not rephrase a query that came back thin, the way a person does. It takes the top passages as the state of the world and calls the next tool on that basis.
Gap three: the index has drifted away from the source of record
Retrieval indexes go stale in ways that are invisible from outside the system. A document is revised and the index still holds the previous text. A document is withdrawn and its embeddings remain. A policy exists in two versions and both are retrievable, so the reranker settles which one is authoritative on the basis of phrasing similarity. Nothing errors, latency does not change, and the system answers with exactly the confidence it always had, from a version of the truth that was superseded months ago.
This is the gap that does the most damage in agentic AI systems. A person reading a stale answer in a chat interface usually has enough surrounding context to notice that something is off. An agent does not. It passes retrieved context into the next step, and the next step is a tool call. Retrieval errors that are merely embarrassing in a chat product become actions taken against a customer account under a rule that no longer exists.
Three checks cover most of it.
- Reconcile the index against the source of record. Compare document counts and content hashes on a schedule, and alert on divergence instead of waiting for someone to report a wrong answer.
- Keep a canary set. A small number of documents you know were edited recently, paired with questions whose correct answer changed at the edit, run on every index sync.
- Log retrieval provenance with every answer. Document id, version, and the timestamp of the indexed copy. Once version is in the log, staleness stops being a theory and becomes a query you can run.
Deletion deserves its own test. Soft deletes in a source system frequently fail to propagate to a vector index, and a withdrawn document that is still retrievable is worse than a stale one, because the reason it was withdrawn is usually that it was wrong or that someone should not be reading it. Write a test that asserts a withdrawn document is unreachable through retrieval, not merely flagged somewhere upstream. The same test protects permissions, since a document that survives deletion in the index is a document whose access rules are no longer being enforced by anything.
What does a retrieval evaluation harness actually contain?
It contains a golden question set with marked source spans, a retrieval scorer that runs without the generation model, segment level reporting, and a gate in CI. Every change that touches the retrieval path runs it: chunk size and strategy, embedding model, reranker, metadata filters, and the index build itself. Each proposed change reports recall and span coverage before and after, and a regression on any segment blocks the change even when the aggregate improves.
That gate is the point. Retrieval changes trade against each other constantly. A chunk size that improves precision on short factual questions degrades recall on procedural ones. A filter that removes noise removes a document class nobody remembered was in scope. Without a per segment gate, those tradeoffs get made accidentally and discovered months later.
Answer evaluation stays separate and runs downstream of it. When a system is producing bad answers, the first question is whether the retrieved context contained the answer at all. If it did not, no amount of prompt work fixes it, and the team that skipped straight to prompt work will spend weeks proving that.
Where do you start if you have none of this?
Instrument first, tune second. Log the retrieved chunks alongside every answer, with scores and provenance, and you have the raw material for everything above. A few dozen real questions with gold spans marked by someone who knows the corpus is enough to tell you which of the three gaps you have, and the three fail differently enough that the diagnosis is rarely ambiguous.
Then spend in order. Hybrid search first, reranking second, freshness monitoring third, and the generation model last. The generation model is almost never the constraint in a system that is answering confidently and wrongly. The confidence is the model working correctly. The wrongness came in through retrieval.
