The retrieval dashboards I inherit all report the same good news, and it is not news I trust any more. Recall@K is climbing, the embedding model was upgraded last quarter, hybrid search is switched on, and the answers are still wrong about one time in five. The chart says the retriever is finding the right passages. The users say the assistant cannot be relied on. Both are true, and the space between them is the whole subject here.
Recall@K measures one thing: whether the passage holding the answer made it into the set you sent to the model. That is necessary, and it is nowhere near sufficient, because a large language model (LLM) is not a database that quietly ignores the rows it does not need. It reads everything in its context, and some of what you hand it makes it answer worse than an empty context would. So the quantity that decides whether a retrieval-augmented generation (RAG) system works is not whether the right passage was retrieved. It is the precision and the ordering of the context that actually reached the model. The levers that move that quantity are the ones that remove and reorder, not the ones that fetch more.
A near-miss hurts more than random noise
Cuconasu et al. (2024) ran what they call the first systematic study of what a retriever should actually fetch for the model downstream of it, varying the relevance of the passages, their position, and how many were included. Two of their findings sit awkwardly together, and the discomfort is the useful part.
The first: "the retriever's highest-scoring documents that are not directly relevant to the query (e.g., do not contain the answer) negatively impact the effectiveness of the LLM." The second: "adding random documents in the prompt improves the LLM accuracy by up to 35%."
Read on its own, each is surprising. Together they rule out the obvious reading, that irrelevant context is simply harmful. Irrelevant context is not the problem. A random paragraph about an unrelated subject is easy for the model to set aside, and in their experiments it did no harm and sometimes helped. The damage comes from a particular kind of irrelevance: the passage that is about the right subject and does not contain the answer. The product from the wrong quarter, the API method with the same name and a different signature, the policy that was superseded last year. It reads as relevant because it is topically close, and it pulls the model toward an answer that is wrong in the way a plausible distractor produces.
Now notice where those distractors come from. A semantic retriever ranks passages by similarity to the query. The passages that are topically close but wrong are, by construction, the ones it scores highest. The better your embedding model, the more convincingly its near-misses resemble the thing you wanted. The retriever is not failing when it surfaces them. It is doing precisely its job, and its job produces the most dangerous inputs the model will see.
This is the first reason Recall@K misleads. The same ranking that places the gold passage at rank three places four convincing near-misses at ranks one, two, four and five, and Recall@5 scores that set as a complete success. The metric counts the passage you wanted. It is blind to the four you did not.
Where the passage lands decides whether the model reads it
Liu et al. (2023) measured how an LLM uses the documents in a long context by moving the one that holds the answer to different positions and watching the accuracy change. On multi-document question answering with twenty documents, GPT-3.5-Turbo scored 75.8% when the answer-bearing document sat first, 53.8% when it sat in the middle, and 63.2% when it sat last. The curve is U-shaped: the model attends to the beginning and the end of its context and reads the middle poorly.
The number that should stop you is the comparison the authors draw to the closed-book setting, where the model is given no documents at all and answers from memory. That baseline is 56.1%. The middle position, 53.8%, is below it. A relevant document, correctly retrieved and placed in the middle of twenty, left the model worse off than handing it nothing.
Follow that through and a comfortable assumption falls over. Retrieving more documents is not free even when one of them is the right one. Every extra passage you pull back to lift recall does two things at once: it adds another candidate near-miss, and it pushes the relevant passage deeper into the context where the model reads it less well. Past some count the set answers worse than no retrieval, and the count is not large. Twenty documents was enough.
This is where reranking earns its place, and not for the reason it is usually sold. Reranking is described as improving relevance, as though the retrieved passages became more relevant by being reordered. They did not; the set is the same. What reranking buys, given the positional effect above, is control over where the one good passage sits. A cross-encoder that lifts the gold passage from rank nine to rank one has not made it more relevant. It has moved it out of the dead zone in the middle and into the position the model actually reads.
The metrics score the list, not the answer
Three retrieval metrics get reported, and they form a ladder. Recall@K asks whether the relevant passage is anywhere in the top K. Mean Reciprocal Rank (MRR) asks how high the first relevant passage sits, rewarding systems that rank it near the top. Normalised discounted cumulative gain (NDCG) asks about the quality of the whole ordering, rewarding relevant passages near the top and discounting those further down.
Given everything above, the ladder points the right way. Recall@K is the weakest, because it is indifferent to position, and position is a lever. MRR and NDCG are better, because they reward putting the relevant passage high, which is also what moves it out of the middle. If you must pick one retrieval metric, pick an ordering-aware one.
But all three share a blind spot that choosing between them does not fix: they score the retrieved list, and the thing you care about is the generated answer. A list with the gold passage at rank one and four hard negatives beneath it scores perfectly on all three. It can still produce a wrong answer, because the model reads the four negatives too, and the near-miss effect from the earlier section is invisible to a metric that only checks where the gold passage sits. The retrieval metrics cannot see the damage done by the passages around the right one.
The metric that can see it is end-to-end: given the context the system actually assembled, did it answer correctly. That is more expensive, because it needs an answer key or a model acting as judge, and a judge has its own failure modes that make it reliable over aggregates and shaky per item. But it is the only number that moves when the quality a user feels moves. I use the retrieval metrics to debug the retriever and an answer metric to decide whether the system got better. A team that conflates the two ships a recall improvement nobody can feel and spends the next sprint wondering why.
The levers that work remove and reorder
Reframe the usual techniques by what they do to the final context rather than by where they sit in a pipeline, and they sort cleanly into precision and ordering.
Metadata filtering is pure removal, and it is the cheapest precision lever there is. Filter the candidate set by source, date, language or tenant before anything is ranked, and the near-misses that filter removes never get a chance to crowd the context. It closes a security hole at the same time, because a passage the caller is not allowed to see should never be a retrieval candidate. Fewer candidates means the ranker and the model both see fewer plausible distractors, which is the thing that was hurting them.
Hybrid search is usually justified as more recall, and that framing undersells it. Its real contribution is precision on exact tokens. A dense embedding is good at concepts and bad at literals: it blurs a product code, an error string, a version number or a surname into the neighbourhood of similar-looking tokens. The BM25 lexical ranking function, the classic keyword scorer, matches those literally. Running both and combining the results is precision for the queries where the exact token is the point, which embeddings alone handle badly. Searching by meaning and searching by token are different instruments, and the queries that need the second are the ones a pure vector search fails silently on.
Reranking with a cross-encoder is the precision-and-ordering lever from the previous section. The embedding retriever is a bi-encoder: it encodes the query and each passage separately, which is what makes it cheap enough to run over a whole corpus. A cross-encoder reads the query and a passage together and scores the pair, which is far more precise and far too slow to run over everything. So it runs over the top N the cheap retriever already returned, promoting the genuinely relevant and demoting the near-misses into positions the model reads less, or out of the context entirely.
Anthropic's contextual retrieval stacks these and reports numbers worth repeating with the caveat that they are a vendor's, measured on the vendor's own evaluation. The metric is one minus recall@20, the share of relevant chunks that fail to appear in the top twenty. Contextual embeddings alone cut that failure rate by 35%, from 5.7% to 3.7%. Adding contextual BM25 took it to 49%, 5.7% to 2.9%. Adding a reranking step took it to 67%, 5.7% to 1.9%. By my own argument the metric is the wrong end of the pipeline, since it scores the list rather than the answer. But the mechanism attacks precision directly: prepending a short description of what a chunk is about before embedding it stops that chunk from sitting near queries it has nothing to do with, which is the near-miss rate falling. And the shape of the result, each precision lever stacking on the last, is the stack this whole piece argues for.
The cheap move and the good move are the same one
There is a reason to care about all of this beyond answer quality, and it is unusual enough to state on its own: the change that improves the answer also lowers the bill. Every document you retrieve to pad recall is input tokens the model pays to read, on every call that carries it, and in an agent loop the same context is resent step after step, so a padded Top-K is paid for many times over. I have argued elsewhere that the reading side of an LLM bill is the side that dominates. A fat retrieved set is one of the fattest parts of it. So trimming Top-K and reranking down to a few precise passages wins twice: fewer tokens on the invoice and a context the model reads better. These two incentives usually fight. Here they point the same way, and that is a lever worth pulling hard.
The ceiling on all of it is set upstream, before any query runs. A near-miss is often not a retrieval failure at all. It is a chunk that should never have existed: a passage that split a sentence down the middle, or carried two hundred words of page furniture ahead of its content, embeds near queries it has no business being near, and no reranker recovers meaning that chunking threw away. The precision you can buy at query time is capped by the precision of the ingest that produced the index, which is a separate problem with its own silent failures. Where that index lives is a further decision, and the vector store you choose constrains what retrieval you can even run. Query-time optimisation is how you make the most of a good index. It is not how you fix a bad one.
What I would defend, and what I would not
Three things I would defend without hedging. Decide whether a retrieval system got better by measuring the answer, not the retrieval, and keep Recall@K, MRR and NDCG for debugging the retriever rather than for grading the system. Spend effort on removal and ordering, filtering and reranking, before spending it on recall, bigger K and better embeddings, because the generator is harmed by inclusion and the positional effect is real. And treat hybrid search and metadata filtering as the precision tools they are, not as ways to retrieve more.
Two things I hold more loosely. The random-noise result is genuinely strange, and I would not act on it by injecting noise into a prompt; I read it as evidence that the composition of the retrieved set dominates the outcome, not as a technique to copy. And the contextual-retrieval figures are a vendor's, on a vendor's benchmark. The mechanism is sound and the direction is right. The magnitude deserves the scepticism any first-party number does.
One thing I expect to change, and it is the boundary of the argument. Lost in the Middle was measured on GPT-3.5-Turbo in 2023. Long-context and reasoning models have moved since, and I have seen the positional dip reduced on current models, not erased. The bias behind it, toward the beginning and end of the context, is not the sort of thing that vanishes with scale, but if a future model reads the middle as well as the ends, the ordering half of this argument weakens. The precision half does not, because the near-miss problem is about what is in the set, not where it sits. A model that reads every position equally well still reads the convincing wrong passage, and still answers as though it were right. Retrieval optimisation survives the shift. It just stops being about position and becomes entirely about what you let into the context in the first place.