Retrieval augmented generation is straightforward to describe. Find text relevant to the question, place it in the context, and have the model answer using it rather than from memory.
Building a version that works on a demonstration set takes an afternoon. Building one that holds up against real questions over real documents takes considerably longer, and the difficulty is concentrated in a part of the pipeline that receives the least attention.
What it is actually for
Worth being precise, because the technique gets applied where it is not the answer.
It suits knowledge the model does not have, such as internal documentation. It suits knowledge that changes faster than any model is retrained. It suits answers that must cite a source. And it suits content that cannot be sent for training but can be retrieved at query time under access control.
It does not improve reasoning, and it does not fix a model that is bad at the task. Supplying better material to something that cannot use it well changes nothing.
Chunking decides more than anything else
Documents are split before indexing, and this is the most consequential decision in the system while receiving the least scrutiny.
Chunks that are too small lose the context that made them meaningful. A paragraph beginning with the word "it" is useless in isolation. Chunks that are too large dilute the retrieval signal, because a long passage matches many queries weakly and none strongly, and it consumes context budget carrying material that is not relevant.
How to split, and how not to
Splitting on a fixed character count is the common default and the common cause of poor results, because it cuts through sentences, tables, and code blocks without regard for meaning. Splitting on structure, using headings, sections, or list boundaries, produces chunks that correspond to units of thought. Modest overlap between adjacent chunks reduces the chance of severing an answer at a boundary.
Attaching the document title and section heading to each chunk is a small change with a large effect, because it restores the context that splitting removed and gives both the retriever and the model something to anchor on.
Similar is not the same as relevant
Embedding search finds text that is semantically close to the query, which is not the same property as text that answers it. A passage discussing the same topic in different terms scores well and may contain nothing useful, while the passage that actually answers the question may be phrased so differently that it scores poorly.
There is a specific and common failure here. Vector search is weak on exact tokens, so product codes, error identifiers, version numbers, and proper nouns retrieve badly, precisely the terms a user is most likely to search for verbatim. Traditional keyword search handles those well and handles paraphrase badly, which is the complementary weakness.
Running both and combining the results addresses the majority of retrieval complaints in practice, and it is usually a larger improvement than changing the embedding model.
Retrieve widely, then narrow
Retrieval and ranking are cheapest when separated. Retrieve a generous set of candidates using fast methods, then apply a more expensive model that scores each candidate against the query directly and keeps only the best few.
The second stage is more accurate because it examines the query and the passage together rather than comparing two independently produced vectors. Applying it to a small candidate set keeps the cost acceptable, and the result is that the material reaching the model is both relevant and compact.
The ceiling nobody works around
If the correct passage is not retrieved, no amount of prompt work recovers the answer. The model receives material that does not contain it and responds anyway, drawing on its general knowledge, and does so with the same confidence it would show if the retrieval had succeeded.
This is why retrieval quality sets the ceiling for the whole system, and why evaluating the pipeline end to end hides the problem. The two stages should be measured separately. For retrieval, whether the passage containing the answer was returned at all. For generation, whether the answer is supported by the passages that were supplied. Conflating them produces a score that moves for reasons nobody can identify.
Questions that no single passage answers
The pipeline described so far assumes the answer exists in some passage waiting to be found. Many real questions are not like that.
A question that requires combining a figure from one document with a definition from another, or that asks how something changed between two versions, or that asks for a count across a set, has no single chunk containing the answer. Retrieval returns passages that each look relevant and the model assembles something plausible from fragments, which is exactly the situation in which a confident wrong answer is produced.
Two things help. Rewriting the question before retrieval, expanding a short query into the several specific lookups it implies, retrieves better material than the original phrasing. And recognising the class of question early, so that aggregation or comparison is answered from structured data where it exists rather than by asking a language model to count passages.
How much to put in front of the model
There is a temptation to pass every retrieved passage, on the reasoning that more context cannot hurt. It can. Additional passages consume budget, dilute attention across material that is not relevant, and increase the chance that the model grounds its answer in the wrong one.
Fewer, better passages generally outperform many mediocre ones, which is the practical argument for reranking. It is worth measuring directly, because the optimum is usually smaller than expected and it changes with the corpus.
Where it goes wrong, and at which stage
| Symptom | Actual stage at fault |
|---|---|
| Answer is confidently wrong | Retrieval returned nothing useful |
| Exact terms never match | Vector search without lexical search |
| Answer ignores a supplied passage | Too many passages, attention diluted |
| Passage found but meaningless | Chunking severed its context |
| Quotes a document the user cannot open | Retrieval not filtered by permissions |
Only the third row is improved by changing the prompt, which is where most debugging effort goes.
The parts discovered late
Permissions are the most serious. An index built across a document store flattens whatever access control that store enforced, and unless retrieval filters by the requesting user's entitlements, the system will cheerfully quote a document that user was never allowed to open. This is a data leak that looks like a feature and is frequently found only in review.
Freshness is the next. Documents are edited and deleted, and an index that is not updated will keep returning content that no longer exists. Deletion in particular has to propagate, or the system becomes a durable copy of material someone intended to remove.
Citation is worth building in from the start rather than adding later. Returning the source alongside the answer lets a reader verify it, and it is the only practical defence against a confident answer assembled from passages that did not support it.
Note: when a retrieval system gives a wrong answer, the instinct is to adjust the prompt. The cause is usually that the right passage was never retrieved, and time spent on wording is time spent on the wrong stage.