RAG Systems: Retrieval Quality Is the Whole Game

Where RAG systems fail

When a retrieval-augmented generation system gives a poor answer, the instinct is to blame the model or rewrite the prompt. In practice, most failures originate further back: the right information was never available to the model in the first place.

Either the content was never ingested, the chunk that contained the answer was split in a way that destroyed its meaning, the retrieval step returned a plausible but wrong passage, or the context window was filled with irrelevant material. All four are retrieval problems, and all four are fixable with engineering.

Ingestion and chunking set the ceiling

If the source material is not extracted cleanly, nothing downstream can recover it. Tables flattened into unreadable strings, headers separated from the sections they describe, and scanned documents with no text layer will all produce a knowledge base that looks complete and answers badly.

Chunking is equally consequential. Chunks need to be small enough to retrieve precisely and large enough to remain self-contained. In most real repositories, this means carrying document and section metadata alongside the text so that retrieval can be filtered, results can be grouped, and the model can be shown where a passage came from.

Retrieval, ranking, and context

Vector similarity is a starting point, not a complete retrieval strategy. Keyword search, metadata filters, and reranking layers each solve problems that embeddings alone handle poorly — exact identifiers, product codes, names, and dates.

The final step is what actually reaches the model. More context is not automatically better: a smaller set of well-ordered, relevant passages typically produces better answers, at lower cost and latency, than a context window stuffed with marginal matches.

Evaluate retrieval separately

The practical recommendation is to measure retrieval independently of generation. Build a set of real questions with known source passages, then track whether those passages appear in the top results. This isolates one variable, and retrieval quality is the variable most likely to be broken.

Once retrieval is measured, generation quality can be assessed on a stable foundation — and improvements to chunking, indexing, or ranking can be verified rather than assumed.