Retrieval Is the Product: Engineering a RAG Pipeline That Holds Up

A polished answer can hide weak retrieval. That is the dangerous part of building with large language models: fluent language makes an incomplete evidence set look authoritative. In a production RAG system, the generator is only the final participant in a much longer chain. The product quality is established when the right evidence is collected, ranked, compressed, and presented with enough provenance for a reader to verify it.

The useful mental model is not “search plus a prompt.” It is an evidence pipeline with measurable handoffs. Documents become structured units, a query becomes several retrieval signals, candidates compete for a limited context budget, and the answer is evaluated against its sources. When teams treat each handoff as a product surface, failures become diagnosable instead of mysterious.

Start with document structure, not a chunk-size constant

Fixed-size chunks are a reasonable baseline, but they are not a content strategy. Contracts, support articles, policies, and product documentation each have different semantic boundaries. A heading, table row, exception clause, or troubleshooting step may carry more meaning than an arbitrary token window. Preserve those boundaries first, then use overlap only where an idea genuinely crosses them.

Every stored unit should retain useful metadata: source, section hierarchy, owner, effective date, audience, access scope, language, and a stable location for citation. This metadata improves filtering and makes stale or unauthorized evidence visible. Without it, a high similarity score can still return the wrong version of the right sentence.

Retrieve broadly, then rank deliberately

Dense embeddings are good at semantic similarity; lexical search is often better for identifiers, error codes, names, and exact phrases. A practical system combines both and treats the first pass as candidate generation. Approximate vector indexes such as HNSW help keep that pass responsive at scale, but the fastest nearest neighbors are not automatically the best evidence for the user’s actual intent.

Reranking is where the system can compare each candidate with the complete query. It costs more than initial retrieval, so apply it to a controlled shortlist. Deduplicate near-identical passages, reward source diversity when the question needs it, and reject weak candidates instead of filling the context window simply because space remains. More context often creates more contradiction, not more confidence.

Build context as an evidence package

Context assembly should be deterministic enough to inspect. Group related passages, include the minimum surrounding text needed to interpret them, and attach stable citation labels before sending anything to the model. If two sources disagree, preserve that disagreement. The model should not silently choose a winner where the underlying knowledge base is inconsistent.

The answer contract should also be explicit: use only supplied evidence for factual claims, cite each important claim, state when evidence is insufficient, and distinguish a source statement from an inference. These rules do not eliminate hallucination, but they turn uncertainty into a visible product behavior rather than a hidden model failure.

Evaluate retrieval before judging generation

Create a test set from real questions, including vague wording, abbreviations, conflicting documents, and questions that should not be answered. Label the evidence a strong answer would need. Then measure whether retrieval found it, whether reranking promoted it, whether the context preserved it, and whether the final citations support the response. A single answer score cannot tell you which stage failed.

Track online signals too: reformulated searches, citation opens, escalation to a person, abandonment, and successful task completion. Review low-confidence traces regularly. The goal is not a beautiful demo with one perfect query; it is a system that can explain why it succeeded, fail honestly when it lacks evidence, and improve from observed misses. In that sense, retrieval is not supporting infrastructure. Retrieval is the product.