Designing a Production RAG Retrieval Pipeline
A practical architecture for hybrid retrieval, reranking, access-aware context assembly, evaluation, and cost control in production RAG systems.
Most RAG failures are not generation failures.
A language model can produce a fluent, well-structured answer and still be wrong because the retrieval layer selected the wrong evidence, missed the relevant passage, ranked useful information too low, or filled the context window with redundant text.
Once weak evidence reaches the generator, prompt engineering can compensate for only so much.
The more reliable approach is to treat RAG as an information retrieval system with an LLM near the end of the pipeline, rather than as a single:
embed -> search -> generateoperation.
A production retrieval system should separate recall, precision, context construction, generation, security, and observability into explicit stages.
The guiding principle is:
Use inexpensive, high-recall stages to discover candidates. Spend expensive compute only where it can materially improve the final evidence set.
The pipeline below is the shape I work with on a production legal-AI retrieval system; the GANI.AI case study describes that work at publishable scope.
The architecture
A useful production pipeline looks closer to this:
The generator is intentionally near the end.
It should receive a small evidence set that has already passed through retrieval, ranking, security, and context-selection decisions.
That changes the role of the LLM from:
find something relevant in all of this text
into:
answer using this carefully selected evidence.
Retrieval is two systems
Production RAG usually contains two related but operationally different systems:
- an offline ingestion path;
- an online query path.
They should not be treated as one application lifecycle.
The ingestion system determines what can be found.
The query system determines what should be used for this request.
That distinction becomes important when documents, embedding models, indexes, and application traffic evolve independently.
1. Prepare documents for retrieval, not storage
Document ingestion is not simply loading files and splitting every 1,000 characters.
Retrieval quality begins before the first query is executed.
The ingestion path typically needs to handle:
- parsing;
- normalization;
- structural segmentation;
- metadata extraction;
- access-control metadata;
- document versioning;
- chunk identifiers;
- embedding generation;
- indexing.
A practical flow is:
The unit stored in the retrieval index should represent a meaningful unit of information.
A paragraph, procedure step, policy clause, API section, or legal provision may be a much better retrieval unit than an arbitrary fixed-length slice.
Chunking therefore belongs to information retrieval design rather than generic text preprocessing.
Give every chunk a stable identity
A production index should not rely on text alone to identify content.
Useful metadata often includes:
document_id
document_version
chunk_id
title
section
source
tenant_id
access_scope
created_at
updated_at
embedding_model
embedding_versionStable identifiers make reindexing, deletion, deduplication, evaluation, and debugging substantially easier.
They also make it possible to answer a basic operational question:
Which exact piece of source knowledge produced this answer?
2. Version embeddings and indexes
Embeddings should be treated as versioned data.
Changing the embedding model without tracking its version can silently mix incompatible vector spaces.
A safer model is:
embedding_model = BAAI/...
embedding_version = 2026-09
index_version = kb-v17When the embedding model changes, build or migrate to a compatible index rather than silently inserting new vectors into an index produced by another model.
For large knowledge bases, reindexing should usually be performed by a separate worker or ingestion pipeline rather than during application startup.
Content hashing also makes embedding caches straightforward.
If the normalized content and embedding-model version are unchanged, recomputing the vector usually provides no value.
3. Normalize the query before retrieval
A user’s natural-language request is not necessarily a good search query.
It may contain:
- conversational filler;
- unresolved references;
- multiple intents;
- spelling variations;
- aliases;
- implicit metadata constraints.
A query-processing layer can perform bounded transformations:
Useful transformations include:
- whitespace and punctuation normalization;
- alias normalization;
- intent detection;
- metadata-filter extraction;
- reference resolution;
- decomposition of multi-intent requests.
The important word is bounded.
Generating many alternate queries for every request may improve recall, but it also multiplies embedding, search, and reranking work.
Query expansion should therefore be a routing decision rather than a default behavior.
4. Dense retrieval is necessary but insufficient
Dense embeddings are excellent for semantic similarity.
They can connect:
rotate database credentials
with documentation containing:
replace the service secret and reload the workload
even when few exact words overlap.
But dense retrieval can still perform poorly on:
- error codes;
- exact API names;
- product identifiers;
- version numbers;
- short keyword-heavy queries;
- rare terminology;
- negation;
- precise configuration keys.
This is where lexical retrieval remains valuable.
5. Combine sparse and dense retrieval
Sparse retrieval such as BM25 preserves lexical evidence.
Dense retrieval preserves semantic evidence.
The two retrieval signals fail differently, which makes them complementary.
A query containing an exact identifier may be dominated by sparse retrieval.
A paraphrased conceptual question may benefit more from dense retrieval.
A production pipeline should generally avoid assuming that one signal is universally sufficient.
6. Fuse rankings instead of raw scores
Sparse and dense retrieval scores usually live on different scales.
A BM25 score cannot safely be treated as equivalent to cosine similarity.
Adding them directly creates an arbitrary scoring system.
Reciprocal Rank Fusion avoids this by operating on ranking positions.
RRF(d) = sum of 1 / (k + rank(d))The precise constant is tunable, but the important property is that fusion does not require the underlying retrievers to produce calibrated scores.
A candidate consistently ranked highly across retrieval paths can move ahead of a document supported strongly by only one path.
Fusion still has an important limitation:
It cannot recover a document that none of the retrievers returned.
Poor chunking, bad filters, incomplete indexing, or an unsuitable embedding model remain upstream recall problems.
7. Rerank only a reduced candidate set
Initial retrieval should optimize recall.
Reranking should optimize precision.
A cross-encoder evaluates the query and candidate text together, allowing deeper token-level interactions than independent embedding similarity.
That additional quality comes with additional compute.
The production pattern is therefore a cascade:
The exact candidate counts should come from measurement, not convention.
A workload with short policy documents may need a very different retrieval depth from a corpus containing millions of technical chunks.
The invariant is more useful than the numbers:
Expensive relevance models should operate on the smallest candidate set that still preserves retrieval recall.
A production-shaped Python implementation
Frameworks are useful for composition, but the architecture should remain understandable without them.
The following example uses:
- LangChain abstractions;
- Qdrant as the retrieval index;
- dense and sparse hybrid retrieval;
- a cross-encoder reranker.
The important production detail is that index creation is deliberately absent from the request path.
The application connects to a collection that has already been built by the ingestion system.
uv add \
langchain-core \
langchain-community \
langchain-classic \
langchain-huggingface \
langchain-qdrant \
qdrant-client \
fastembed \
sentence-transformersfrom __future__ import annotations
import os
from dataclasses import dataclass
from langchain_classic.retrievers.document_compressors import (
CrossEncoderReranker,
)
from langchain_community.cross_encoders import HuggingFaceCrossEncoder
from langchain_core.documents import Document
from langchain_huggingface import HuggingFaceEmbeddings
from langchain_qdrant import (
FastEmbedSparse,
QdrantVectorStore,
RetrievalMode,
)
from qdrant_client import QdrantClient, models
@dataclass(frozen=True)
class RetrievalConfig:
candidate_k: int = 80
rerank_k: int = 10
class RetrievalPipeline:
def __init__(self, config: RetrievalConfig) -> None:
self.config = config
client = QdrantClient(
url=os.environ["QDRANT_URL"],
api_key=os.environ.get("QDRANT_API_KEY"),
)
dense_embeddings = HuggingFaceEmbeddings(
model_name=os.environ["EMBEDDING_MODEL"],
)
sparse_embeddings = FastEmbedSparse(
model_name="Qdrant/bm25",
)
self.store = QdrantVectorStore(
client=client,
collection_name=os.environ["QDRANT_COLLECTION"],
embedding=dense_embeddings,
sparse_embedding=sparse_embeddings,
retrieval_mode=RetrievalMode.HYBRID,
vector_name="dense",
sparse_vector_name="sparse",
)
cross_encoder = HuggingFaceCrossEncoder(
model_name=os.environ["RERANKER_MODEL"],
)
self.reranker = CrossEncoderReranker(
model=cross_encoder,
top_n=config.rerank_k,
)
async def retrieve(
self,
query: str,
*,
tenant_id: str,
) -> list[Document]:
access_filter = models.Filter(
must=[
models.FieldCondition(
key="metadata.tenant_id",
match=models.MatchValue(value=tenant_id),
),
],
)
candidates = await self.store.asimilarity_search(
query=query,
k=self.config.candidate_k,
filter=access_filter,
)
candidates = deduplicate(candidates)
reranked = await self.reranker.acompress_documents(
candidates,
query,
)
return list(reranked)
def deduplicate(documents: list[Document]) -> list[Document]:
seen: set[str] = set()
result: list[Document] = []
for document in documents:
key = str(
document.metadata.get("chunk_id")
or document.metadata.get("_id")
or hash(document.page_content)
)
if key in seen:
continue
seen.add(key)
result.append(document)
return resultThis is intentionally not a full application.
Authentication, query classification, caching, tracing, retries, generation, and deployment concerns belong around this retrieval component rather than being hidden inside one enormous chain.
Why not put everything inside LangChain?
Framework abstractions are useful when they reduce repetitive integration code.
They are less useful when they hide the behavior that determines retrieval quality.
For critical retrieval stages, I prefer architecture where these decisions remain explicit:
query processing
-> authorization filters
-> candidate generation
-> fusion
-> deduplication
-> reranking
-> context assemblyThe framework can orchestrate those components, but it should not make them difficult to inspect independently.
This matters when debugging production failures.
If retrieval quality drops, an engineer should be able to determine whether the failure came from:
- query transformation;
- access filters;
- dense retrieval;
- sparse retrieval;
- fusion;
- reranking;
- context construction.
A single opaque rag_chain.invoke() call provides very little diagnostic value.
8. Treat authorization as part of retrieval
Metadata filters are often described as a search optimization.
In multi-tenant or permissioned systems, they are much more important:
retrieval filtering is part of the security boundary.
A highly relevant document belonging to another tenant is still an invalid retrieval result.
Access constraints should ideally be enforced before candidate generation and validated again before content enters the model context.
Do not rely on the LLM to respect document permissions after unauthorized information has already reached the prompt.
9. Assemble context as an information budget
Top-ranked documents are not automatically a good prompt.
Context construction should account for:
- duplicates;
- overlapping chunks;
- parent-child relationships;
- token budgets;
- document boundaries;
- source provenance;
- authorization;
- recency or version metadata.
The goal is not:
maximize context.
The goal is:
provide enough high-quality evidence to answer the request.
A simple context assembler can make the budget explicit:
from collections.abc import Callable
from langchain_core.documents import Document
def assemble_context(
documents: list[Document],
*,
count_tokens: Callable[[str], int],
max_tokens: int,
) -> list[Document]:
selected: list[Document] = []
used_tokens = 0
for document in documents:
tokens = count_tokens(document.page_content)
if used_tokens + tokens > max_tokens:
continue
selected.append(document)
used_tokens += tokens
return selectedA real system may additionally merge adjacent chunks, preserve complete sections, or compress passages before context construction.
The important part is that context size is an explicit system decision.
10. Design a degraded mode
Distributed systems should assume dependencies will eventually fail.
Embedding services, rerankers, vector databases, and model providers are no exception.
A retrieval pipeline can often degrade gracefully.
For example:
- embedding unavailable - fall back to lexical retrieval;
- reranker unavailable - use fused candidate order;
- query-expansion model unavailable - use original query;
- generation unavailable - return a controlled service error instead of fabricated content.
Not every system needs every fallback.
But degraded behavior should be an intentional design decision rather than an accident discovered during an outage.
11. Make cost a routing decision
RAG cost is distributed across the pipeline.
Relevant cost drivers include:
- ingestion compute;
- embedding generation;
- index storage;
- vector search;
- sparse search;
- query expansion;
- reranking;
- network calls;
- context size;
- generation tokens.
A useful production system can assign different retrieval budgets to different workloads.
For example:
simple exact query
-> sparse + dense
-> fusion
-> small context
ambiguous query
-> query expansion
-> hybrid retrieval
-> reranking
-> larger context budget
high-value workflow
-> stronger reranker
-> stricter evaluation / verification pathExpensive behavior becomes explicit rather than being applied indiscriminately to every request.
12. Cache deterministic work
Several RAG operations can be reused safely when their inputs and model versions are part of the cache key.
For document embeddings:
cache_key =
hash(
normalized_content
+ embedding_model
+ embedding_version
)The same principle can apply to:
- query embeddings;
- deterministic normalization;
- query rewrites;
- reranking results;
- parsed documents.
Caching without version information is dangerous.
Reusing a vector generated by a different embedding model can silently corrupt retrieval behavior without producing an obvious runtime error.
13. Evaluate retrieval independently from generation
An end-to-end answer score cannot tell you where the system failed.
A useful evaluation case should contain at least:
from dataclasses import dataclass
@dataclass(frozen=True)
class EvalCase:
query: str
relevant_chunk_ids: set[str]
tenant_id: strThen retrieval can be evaluated independently.
def recall_at_k(
retrieved: list[str],
relevant: set[str],
k: int,
) -> float:
if not relevant:
return 1.0
hits = set(retrieved[:k]) & relevant
return len(hits) / len(relevant)
def precision_at_k(
retrieved: list[str],
relevant: set[str],
k: int,
) -> float:
if k <= 0:
return 0.0
returned = retrieved[:k]
if not returned:
return 0.0
hits = sum(chunk_id in relevant for chunk_id in returned)
return hits / len(returned)
def reciprocal_rank(
retrieved: list[str],
relevant: set[str],
) -> float:
for rank, chunk_id in enumerate(retrieved, start=1):
if chunk_id in relevant:
return 1.0 / rank
return 0.0Useful retrieval metrics include:
- Recall@k - did candidate generation contain the evidence?
- Precision@k - how much retrieved material was useful?
- MRR - how early did the first relevant result appear?
- nDCG - how well did ranking preserve graded relevance?
These metrics answer different questions.
A reranker cannot fix poor candidate recall.
Likewise, excellent candidate recall does not guarantee a useful top-five context.
14. Evaluate the answer separately
After retrieval is validated, generation can be measured independently.
Useful response-level dimensions include:
- faithfulness;
- answer relevance;
- citation correctness;
- completeness;
- refusal behavior when evidence is absent.
One of the most important RAG behaviors is the ability to say:
the available evidence does not support an answer.
A system that produces fewer unsupported answers may be more useful than one that answers every question fluently.
15. Instrument every stage
One of the highest-value production features is a retrieval trace.
A useful trace might contain:
trace_id
query.original
query.normalized
query.intent
filters.tenant
filters.document_type
filters.version
retrieval.dense.count
retrieval.sparse.count
fusion.candidate_count
dedupe.candidate_count
reranker.input_count
reranker.output_count
context.chunk_ids
context.token_count
latency.preprocessing_ms
latency.retrieval_ms
latency.reranking_ms
latency.generation_ms
cost.embedding
cost.reranking
cost.generationConceptually:
Without this visibility, retrieval failures often appear to engineers as unpredictable LLM behavior.
With a trace, a bad answer can be investigated stage by stage.
16. Measure latency by stage
Only measuring total request latency hides where the system spends time.
Track at least:
P50 / P95 query preprocessing
P50 / P95 sparse retrieval
P50 / P95 dense retrieval
P50 / P95 reranking
P50 / P95 context construction
P50 / P95 generationThe same applies to cost.
A pipeline may reduce generation cost while accidentally introducing an expensive reranking path on every request.
Stage-level metrics expose those trade-offs.
17. Tune ANN indexes against recall
Approximate nearest-neighbor indexes trade some retrieval accuracy for speed and memory efficiency.
That trade-off should be measured against a labeled evaluation set.
Do not optimize an ANN index for latency in isolation.
An index that is extremely fast but consistently omits the required evidence cannot be repaired by a better reranker or a larger language model.
For smaller corpora, exact search can also be a useful baseline.
Complex indexing strategies should be introduced because measured workload characteristics justify them, not simply because they are available.
18. Keep an explicit failure taxonomy
A useful RAG debugging model separates failures by stage.
| Failure | Likely subsystem |
|---|---|
| Correct document never retrieved | chunking, embeddings, filters, candidate retrieval |
| Exact identifier missing | sparse retrieval |
| Relevant document retrieved but ranked too low | fusion or reranking |
| Same evidence repeated several times | deduplication / chunking |
| Correct evidence retrieved but omitted from prompt | context assembly |
| Unauthorized evidence retrieved | filtering / security boundary |
| Good context but unsupported answer | generation |
| High quality but excessive latency | routing / reranking / generation |
| Cost spikes after deployment | retrieval depth, context size, model routing |
| Results changed after reindexing | embedding/index versioning |
This is more useful operationally than labeling every failure as a hallucination.
19. Roll out retrieval changes like infrastructure changes
Changing an embedding model, chunking policy, fusion strategy, or reranker can affect the behavior of the entire product.
Treat those changes as production changes.
A practical rollout sequence is:
Before deployment:
- run the labeled retrieval evaluation set;
- compare against the current baseline;
- inspect regressions by query class;
- measure latency and cost differences;
- run shadow traffic when possible;
- canary the new configuration;
- monitor empty retrievals and fallback rates.
A higher average benchmark score is not sufficient if an important query class regresses.
20. Start simple and add complexity only for observed failures
A production RAG system does not need every technique described in research papers.
A reasonable progression is:
- establish a hand-checked evaluation set;
- implement dense retrieval;
- enforce metadata and authorization filters;
- add source provenance and citations;
- add sparse retrieval for lexical failure cases;
- fuse rankings;
- add reranking if ranking is demonstrably the bottleneck;
- introduce query expansion only for query classes that need it;
- add caching and request budgets;
- add stage-level traces;
- introduce more sophisticated retrieval representations only when evaluation demonstrates a need.
This order matters.
Each additional component creates:
- more latency;
- more cost;
- more failure modes;
- more deployment complexity;
- more debugging surface.
Complexity should pay rent.
What I would not put in the default pipeline
Several techniques are useful but should not automatically become mandatory stages.
HyDE
Hypothetical Document Embeddings can help when a short or abstract query is structurally different from the corpus.
It also adds an LLM call before retrieval and can introduce assumptions that were not present in the user’s request.
Use it behind routing logic rather than applying it universally.
Multi-query expansion
LLM-based reranking
The production boundary
The design principle
Production RAG is an information retrieval system with a generative model at the end.
The system becomes more dependable when:
- ingestion and serving are separate;
- embeddings and indexes are versioned;
- dense and sparse signals are complementary;
- access control is enforced during retrieval;
- candidate generation optimizes recall;
- reranking optimizes precision;
- context is treated as a budget;
- expensive stages are routed deliberately;
- degraded behavior is designed explicitly;
- every stage is observable;
- retrieval is evaluated separately from generation;
- architectural complexity is added only when measurement justifies it.
The best RAG architecture is rarely the one with the largest number of retrieval techniques.
It is the one that can explain:
why a piece of evidence was retrieved, why it was ranked where it was, why it was allowed into the context, how much that decision cost, and whether the final answer can be traced back to that evidence.
What the current evidence says about these choices
The advice above is not a house style; it is what the current literature and benchmarks keep finding.
- Hybrid retrieval followed by a neural reranker is the strongest single change. A 2026 benchmark over mixed text-and-table financial documents measured Recall@5 of 0.816 for hybrid retrieval plus reranking, against 0.695 for unreranked hybrid retrieval and 0.644 for BM25 alone.
- Sparse retrieval is not obsolete. The same benchmark found BM25 outperforming dense retrieval on that corpus, which is the practical argument for keeping both signals instead of replacing one with the other.
- Candidate depth matters more than the reranker choice. Reranking 20 candidates was ineffective, 50 was a large improvement, and 100 kept improving. Below the cut-off the right document is simply not available to rerank.
- Agentic retrieval is a cost decision, not a free upgrade. A 2026 comparison measured agentic RAG as up to 3.6 times more expensive than an enhanced static pipeline, with the largest gains coming from adaptive query rewriting - and it still benefited from an explicit reranking stage.
- Judge automated evaluation for what it can actually support. The TREC 2025 RAG track scores coverage with information nuggets, requires sentence-level citation support, and reports that automated judges track human assessment well at run level but far less reliably per query.
- Evaluation is multi-dimensional. Current reviews converge on context relevance, faithfulness, answer relevance, correctness and citation quality rather than a single score.
None of this changes the shape of the pipeline. It changes the order in which effort should be spent: fix ingestion, measure retrieval, add a reranker, and treat agentic loops as a last resort.
References and further reading
Primary sources for the techniques named above:
- Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020).
- Robertson and Zaragoza, The Probabilistic Relevance Framework: BM25 and Beyond (2009).
- Cormack, Clarke and Buettcher, Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods (SIGIR 2009) - the source of the
k = 60default. - Karpukhin et al., Dense Passage Retrieval for Open-Domain Question Answering (2020).
- Nogueira and Cho, Passage Re-ranking with BERT (2019).
- Gao et al., Precise Zero-Shot Dense Retrieval without Relevance Labels (HyDE, 2022).
- Jagerman et al., Query Expansion by Prompting Large Language Models (2023).
- Malkov and Yashunin, Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs (2018).
- Liu et al., Lost in the Middle: How Language Models Use Long Contexts (2023).
- Thakur et al., BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models (2021).
- Es et al., Ragas: Automated Evaluation of Retrieval Augmented Generation (2023).
Current state of the field (2025-2026):
- NIST, Overview of the TREC 2025 Retrieval Augmented Generation (RAG) Track (2026) - nugget-based coverage, sentence-level citation support and judge reliability.
- Singh et al., Agentic Retrieval-Augmented Generation: A Survey on Agentic RAG (2025).
- Is Agentic RAG worth it? An experimental comparison of RAG approaches (2026) - the cost and reranking findings quoted above.
- From BM25 to Corrective RAG: Benchmarking Retrieval Strategies (2026) - the hybrid, BM25 and candidate-depth numbers quoted above.
- SoK: Agentic Retrieval-Augmented Generation - Taxonomy, Architectures, Evaluation, and Research Directions (2026).
- Knollmeyer et al., Evaluating Retrieval Augmented Generation: A Comprehensive Review of Evaluation Dimensions, Question Types, and Application (SN Computer Science, 2026).
- Wang, Li and Xiao, jina-reranker-v3: Last but Not Late Interaction for Listwise Document Reranking (2025).
- Qdrant, Hybrid search: prefetch and fusion and Hybrid search in Qdrant - weighted RRF and distribution-based score fusion in a current implementation.
- LangChain, Context engineering for agents (2025) - the write, select, compress and isolate framing for the context stage.
About this article
I am Wigtono, a software engineer working on backend, distributed systems and AI/LLM systems. The pipeline described here is the shape I work with on a production legal-AI retrieval system, written up in the GANI.AI case study; the quality trade-offs behind it are in how I work. The rest of my work is at projects, and the fastest way to reach me is contact.