RAG Architecture Production Patterns: Retrieval, Ranking, and Evaluation

Introduction

Retrieval-Augmented Generation (RAG) grounds LLM responses in external knowledge—documentation, tickets, codebases—reducing hallucinations and enabling fresh data without full model retraining. Production RAG is an information retrieval system glued to a generation API, not merely "embed and ask."

This guide covers ingestion pipelines, chunking, embedding models, vector and hybrid search, reranking, query transformation, caching, observability, and eval frameworks—the patterns teams use when RAG must work at scale with measurable quality.

Table of Contents

• RAG Pipeline Overview
• Document Ingestion and Chunking
• Embeddings and Vector Stores
• Hybrid Search and Filtering
• Reranking and Context Assembly
• Generation and Citation
• Evaluation and Monitoring
• Best Practices
• Common Mistakes
• Interview Questions
• Summary
• References

RAG Pipeline Overview

[Sources] → Extract → Chunk → Embed → [Vector DB]
                                              ↓
User Query → Transform → Retrieve → Rerank → Assemble Context → LLM → Response

Components:

1. Ingestion — PDF, HTML, Markdown, tickets; metadata (tenant, date, ACL)
2. Retrieval — semantic + keyword; filters for security
3. Augmentation — pack context within token budget
4. Generation — answer with citations; refuse if insufficient context

Failure modes: wrong chunks retrieved, stale index, ACL leak, context overflow truncating key facts.

Document Ingestion and Chunking

Chunking dominates retrieval quality:

def chunk_markdown(text: str, max_tokens: int = 512, overlap: int = 64):
    paragraphs = text.split("\n\n")
    chunks = []
    current = []
    size = 0
    for para in paragraphs:
        para_tokens = estimate_tokens(para)
        if size + para_tokens > max_tokens and current:
            chunks.append("\n\n".join(current))
            # overlap: keep tail of previous chunk
            current = current[-1:] if overlap else []
            size = estimate_tokens(current[0]) if current else 0
        current.append(para)
        size += para_tokens
    if current:
        chunks.append("\n\n".join(current))
    return chunks

Strategies:

• Fixed size with overlap — simple baseline
• Semantic chunking — split on embedding similarity cliffs
• Structure-aware — respect headings in docs (## Section boundaries)
• Parent-child — retrieve small chunks, pass larger parent to LLM

Store metadata: source_url, title, section, updated_at, tenant_id, acl_groups.

Embeddings and Vector Stores

Choose embedding model for domain:

Model class	Use case
General text (text-embedding-3-large)	Docs, support
Code (voyage-code, specialized)	Repositories
Multilingual	Global products

const embedding = await openai.embeddings.create({
  model: "text-embedding-3-small",
  input: chunkText,
});

await vectorStore.upsert({
  id: chunkId,
  vector: embedding.data[0].embedding,
  metadata: { docId, tenantId, text: chunkText },
});

Vector DB options: Pinecone, Weaviate, pgvector, Qdrant. pgvector simplifies ops when already on PostgreSQL:

CREATE TABLE chunks (
  id UUID PRIMARY KEY,
  tenant_id TEXT NOT NULL,
  content TEXT NOT NULL,
  embedding vector(1536)
);
CREATE INDEX ON chunks USING hnsw (embedding vector_cosine_ops);

Tune HNSW ef_search for recall vs latency tradeoff.

Hybrid Search and Filtering

Pure vector search misses exact matches (SKUs, error codes). Hybrid combines BM25 + vector:

def hybrid_search(query: str, tenant_id: str, k: int = 20):
    vector_results = vector_db.search(
        embedding=embed(query),
        filter={"tenant_id": tenant_id},
        top_k=k,
    )
    keyword_results = elasticsearch.search(
        index="chunks",
        query={"bool": {
            "must": [{"match": {"content": query}}],
            "filter": [{"term": {"tenant_id": tenant_id}}],
        }},
        size=k,
    )
    return reciprocal_rank_fusion(vector_results, keyword_results)

Always filter by ACL at retrieval—never fetch then filter in app memory for security-critical data.

Reranking and Context Assembly

First-stage retrieval returns 50–100 candidates; cross-encoder reranker improves precision:

pairs = [(query, c.text) for c in candidates]
scores = reranker.predict(pairs)
top = sorted(zip(candidates, scores), key=lambda x: -x[1])[:5]

Context assembly:

<context>
[1] Source: docs/api.md#auth
{chunk text}

[2] Source: docs/api.md#tokens
{chunk text}
</context>

Answer using only the context above. Cite sources as [1], [2].
If insufficient information, say "I don't know."

Respect token limits—prioritize highest rerank scores; deduplicate overlapping chunks.

Generation and Citation

const completion = await client.chat.completions.create({
  model: "gpt-4o",
  messages: [
    { role: "system", content: SYSTEM_WITH_CITATION_RULES },
    { role: "user", content: `${assembledContext}\n\nQuestion: ${query}` },
  ],
  temperature: 0.2,
});

Post-process: verify cited IDs exist in context. Groundedness check—second LLM or NLI model scores if answer supported by chunks.

Cache query → retrieval results for identical questions; cache full responses only when safe (no user-specific ACL variance).

Evaluation and Monitoring

Offline eval:

• Question set with golden answers and source doc IDs
• Metrics: recall@k, MRR, answer correctness, citation accuracy
• LLM-as-judge with human calibration subset

Online:

• User thumbs down → trace retrieval set
• Latency: embed, retrieve, rerank, generate p95
• Index freshness lag

# Example eval config
datasets:
  - name: support_v1
    cases: 500
thresholds:
  recall_at_5: 0.85
  answer_accuracy: 0.80

Run eval on index updates and embedding model changes before rollout.

Best Practices

1. Structure-aware chunking for technical docs.
2. Hybrid search for codes, IDs, and rare terms.
3. Rerank when first-stage k > 10.
4. Tenant and ACL filters in vector query, not post-hoc.
5. Citation-required prompts and groundedness checks.
6. Incremental indexing with versioned embeddings.

Common Mistakes

1. Tiny chunks—lost semantic context; huge chunks—diluted relevance.
2. No metadata filters—cross-tenant data leaks.
3. Skipping reranking—noise in top-k confuses LLM.
4. Stale index—product changes but answers reference old docs.
5. Eval only on happy path—no adversarial or empty retrieval cases.
6. Embedding model mismatch—query and docs embedded with different models/versions.

Interview Questions

1. Walk through a production RAG pipeline end-to-end.
2. How does chunk size affect retrieval quality?
3. Vector search vs keyword search—when to combine?
4. What is reranking and why use it?
5. How to prevent ACL leaks in multi-tenant RAG?
6. How do you evaluate RAG systems?
7. RAG vs fine-tuning—tradeoffs?
8. How to handle document updates incrementally?

Multi-Tenant RAG Isolation

Every ingestion and query path must include tenant context:

def retrieve(query: str, tenant_id: str, user_groups: list[str]):
    return vector_db.search(
        embedding=embed(query),
        filter={
            "tenant_id": tenant_id,
            "acl_groups": {"$in": user_groups},
        },
        top_k=20,
    )

Run penetration tests attempting cross-tenant retrieval via crafted queries and forged metadata. Log and alert on filter bypass attempts.

Separate indexes per tenant for highest isolation—higher ops cost but simpler compliance story for regulated industries.

Cost and Latency Budgeting

Typical production RAG latency breakdown:

Stage	Typical p95
Embed query	50–150ms
Vector search	20–80ms
Rerank	100–300ms
LLM generation	1–5s

Optimize the slowest stage first—often generation token count. Use smaller models for simple FAQ; reserve large models for complex synthesis.

Track cost per query = embedding + rerank + generation tokens. Set per-user daily budgets for free tiers.

Failure Modes and Fallbacks

Failure	Fallback
Empty retrieval	Canned response + human handoff
LLM timeout	Retry once; then shorter model
Index stale	Show "information may be outdated" banner
ACL filter empty	Refuse—never widen search silently

Design user-visible degradation—silent wrong answers damage trust more than honest limits.

RAG Evaluation Metrics Deep Dive

Metric	Measures
Recall@k	Golden doc in top k results
MRR	Rank of first correct doc
Faithfulness	Answer supported by context
Answer relevance	Answer addresses question

Combine automated scores with weekly human review sample—models drift; retrieval quality drops when documentation restructure changes chunk boundaries without reindex.

Schedule full reindex on major doc migrations; incremental updates alone miss deleted pages that still appear in stale chunks.

Production RAG Checklist

Before launch:

• ACL filters tested cross-tenant
• Empty retrieval fallback defined
• p95 latency under product SLA
• Cost per query within budget
• Eval suite passes on staging index
• Incident runbook for index corruption
• User feedback loop to trace bad answers

Treat RAG as a search product with SLAs—not a one-time LLM integration.

Summary

Production RAG combines ingestion, hybrid retrieval, reranking, constrained generation, and continuous evaluation. Chunking and metadata matter as much as model choice. Treat retrieval as a search product—measure recall, secure with filters, rerank for precision, and never ship without eval gates on index and prompt changes.

References

• LlamaIndex — RAG Guide
• Pinecone — RAG Handbook
• Anthropic — Contextual Retrieval
• pgvector Documentation
• RAGAS — Evaluation Framework