Retrieval-Augmented Generation (RAG) is only as good as the context it retrieves. While basic vector search is fast, it often struggles to prioritize the most relevant documents within its top results. This is where rerankers come in—acting as a critical second stage to refine and reorder retrieved contexts.
Table of Contents
- What Reranking Does
- Stage One: High-Recall Retrieval
- Stage Two: Rescore and Reorder
- Cross-Encoder Rerankers
- Bi-Encoder vs Cross-Encoder
- Production Considerations
- Interview Questions
- Summary
What Reranking Does
In a standard RAG pipeline, retrieval typically relies on a bi-encoder architecture to quickly find documents semantically close to the user's query. However, "closeness" in vector space doesn't always equal relevance for answering a specific question.
Two-stage retrieval decouples fast initial retrieval from deeper relevance scoring.
Stage One: High-Recall Retrieval
The first stage uses dense vector search (HNSW, IVF) or sparse lexical search (BM25). The goal is high recall: grab a broad candidate set (e.g., top 100) prioritizing speed over precision.
candidates = vector_db.search(query_embedding, top_k=100)
Best Practices
- Retrieve more candidates than you need (50–200)
- Combine dense + sparse retrieval (hybrid search)
- Filter by metadata before vector search when possible
Common Mistakes
- Using top-5 vector results directly without reranking
- Ignoring chunk size and overlap during indexing
Stage Two: Rescore and Reorder
A reranker evaluates query-document pairs jointly, outputting calibrated relevance scores. Only the top N (e.g., 5) pass to the LLM.
scores = reranker.predict([(query, doc) for doc in candidates])
ranked = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True)
context = [doc for doc, _ in ranked[:5]]
Cross-Encoder Rerankers
Cross-encoders (BERT, RoBERTa-based) process query and document together through self-attention layers. They are computationally expensive but highly accurate.
Bi-Encoder vs Cross-Encoder
| Approach | Speed | Accuracy | Use Case |
|---|---|---|---|
| Bi-Encoder | Fast | Good | Stage 1 retrieval |
| Cross-Encoder | Slow | Excellent | Stage 2 reranking |
Production Considerations
Performance
- Batch rerank requests
- Cache reranker results for repeated queries
- Use smaller reranker models for latency-sensitive paths
Security
- Sanitize document content before reranking
- Rate-limit reranker API calls
Failure Modes
- Truncation: Long documents lose signal—truncate intelligently
- Drift: Embedding model updates break retrieval—version indexes
- Tail latency: Reranking adds p99 spikes—set timeouts and fallbacks
Interview Questions
- Why can't you use a cross-encoder for initial retrieval over millions of documents?
- How do you choose candidate set size for reranking?
- When is reranking not worth the latency cost?
Summary
Reranking is the highest-ROI upgrade for production RAG systems struggling with irrelevant context. Start with hybrid retrieval + a cross-encoder reranker on a 50–100 candidate set.
References
- Cohere Rerank Documentation
- Sentence Transformers Cross-Encoders
- Lewis et al., "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks"
