Theory: Advanced RAG

Moving beyond naive retrieval with hybrid search, reranking, query expansion, and systematic evaluation.

Introduction

In Lesson 08, we built a basic RAG pipeline: embed documents, store in a vector database, retrieve by cosine similarity, and generate an answer. This pipeline works surprisingly well for simple cases, but it hits a ceiling quickly. Queries with specific technical terms may miss relevant documents. Retrieved context often includes marginally relevant chunks that dilute the signal. And without systematic evaluation, you cannot tell whether your pipeline is actually improving.

This lesson covers four techniques that address these limitations: hybrid search for better recall, reranking for better precision, query expansion for handling diverse query formulations, and RAGAS-style evaluation for measuring quality. Together, these techniques can take a mediocre RAG pipeline and make it production-ready.

Naive RAG Limitations

The fundamental issue with naive RAG is that a single embedding-based similarity metric cannot handle all query types equally well. Dense embeddings excel at capturing semantic meaning but struggle with exact keyword matching. Consider these failure modes:

Vocabulary mismatch: A user asking about "caching function results" may not retrieve a document that discusses lru_cache and memoization but never uses the word "caching."
Rare terms: Proper nouns, acronyms, and domain-specific jargon are often poorly represented in embedding space. A query for "BM25" may not match a document about "Okapi BM25 scoring function."
Precision dilution: When you retrieve top-5 chunks and only 2 are truly relevant, the 3 irrelevant chunks add noise to the context, potentially confusing the LLM.
Question-passage asymmetry: Questions and answer passages have fundamentally different structures. A question embedding may be far from the passage embedding even when the passage perfectly answers the question.

BM25

BM25 (Best Matching 25) is a classical information retrieval scoring function based on term frequency and inverse document frequency. It has been the backbone of search engines for decades and remains competitive with neural methods for keyword-heavy queries.

The key advantages of BM25 over dense retrieval:

Exact match: BM25 always finds documents containing the exact query terms.
No model needed: BM25 is a statistical algorithm — no GPU, no embedding model, no latency.
Interpretable: You can trace exactly why a document scored high (which terms matched).

The rank_bm25 Python library provides a simple implementation:

from rank_bm25 import BM25Okapi

tokenized_corpus = [doc.lower().split() for doc in documents]
bm25 = BM25Okapi(tokenized_corpus)
scores = bm25.get_scores(query.lower().split())
top_indices = scores.argsort()[::-1][:k]

Reciprocal Rank Fusion

Given ranked lists from multiple retrieval methods (e.g., BM25 and dense), Reciprocal Rank Fusion (RRF) produces a single merged ranking. The formula is:

RRF_score(d) = sum over all rankers of 1 / (k + rank_i(d))

where k is a constant (typically 60). Documents that appear near the top of multiple lists receive the highest fused scores. RRF is simple, effective, and requires no training or tuning. It consistently outperforms individual retrieval methods in benchmarks.

The constant k dampens the impact of being ranked first vs second (1/61 vs 1/62 is a small difference), which prevents any single retrieval method from dominating. A document ranked 1st by BM25 and 10th by dense search would score 1/61 + 1/70 = 0.0306, while a document ranked 3rd by both would score 1/63 + 1/63 = 0.0317, correctly ranking the consistently relevant document higher.

Cross-Encoder Reranking

The initial retrieval stage (whether BM25, dense, or hybrid) must be fast because it searches the entire corpus. This constrains it to bi-encoder architectures where queries and documents are encoded independently. Reranking relaxes this constraint: given a small candidate set (typically 10-20 documents), a cross-encoder can examine each (query, document) pair jointly.

A cross-encoder passes the query and document through the same transformer together, allowing full cross-attention between query tokens and document tokens. This produces much more accurate relevance scores but is too slow for full-corpus search. The typical pattern:

Retrieve top-20 candidates with fast bi-encoder or hybrid search
Score all 20 with a cross-encoder
Take the top-5 reranked results as final context

Popular cross-encoder models include cross-encoder/ms-marco-MiniLM-L-6-v2 (fast, good quality) and cross-encoder/ms-marco-MiniLM-L-12-v2 (slower, better quality). Cohere and Jina also offer API-based reranking services.

Query Expansion & HyDE

Query expansion addresses the vocabulary mismatch problem by generating multiple versions of the query. An LLM rewrites the original query into 2-3 alternative phrasings, each emphasizing different aspects or using different terminology. You then retrieve with all queries and merge the results (again, RRF works well here).

HyDE (Hypothetical Document Embeddings) takes a different approach: instead of reformulating the query, ask the LLM to generate a hypothetical passage that would answer the query. Then embed this hypothetical passage and use it for retrieval. The insight is that a generated passage is structurally more similar to real documents than a question is, so it maps to a more useful region of embedding space.

Both techniques add one LLM call before retrieval, increasing latency by 0.5-2 seconds. They are most valuable when:

Users ask questions in casual language but documents use formal/technical language
The corpus uses domain-specific jargon that users do not know
Recall is more important than latency (e.g., research, legal, medical)

RAGAS Evaluation

RAGAS (Retrieval Augmented Generation Assessment) provides a framework for evaluating RAG pipelines using LLM-as-judge. The four core metrics are:

Faithfulness: Does the generated answer contain only claims supported by the retrieved context? The LLM extracts individual claims from the answer and checks each against the context. Score = (supported claims) / (total claims).
Answer relevancy: Does the answer actually address the question? The LLM generates questions from the answer and measures their semantic similarity to the original question.
Context precision: Are the retrieved documents relevant to the question? The LLM labels each retrieved chunk as relevant or not. Score = (relevant chunks) / (total chunks).
Context recall: Did we retrieve all the information needed to answer? Compare retrieved context against a ground truth answer to check coverage.

A simpler starting point is to use an LLM as a faithfulness judge: for each generated answer, ask a second LLM to rate on a 1-5 scale whether the answer is fully supported by the provided context. This single metric catches the most critical failure mode of RAG systems: hallucinating information not present in the retrieved documents.