Lab Guide — Advanced RAG

Add BM25 and implement reciprocal rank fusion

Extend your Lesson 08 pipeline with a BM25 retriever and merge results using RRF.

1. Create a BM25 index from the same 10 documents using rank_bm25.BM25Okapi
2. Implement bm25_search(query, k=5) that returns document indices sorted by BM25 score
3. Implement dense_search(query, collection, k=5) that returns document indices from ChromaDB
4. Implement reciprocal_rank_fusion(ranked_lists, k=60) that merges multiple ranked lists
5. Implement hybrid_search(query, k=5) that combines BM25 and dense via RRF

Test your implementation

# This query should benefit from BM25 (exact keyword match)
results = hybrid_search("lru_cache decorator", k=3)
print(results)  # functools doc should rank high

# This query should benefit from dense search (semantic match)
results = hybrid_search("how to measure code performance", k=3)
print(results)

Checkpoint: Hybrid search returns relevant results for both keyword-heavy and semantic queries.

Add cross-encoder reranking to the hybrid pipeline

1. Load a cross-encoder model: CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
2. Implement rerank(query, doc_indices, top_k=3) that scores each (query, doc) pair and returns the top_k by score
3. Build advanced_search(query, retrieve_k=10, final_k=3) that does hybrid retrieval then reranking

Test your implementation

# Compare naive vs hybrid vs hybrid+rerank
query = "How do I create a custom dictionary with default values?"
print("Naive:", naive_search(query, k=3))
print("Hybrid:", hybrid_search(query, k=3))
print("Advanced:", advanced_search(query, retrieve_k=10, final_k=3))

Checkpoint: Reranking consistently promotes the most relevant document to position 1.

Evaluate naive vs advanced on 5 questions using faithfulness scoring

1. Define 5 test questions with expected ground truth answers
2. For each question, generate answers using both the naive pipeline (Lesson 08) and the advanced pipeline
3. Implement judge_faithfulness(question, context, answer) that asks Claude to rate faithfulness 1–5
4. Build a comparison table: Question | Naive Score | Advanced Score

Judge prompt template

judge_prompt = f"""Rate the faithfulness of this answer on a scale of 1-5.
1 = completely unfaithful (hallucinated or contradicts context)
5 = completely faithful (every claim is supported by context)

Context: {context}
Question: {question}
Answer: {answer}

Respond with only a single integer (1-5) and a one-sentence justification."""

Checkpoint: You have a comparison table showing faithfulness scores for both pipelines across 5 questions.