Homework

Three tasks exploring query expansion, HyDE, and building a systematic evaluation dataset.

Complete all three lab exercises before starting the homework. These tasks extend the advanced RAG techniques from the lab.

Task 1 — Query Expansion

Use Claude to generate alternative phrasings for each query, retrieve with all phrasings, and merge results.

Implement expand_query(query, n=3) that asks Claude to generate 3 alternative phrasings of the query
For each expanded query, retrieve top-5 documents
Merge all results using reciprocal rank fusion
Test on 5 questions and compare retrieval results with vs without expansion
Document: did expansion find any relevant documents that the original query missed?

A notebook with the query expansion function, 5 test queries with side-by-side comparison, and a paragraph of analysis.

Implement the HyDE technique: generate a hypothetical answer, embed it, and use it for retrieval instead of the raw query.

Implement hyde_retrieve(query, collection, top_k=3):
- Ask Claude to generate a hypothetical paragraph that would answer the query
- Embed the hypothetical paragraph (not the original query)
- Retrieve top_k documents using the hypothetical embedding
Test on 5 questions and compare results with standard dense retrieval
Measure: does HyDE improve the relevance of top-1 result?
Note the latency overhead of the extra LLM call

A notebook demonstrating HyDE retrieval with comparison to standard retrieval and latency measurements.

Create a structured evaluation dataset for your RAG pipeline with ground truth answers.

Write 10 question-answer pairs based on the sample documents. Each pair should include:
- question: the user's question
- ground_truth: the correct answer (written by you from the source docs)
- source_doc: which document(s) contain the answer
For each question, run your best RAG pipeline and record:
- retrieved_docs: which documents were retrieved
- generated_answer: what Claude answered
- faithfulness_score: LLM-as-judge 1–5
- context_precision: was the source doc in the retrieved set? (0 or 1)
Calculate aggregate scores: average faithfulness and context precision
Identify the weakest questions and propose improvements

A JSON or CSV evaluation dataset plus a summary table with per-question and aggregate scores.

Tip: Save your evaluation dataset as JSON — you will reuse it in later lessons when you learn about systematic evaluation frameworks.

Task 1: Query expansion function; 5 test queries; comparison analysis
Task 2: HyDE implementation; comparison with standard retrieval; latency notes
Task 3: 10-item evaluation dataset; per-question scores; aggregate metrics; improvement proposals
All code runs without error after kernel restart + run all