Homework

Task 1: Domain-Specific FAQ Retrieval

Goal: Build a working FAQ retrieval system for a specific domain and measure how well it performs.

Instructions

1. Choose a domain you are familiar with (e.g., your company's product, a technology you know well, a hobby, a course subject). Create or collect at least 20 FAQ entries, each with a question and a short answer (2–4 sentences).
2. Embed the FAQ questions using all-MiniLM-L6-v2 (or another sentence-transformers model of your choice) and store them in ChromaDB.
3. Write 10 test queries — natural language questions that a user might ask. For each query, you should know which FAQ entry is the correct match. Include:
   • 5 queries that use similar wording to the stored FAQ
   • 5 queries that paraphrase the same intent with completely different words
4. For each test query, retrieve the top-3 results and check whether the correct FAQ appears in the results.
5. Calculate Recall@3: the percentage of test queries where the correct answer appears in the top 3 results.

Deliverable

A notebook or script that:

• Defines the FAQ corpus and test queries
• Runs retrieval for all 10 queries
• Prints each query, the top-3 results, and whether the correct answer was found
• Prints the final Recall@3 score

Self-check

• ✓ Your FAQ corpus has at least 20 entries.
• ✓ Your test set includes both similar-wording and paraphrased queries.
• ✓ You report a concrete Recall@3 number.
• ✓ You can explain why paraphrased queries may have lower recall than similar-wording queries.

Task 2: Cosine vs. Euclidean Distance

Goal: Understand when cosine similarity and Euclidean distance give different results, and why it matters for retrieval.

Instructions

1. Using the embeddings from your lab (Exercise 1 sentences or your FAQ corpus), compute both the cosine similarity matrix and the Euclidean distance matrix for all pairs.
2. Rank all pairs by cosine similarity (descending) and by Euclidean distance (ascending). Compare the top-10 rankings from each metric.
3. Now create a scenario where the metrics diverge:
   • Take an embedding vector and scale it by 3x (multiply by 3.0). This simulates a non-normalised embedding.
   • Recompute both metrics for pairs involving the scaled vector.
   • Observe how cosine similarity is unaffected but Euclidean distance changes dramatically.
4. Write a short explanation (100–150 words) of when you would prefer each metric in a production system.

Self-check

• ✓ You computed both metrics for the same set of pairs.
• ✓ You demonstrated that scaling a vector changes Euclidean distance but not cosine similarity.
• ✓ Your explanation mentions normalisation as the key factor.
• ✓ You can state a concrete scenario where Euclidean distance would be the better choice.

Task 3: Matryoshka Embeddings Explainer

Goal: Research and explain a cutting-edge embedding technique in your own words.

Instructions

Read about Matryoshka Representation Learning (MRL) from any of these sources:

• The original paper: Kusupati et al., 2022
• OpenAI's embedding documentation (which implements MRL in text-embedding-3 models)
• Blog posts or summaries you find via search

Write exactly 150 words (±15) explaining:

1. What Matryoshka embeddings are and how they differ from standard fixed-dimension embeddings
2. How they are trained (the key idea, not the full math)
3. One concrete production scenario where they provide a significant advantage

Self-check

• ✓ Your explanation is 135–165 words.
• ✓ You explain the nesting property (smaller dimensions are a prefix of larger ones).
• ✓ You describe the training approach (joint loss across multiple dimensionalities).
• ✓ Your production scenario is specific and practical (not just "it saves memory").