Lab Guide

Format: Jupyter notebook (lab.ipynb)  ·  Time estimate: 45–60 minutes  ·  Prerequisites: Python 3.9+, sentence-transformers, chromadb, numpy installed

Setup

Before starting, install the required packages:

# From the project root
pip install sentence-transformers chromadb numpy scikit-learn

No API keys are required for this lab — all embedding models run locally. If you want to try API-based models in the bonus section, you will need an OpenAI or Cohere API key.

Exercise 1: Generate Embeddings

Goal

Load a pre-trained embedding model, embed a set of sentences, and inspect the resulting vectors to build intuition about what embeddings look like.

Steps

1. Import SentenceTransformer from sentence_transformers and load the all-MiniLM-L6-v2 model.
2. Define a list of 6–8 sentences covering different topics. Include some semantically similar pairs and some unrelated pairs. For example:

   sentences = [
       "The cat sat on the mat.",
       "A kitten was resting on the rug.",
       "Python is a popular programming language.",
       "JavaScript is widely used for web development.",
       "The stock market crashed yesterday.",
       "Financial markets experienced a sharp decline.",
       "I love eating pizza on Friday nights.",
       "The weather forecast predicts rain tomorrow.",
   ]

3. Generate embeddings using model.encode(sentences).
4. Print the shape of the resulting array and inspect the first 10 values of the first embedding.
5. Verify that each embedding vector is unit-normalised (norm close to 1.0).

Expected Output

Embedding shape: (8, 384)
First 10 values of embedding[0]: [-0.0231  0.0548  0.0112 ...]
Norm of embedding[0]: 1.0000
Norm of embedding[1]: 1.0000

Key Observations

• Each sentence is represented as a 384-dimensional vector (for this model).
• The values are floating-point numbers, typically in the range [-0.3, 0.3].
• The norms should all be very close to 1.0, confirming unit normalisation.

Exercise 2: Compute Similarities

Goal

Build a full cosine similarity matrix and identify which sentence pairs the model considers most and least similar.

Steps

1. Write a function to compute the cosine similarity between two vectors:

   import numpy as np
   
   def cosine_similarity(a, b):
       return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

2. Compute the full similarity matrix (N x N) for all sentence pairs. You can use sklearn.metrics.pairwise.cosine_similarity or compute it manually.
3. Display the matrix as a formatted table or heatmap.
4. Find and print the top 3 most similar pairs (excluding self-similarity) and the top 3 least similar pairs.
5. Verify that semantically related sentences (e.g., cat/kitten, stock market/financial markets) have high similarity scores.

Expected Output

Most similar pairs:
  1. "The stock market crashed yesterday" <> "Financial markets experienced a sharp decline" — 0.8234
  2. "The cat sat on the mat" <> "A kitten was resting on the rug" — 0.7891
  3. "Python is a popular programming language" <> "JavaScript is widely used for web development" — 0.6543

Least similar pairs:
  1. "I love eating pizza on Friday nights" <> "Python is a popular programming language" — 0.0412
  2. ...

Exercise 3: Semantic Search with ChromaDB

Goal

Build a simple semantic search engine that finds the most relevant documents for a natural language query, using ChromaDB as the vector store.

Steps

1. Create a small document corpus (10–15 short passages on mixed topics):

   documents = [
       "Python was created by Guido van Rossum and first released in 1991.",
       "Machine learning is a subset of artificial intelligence.",
       "The Eiffel Tower is located in Paris, France.",
       "Neural networks are inspired by biological brain structures.",
       "The Great Wall of China is visible from low Earth orbit.",
       "Deep learning uses multiple layers of neural networks.",
       "Tokyo is the capital city of Japan.",
       "Natural language processing enables computers to understand human language.",
       "The Amazon rainforest produces about 20% of the world's oxygen.",
       "Transformers revolutionized NLP with the attention mechanism.",
       "Mount Everest is the tallest mountain above sea level.",
       "Gradient descent is the primary optimization algorithm in deep learning.",
   ]

2. Create a ChromaDB collection and add the documents:

   import chromadb
   
   client = chromadb.Client()
   collection = client.create_collection(
       name="lesson23_docs",
       metadata={"hnsw:space": "cosine"}
   )
   
   collection.add(
       documents=documents,
       ids=[f"doc{i}" for i in range(len(documents))]
   )

3. Write a search function that accepts a query string and returns the top-k results:

   def search(query, k=3):
       results = collection.query(query_texts=[query], n_results=k)
       for doc, dist in zip(results["documents"][0], results["distances"][0]):
           print(f"  [{1 - dist:.4f}] {doc}")
       return results

4. Test with several queries:
   • "How do neural networks learn?"
   • "Famous landmarks in Asia"
   • "Who invented Python?"
   • "What is attention in AI?"
5. Experiment with queries that are paraphrases of the documents versus queries that use completely different wording. Note how well the semantic search handles both cases.

Expected Output

Query: "How do neural networks learn?"
  [0.7821] Gradient descent is the primary optimization algorithm in deep learning.
  [0.7156] Deep learning uses multiple layers of neural networks.
  [0.6893] Neural networks are inspired by biological brain structures.

Query: "Famous landmarks in Asia"
  [0.6234] The Great Wall of China is visible from low Earth orbit.
  [0.5891] Tokyo is the capital city of Japan.
  [0.4123] Mount Everest is the tallest mountain above sea level.

Exercise 4: Compare Embedding Models

Goal

Embed the same set of sentences with two different models and compare how their similarity rankings differ.

Steps

1. Load two sentence-transformers models:

   from sentence_transformers import SentenceTransformer
   
   model_a = SentenceTransformer("all-MiniLM-L6-v2")      # 384 dims
   model_b = SentenceTransformer("all-mpnet-base-v2")      # 768 dims

2. Embed the same set of sentences from Exercise 1 with both models.
3. Compute the cosine similarity matrix for each model.
4. For each model, rank all sentence pairs by similarity and compare the top-5 rankings.
5. Print a comparison table showing where the two models agree and disagree:

   Pair                                    | MiniLM  | MPNet   | Diff
   "cat/mat" vs "kitten/rug"              | 0.7891  | 0.8234  | +0.034
   "stock market" vs "financial markets"   | 0.8234  | 0.8567  | +0.033
   ...

6. Note the dimensionality difference (384 vs 768) and whether the higher-dimensional model consistently produces higher or lower similarity scores.

Key Observations

• The two models should broadly agree on which pairs are most/least similar, but the exact scores and rankings may differ.
• Higher dimensionality does not always mean better quality — model architecture and training data also matter.
• The all-mpnet-base-v2 model typically scores higher on MTEB benchmarks than all-MiniLM-L6-v2, but is slower to encode.

from openai import OpenAI
client = OpenAI()
response = client.embeddings.create(input=sentences, model="text-embedding-3-small")
api_embeddings = [e.embedding for e in response.data]

Finishing Up

Once you have completed all four exercises:

1. Run all cells from top to bottom to verify everything executes without errors (Kernel → Restart & Run All).
2. Save the notebook.
3. Compare your implementation with lab_answer.ipynb — focus on anything you did differently and whether your approach is equivalent or could be improved.

Self-Check

• ✓ Exercise 1: Can you explain what each dimension in an embedding vector represents (or rather, why it has no simple interpretation)?
• ✓ Exercise 2: Do the highest-similarity pairs match your intuition about semantic similarity?
• ✓ Exercise 3: Does your semantic search return relevant results even when the query uses different words than the stored documents?
• ✓ Exercise 4: Can you articulate at least one difference in how the two models rank sentence pairs?
• ✓ Can you explain why you must use the same embedding model for both indexing and querying?