Theory

Reading time: ~20 minutes (~1500 words) · Prerequisites: Lessons 20–22 (tokenization, transformers, pre-training)

1. Why Embeddings Matter

Computers work with numbers, not words. An embedding is a learned mapping from discrete objects — words, sentences, images, anything — into a continuous vector space where geometric proximity encodes semantic similarity. If two pieces of text mean roughly the same thing, their embedding vectors point in roughly the same direction.

Embeddings are the bridge between human language and machine computation. They power semantic search, recommendation systems, clustering, classification, anomaly detection, and the retrieval step of RAG (Retrieval-Augmented Generation) pipelines. Understanding how they work, what trade-offs they involve, and how to evaluate them is essential for building production AI systems.

2. Word2Vec: The Foundation

The modern embedding era began with Word2Vec (Mikolov et al., 2013). The core insight was simple: words that appear in similar contexts should have similar vector representations. Word2Vec offered two training objectives:

CBOW (Continuous Bag of Words): Given a window of surrounding words, predict the centre word. The model learns to compress context into a representation that is maximally predictive of the target word. CBOW is faster to train and tends to produce better representations for frequent words.
Skip-gram: Given a single word, predict each of its surrounding context words. Skip-gram inverts the CBOW objective and tends to perform better on rare words and smaller datasets because each training example provides multiple gradient updates.

Both architectures are shallow neural networks — just an input layer, one hidden layer, and an output layer. The hidden layer weights become the word vectors. Despite this simplicity, Word2Vec vectors famously captured relational structure: vec("king") - vec("man") + vec("woman") lands near vec("queen").

Limitations of Static Embeddings

Word2Vec assigns a single vector to each word regardless of context. The word "bank" gets the same vector whether it means a financial institution or a river bank. This is a fundamental limitation that later contextual models (ELMo, BERT, and the transformer-based embedding models we use today) resolved by producing different vectors for the same word depending on its surrounding context.

3. Sentence and Document Embeddings

Most practical tasks require embedding entire sentences, paragraphs, or documents — not individual words. Several approaches emerged:

Averaging word vectors: Take the mean of all word vectors in a passage. Loses word order and weighting, but serves as a surprisingly competitive baseline.
Doc2Vec (2014): Extended Word2Vec with a document-level vector trained alongside word vectors. Fell out of favour with the rise of transformers.
Sentence-BERT (Reimers & Gurevych, 2019): Fine-tuned BERT using a siamese network architecture with contrastive or triplet loss. The result is a model that produces fixed-size sentence embeddings by mean-pooling the final transformer layer. This approach is implemented in the widely-used sentence-transformers Python library.
API embedding models: Services like OpenAI (text-embedding-3-large), Cohere (embed-v4), Voyage AI (voyage-3), and Google (text-embedding-005) provide high-quality embeddings via API call. No local GPU is needed — you send text, you receive vectors.

Choosing Between Local and API Models

Use local models (sentence-transformers) when you need low latency, offline operation, full data privacy, or want to fine-tune. Use API models when you want state-of-the-art quality without managing infrastructure, or when your volume is low enough that per-call pricing is cheaper than GPU hosting.

4. Cosine Similarity and Distance Metrics

With vectors in hand, you need a way to measure how close two vectors are. The three main options are:

Cosine similarity: Measures the angle between two vectors, ignoring magnitude. Defined as the dot product divided by the product of the norms. Returns a value between -1 and 1, where 1 means identical direction (maximum similarity) and 0 means orthogonal (no relationship). This is the default choice for text embeddings.
Dot product: Like cosine similarity but without normalisation. If vectors are already unit-normalised (most API embedding models return normalised vectors), dot product and cosine similarity produce identical rankings. If vectors are not normalised, the dot product also considers magnitude, which can be useful when vector length encodes something meaningful like confidence or importance.
Euclidean distance (L2): Measures straight-line distance in the vector space. Useful for clustering algorithms that assume geometric distance. For normalised vectors, Euclidean distance is a monotonic transformation of cosine similarity, so rankings are the same.

Practical guidance: Use cosine similarity unless you have a specific reason not to. It is robust, intuitive, and universally supported by vector databases.

5. Embedding Dimensions and Quality

Embedding models produce vectors of fixed dimensionality — typically between 384 and 3072 dimensions. Higher dimensionality allows the model to represent finer distinctions, but it comes with costs: more storage per vector, slower similarity computation, and higher memory usage for vector indices.

A recent innovation is Matryoshka Representation Learning (MRL), used in models like OpenAI's text-embedding-3 series. Matryoshka embeddings are trained so that the first d dimensions of a larger vector form a useful embedding on their own. You can truncate a 3072-dimensional vector to 256 dimensions and still get reasonable quality — useful for building tiered retrieval systems where a fast, low-dimensional first pass narrows candidates before a full-dimensional reranking step.

6. Vector Databases

When you have thousands or millions of embeddings, you need specialised storage that supports fast approximate nearest neighbour (ANN) search. Exact brute-force comparison (check every vector) is O(n) per query and becomes impractical past a few hundred thousand vectors.

Vector databases solve this with index structures like HNSW (Hierarchical Navigable Small World), IVF (Inverted File Index), and PQ (Product Quantisation). These trade a small amount of recall for dramatically faster search — typically sub-millisecond for millions of vectors.

Database	Type	Best for
ChromaDB	Open-source, embedded	Prototyping, small–medium datasets, Python-native workflows
Pinecone	Managed cloud	Production at scale, zero ops overhead
Weaviate	Open-source + cloud	Hybrid vector + keyword search, multi-modal
pgvector	PostgreSQL extension	Adding vector search to existing Postgres infrastructure
Qdrant	Open-source + cloud	High performance, filtering, Rust-based
FAISS	Library (Meta)	Research, custom index tuning, GPU-accelerated search

7. Retrieval with Embeddings

The standard retrieval pipeline with embeddings works as follows:

Indexing (offline): Split your document corpus into chunks. Embed each chunk using your chosen model. Store the vectors (plus metadata) in a vector database.
Querying (online): Embed the user's query with the same model. Search the vector database for the top-k nearest vectors. Return the associated text chunks.
Post-processing: Optionally rerank results with a cross-encoder or LLM for higher precision. Feed the top results into an LLM as context (RAG).

Critical detail: the query and the documents must be embedded with the same model. Vectors from different models live in different vector spaces and cannot be compared meaningfully.

Common Pitfall: Chunking Strategy

How you split documents into chunks has a large impact on retrieval quality. Chunks that are too small lose context; chunks that are too large dilute the embedding signal. A typical starting point is 256–512 tokens per chunk with 10–20% overlap. Experiment with your specific data to find the optimal size.

8. MTEB: Evaluating Embedding Quality

The Massive Text Embedding Benchmark (MTEB) is the standard evaluation suite for embedding models. It tests models across 8 task categories — retrieval, semantic textual similarity (STS), classification, clustering, reranking, pair classification, summarisation, and bitext mining — using 56+ datasets.

MTEB scores reveal that no single model dominates all tasks. A model that excels at retrieval may be mediocre at clustering. Always check MTEB scores for the specific task type you care about, not just the overall average. The leaderboard is available at huggingface.co/spaces/mteb/leaderboard.

9. Fine-Tuning Embeddings

When off-the-shelf models do not capture domain-specific notions of similarity, fine-tuning can help. The sentence-transformers library supports several training approaches:

Contrastive loss: Train with pairs of (similar, dissimilar) texts. The model learns to pull similar pairs closer and push dissimilar pairs apart.
Triplet loss: Train with (anchor, positive, negative) triplets. More explicit signal about relative ordering.
Multiple Negatives Ranking Loss (MNRL): Efficient for retrieval tasks — treats every other example in the batch as a negative. Requires only positive pairs.

Fine-tuning typically requires hundreds to thousands of labelled examples. Even a small amount of domain-specific training data can yield significant improvements over general-purpose models for specialised vocabularies (legal, medical, scientific) or custom similarity definitions.