How machines represent meaning as numbers — from Word2Vec to production-grade vector search over millions of documents.
In 2013, Mikolov et al. showed that training a shallow neural network on a simple objective — predict a word from its neighbours — produces vectors where semantic relationships become arithmetic.
Given surrounding context words, predict the centre word. Fast to train, works well on frequent words.
Input: ["the", "cat", "on", "mat"] → Predict: "sat"
Given the centre word, predict each surrounding word. Better for rare words and small datasets.
Input: "sat" → Predict: ["the", "cat", "on", "mat"]
vec("king") - vec("man") + vec("woman") ≈ vec("queen")
This showed the vectors encode relational structure, not just co-occurrence statistics.
Word2Vec gives one vector per word, but most real tasks need to represent entire passages. Several approaches evolved:
Simple mean of word vectors in a sentence. Cheap but loses word order. Surprisingly effective as a baseline.
Fine-tuned BERT with a siamese/triplet network objective to produce fixed-size sentence vectors. Mean-pools the final-layer token embeddings. This is the foundation of the sentence-transformers library.
OpenAI text-embedding-3-large, Cohere embed-v4, Voyage voyage-3, and Google text-embedding-005 provide high-quality embeddings as a service — no GPU required.
Texts with similar meaning land close together in embedding space; dissimilar texts are far apart. The quality of "similar" depends on what the model was trained or fine-tuned to capture.
Once you have two vectors, you need a way to measure how similar they are. The most common choice is cosine similarity.
import numpy as np
def cosine_similarity(a, b):
return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))| Metric | Range | Best for |
|---|---|---|
| Cosine similarity | -1 to 1 | Normalised embeddings (most common) |
| Dot product | -∞ to +∞ | When magnitude encodes importance |
| Euclidean (L2) | 0 to +∞ | Clustering, when scale matters |
If embeddings are unit-normalised (most API models return normalised vectors), cosine similarity and dot product give identical rankings. Cosine is the safe default.
Embedding models produce vectors of a fixed dimensionality. More dimensions can capture finer distinctions, but at a cost.
| Model | Dimensions | MTEB Avg | Speed |
|---|---|---|---|
| all-MiniLM-L6-v2 | 384 | ~56 | Very fast (local) |
| text-embedding-3-small (OpenAI) | 1536 | ~62 | API latency |
| text-embedding-3-large (OpenAI) | 3072 | ~64 | API latency |
| voyage-3 (Voyage AI) | 1024 | ~67 | API latency |
| embed-v4 (Cohere) | 1024 | ~66 | API latency |
Once you have embeddings, you need somewhere to store and search them efficiently. Vector databases use approximate nearest neighbour (ANN) algorithms to find similar vectors in milliseconds.
Open-source, Python-native, embedded mode. Great for prototyping and small-to-medium datasets. Zero infrastructure needed.
Fully managed cloud service. Scales to billions of vectors. Serverless or pod-based tiers. No infrastructure to manage.
Open-source with cloud option. Hybrid search (vector + keyword). Built-in module system for embedding generation.
PostgreSQL extension. Add vector search to your existing Postgres database. Perfect when you already use Postgres and want to avoid a second database.
import chromadb
client = chromadb.Client()
collection = client.create_collection("docs")
collection.add(
documents=["AI is transforming healthcare", "The stock market rose today"],
ids=["doc1", "doc2"]
)
results = collection.query(query_texts=["medical AI"], n_results=1)
print(results["documents"]) # [['AI is transforming healthcare']]The Massive Text Embedding Benchmark (MTEB) evaluates embedding models across 8 task categories and 56+ datasets. It is the standard way to compare embedding quality.
Off-the-shelf models work well for general text. Fine-tuning helps when:
from sentence_transformers import SentenceTransformer, InputExample, losses
from torch.utils.data import DataLoader
model = SentenceTransformer("all-MiniLM-L6-v2")
train_examples = [
InputExample(texts=["anchor text", "positive match"], label=1.0),
InputExample(texts=["anchor text", "negative example"], label=0.0),
]
train_dataloader = DataLoader(train_examples, shuffle=True, batch_size=16)
train_loss = losses.CosineSimilarityLoss(model)
model.fit(train_objectives=[(train_dataloader, train_loss)], epochs=3)sentence-transformers to embed a set of sentences and inspect the resulting vectors