Lesson 08 RAG Basics
Theme:

Theory: RAG Basics

Retrieval-Augmented Generation — grounding LLM answers in your own data without fine-tuning.

Introduction

Retrieval-Augmented Generation (RAG) is a technique that enhances LLM responses by retrieving relevant information from an external knowledge base at query time and injecting it into the prompt. First described by Lewis et al. (2020) at Facebook AI Research, RAG has become the dominant pattern for building knowledge-grounded AI applications. The appeal is straightforward: instead of encoding all knowledge into model weights (expensive, static, opaque), you store knowledge in a searchable database and retrieve it on demand (cheap, updatable, auditable).

This lesson covers the foundational components: embedding models, vector stores, the RAG pipeline, and chunking strategies. By the end, you will be able to build a working RAG system over any text corpus.

Why RAG?

Large language models have three well-documented limitations that RAG addresses:

Compared to fine-tuning, RAG is faster to set up (no training required), easier to update (just add or remove documents), and more transparent (you can inspect exactly what context the model received). Fine-tuning changes model behaviour; RAG changes model knowledge.

Embedding Models

An embedding model maps a text string to a fixed-size numeric vector in a high-dimensional space. The key property is that semantically similar texts are mapped to nearby vectors. This enables semantic search: given a query, you find the documents whose vectors are closest to the query vector.

Modern embedding models are typically transformer-based neural networks trained with contrastive learning objectives. The model learns to place texts with similar meanings close together and dissimilar texts far apart. Popular open-source options include:

API-based options include OpenAI's text-embedding-3-small (1536d) and Voyage AI's models. The trade-off is convenience and quality versus cost and data privacy.

Vector Similarity

Given two embedding vectors, similarity is measured using a distance or similarity metric. The most common choice is cosine similarity, which measures the angle between two vectors regardless of their magnitude:

cosine_similarity(A, B) = (A · B) / (||A|| * ||B||)

Values range from -1 (opposite) to 1 (identical). In practice, well-trained embedding models produce values between 0 and 1 for English text. A cosine similarity of 0.8+ typically indicates strong semantic relevance.

Other metrics include Euclidean distance (L2) and dot product. ChromaDB defaults to L2 distance but supports cosine similarity via configuration. The choice of metric should match the metric used during embedding model training — most sentence-transformer models are trained with cosine similarity.

ChromaDB

ChromaDB is an open-source vector database designed for simplicity. It runs in-memory by default (no server needed) and supports persistent storage to disk. Key features:

import chromadb
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")
client = chromadb.Client()
collection = client.create_collection("docs")

# Index documents
docs = ["Python pathlib handles file paths", "JavaScript uses async/await"]
embeddings = model.encode(docs).tolist()
collection.add(
    ids=["d1", "d2"],
    documents=docs,
    embeddings=embeddings,
    metadatas=[{"topic": "python"}, {"topic": "javascript"}]
)

# Query with metadata filter
query_emb = model.encode(["file handling"]).tolist()
results = collection.query(
    query_embeddings=query_emb, n_results=2,
    where={"topic": "python"}
)

The RAG Pipeline

A complete RAG system has two distinct phases:

Indexing (offline, run once per corpus update)

  1. Load documents from files, databases, or APIs.
  2. Chunk each document into retrieval-sized units.
  3. Embed each chunk using the embedding model.
  4. Store embeddings and metadata in the vector store.

Querying (online, per user question)

  1. Embed the user's question with the same model.
  2. Retrieve the top-k most similar chunks from the vector store.
  3. Build prompt: inject retrieved chunks as context, add the user question.
  4. Generate: send the augmented prompt to Claude and return the answer.

The prompt template typically looks like:

Use the following context to answer the question. If the context does not
contain enough information, say so.

Context:
{retrieved_chunks}

Question: {user_question}

Chunking for Retrieval

Chunking strategy has an outsized impact on RAG quality. The goal is to create chunks that are large enough to carry meaningful context but small enough to be precisely relevant to a query.

Size: 256-512 tokens is the most common choice. Smaller chunks (128 tokens) improve precision but may miss context. Larger chunks (1024+) include more context but dilute relevance when only a sentence within the chunk is actually relevant.

Overlap: Adding 10-20% overlap between consecutive chunks ensures that information at chunk boundaries is not lost. A sentence that falls exactly on a boundary would be split across two chunks without overlap, potentially making it irretrievable by either chunk.

Strategy: The simplest approach is fixed-size word or token splitting with overlap. More sophisticated approaches include sentence-boundary splitting (never break mid-sentence), recursive character splitting (try paragraph breaks first, then sentence breaks, then word breaks), and semantic chunking (split where embedding similarity between consecutive sentences drops).

Metadata Filtering

Metadata filtering is a powerful feature that narrows the search space before vector similarity is computed. Common metadata fields include:

Filtering happens before the vector search, so it both improves relevance and speeds up queries by reducing the candidate set. In ChromaDB, use the where parameter:

results = collection.query(
    query_embeddings=query_emb,
    n_results=5,
    where={"topic": "python"}  # only search Python docs
)

Further Reading