Context Windows & Long Documents

Understanding context limits, handling documents that exceed them, and choosing the right strategy for your use case.

Introduction

Every large language model processes input through a "context window" — the maximum amount of text (measured in tokens) it can read and reason over in a single call. Modern Claude models support 200,000 token windows, which at roughly 750 words per 1,000 tokens equals approximately 150,000 words, or about 600 pages. This is substantial, but real-world applications frequently involve documents, codebases, or conversation histories that exceed even these limits — and even when documents fit, sending 150,000 tokens on every query is expensive and slow.

This lesson provides the tools to handle long documents practically: understanding the attention behavior of models over long contexts, breaking documents into manageable chunks, and choosing the right summarization or retrieval strategy for your use case.

Context Windows Explained

A "token" is the basic unit of text that models process. Most modern tokenizers (including Claude's) split text into subword units — roughly 1 token per 3/4 of a word in English, meaning 1,000 tokens is approximately 750 words. The context window defines the maximum number of tokens in the combined input (system prompt + conversation history + user message) that the model can process.

Text that exceeds the context window simply cannot be processed in a single API call. But even well within the limit, the behavior of models on very long contexts has important quirks.

The Lost-in-the-Middle Problem

Research by Liu et al. (2023) examined how well models could identify relevant information planted at different positions in a long context. The findings were striking: models were significantly better at recalling information placed near the beginning or end of the context, and significantly worse at recalling information placed in the middle.

This U-shaped recall pattern is known as the "lost-in-the-middle" problem. It arises from how Transformer attention weights are distributed: the model pays relatively more attention to tokens near the beginning (which it has seen longest during pre-training sequences) and the end (due to recency effects), but attention to middle tokens dilutes as context grows.

Practical Implications

• Place the most critical instructions at the very beginning or end of your prompt.
• When constructing RAG prompts with retrieved chunks, put the highest-relevance chunks near the start or end — not buried in the middle.
• For very long contexts, consider repeating key instructions at the end as a reminder.
• Do not assume that filling the full 200K context window will produce perfect recall of all inserted content.

Chunking Strategies

When a document exceeds the context window, or when you need to process multiple documents efficiently, you must split the text into chunks. The choice of chunking strategy significantly affects downstream quality.

Fixed-Size with Overlap

The simplest approach: split text into chunks of N words (or tokens), with each chunk overlapping the previous one by O words. Overlap ensures that sentences crossing chunk boundaries are fully represented in at least one chunk.

def chunk_text(text: str, chunk_size: int = 500, overlap: int = 50) -> list[str]:
    """Split text into overlapping word-count chunks."""
    words = text.split()
    chunks = []
    i = 0
    while i < len(words):
        chunk = " ".join(words[i : i + chunk_size])
        chunks.append(chunk)
        i += chunk_size - overlap
    return chunks

Typical parameters: 300–800 words per chunk, 10–15% overlap. Smaller chunks increase granularity and retrieval precision but increase the number of API calls. Larger chunks provide more context per call but risk the lost-in-the-middle effect within the chunk itself.

Sentence-Based Chunking

Split on sentence boundaries (using punctuation or an NLP sentence tokenizer), then aggregate sentences into chunks of approximately the target size. This avoids mid-sentence cuts and produces more natural chunk boundaries. The nltk.sent_tokenize function or a simple regex suffices for most English text.

Semantic Chunking

Group paragraphs or sections by topic using embeddings (compute similarity between adjacent passages; split where similarity drops) or by structural signals (headings, section breaks). Semantic chunking produces the most coherent chunks but requires additional processing. It is most valuable when documents have clear topical structure (research papers, legal documents, product manuals).

Summarization Patterns (Map-Reduce vs Refine)

Once a document is chunked, you typically need to synthesize across chunks to answer holistic questions or produce a document-level summary. Two main patterns address this.

Map-Reduce

The map-reduce pattern splits the summarization task into two phases:

1. Map: Summarize each chunk independently, in parallel.
2. Reduce: Combine all chunk summaries into a final summary.

def map_reduce_summarize(chunks: list[str]) -> str:
    # Map: summarize each chunk independently
    chunk_summaries = [summarize_chunk(chunk) for chunk in chunks]

    # Reduce: combine all summaries into one
    combined = "\n\n".join(
        f"Section {i+1} summary:\n{s}"
        for i, s in enumerate(chunk_summaries)
    )
    reduce_prompt = (
        "You are a research assistant. Below are summaries of sequential sections "
        "of a longer document. Synthesize them into a single coherent summary "
        "that captures the key themes and most important information. "
        "Length: 3-5 paragraphs.\n\n" + combined
    )
    return ask(reduce_prompt, max_tokens=800)

Pros: Parallelizable (map calls can run concurrently), scales to arbitrary document length, predictable cost structure.
Cons: Each chunk is summarized in isolation — cross-chunk relationships may be missed. The reduce step must synthesize summaries, which may lose nuance.

Refine

The refine pattern processes chunks sequentially, each time improving a running summary with new information from the next chunk:

def refine_summarize(chunks: list[str]) -> str:
    # Initialize with first chunk
    summary = ask(f"Summarize the following text:\n\n{chunks[0]}", max_tokens=400)

    # Refine with each subsequent chunk
    for chunk in chunks[1:]:
        refine_prompt = (
            f"Here is a running summary of a document:\n\n{summary}\n\n"
            f"Here is additional content from the document:\n\n{chunk}\n\n"
            "Update and improve the summary to incorporate this new information. "
            "Keep the summary concise (3-4 paragraphs)."
        )
        summary = ask(refine_prompt, max_tokens=400)

    return summary

Pros: Each refinement has access to the accumulated context from all prior chunks — better cross-chunk coherence.
Cons: Sequential (cannot parallelize), accumulating cost per chunk, and the running summary may "drift" if early chunks are over-emphasized.

Full Context vs RAG Decision Guide

Many practitioners default to Retrieval-Augmented Generation (RAG) — retrieving only the most relevant document chunks for each query — as a scalable solution. But RAG has its own failure modes: retrieval can miss relevant context, and fragmented chunks may lack the holistic understanding needed for some tasks.

Cost Implications

API pricing for Claude and most other frontier models is based on input tokens + output tokens, with input tokens priced lower than output tokens. Processing a 100,000-token document in a single call costs significantly more per query than processing chunked summaries — but may require fewer total queries if the document only needs to be processed once.

Key cost considerations:

• Single-pass: High per-query cost, but simplest implementation. Amortized cost is low if you ask many questions about the same document.
• Map-reduce: N map calls (small, cheap) + 1 reduce call (medium). Good for one-time summarization.
• RAG: Variable cost per query based on retrieved chunk size. Lowest cost for targeted queries against large corpora.

Further Reading

• Liu et al. (2023) — Lost in the Middle: How LLMs Use Long Contexts
• Anthropic Docs — Context Windows
• LangChain Summarization Docs — map_reduce and refine patterns
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al.)