Modern models have enormous context windows — but bigger doesn't always mean better. Learn to work with long documents effectively and economically.
Context windows have grown dramatically since 2022. The constraint is no longer "fits in context?" but cost, latency, and attention quality.
| Model | Context window | Approx. pages | Notes |
|---|---|---|---|
| Claude Haiku 3.5 | 200 K tokens | ~500 pages | Low per-token cost |
| Claude Sonnet 4 | 200 K tokens | ~500 pages | Medium per-token cost |
| GPT-4o | 128 K tokens | ~320 pages | Medium per-token cost |
| Gemini 1.5 Pro | 1 M tokens | ~2 500 pages | Watch output costs |
Rule of thumb: 1 token ≈ 0.75 English words ≈ 4 characters. A 200-page PDF is roughly 50 K tokens — well within any modern model, but still costs money on every call.
Liu et al. (2023) demonstrated that LLM performance degrades for information placed in the middle of long contexts. Models recall the beginning and end much more reliably.
Key finding: on multi-document QA tasks, accuracy was ~20 percentage points lower when the relevant document was placed in the centre of a 20-document context versus at position 1 or position 20.
Recall quality by document position (schematic — start and end are high, middle dips):
← Start of context End of context →
When a document is too long or too expensive to fit in context, split it into chunks and process each independently.
| Strategy | How it works | Pros | Cons |
|---|---|---|---|
| Fixed-size with overlap | Split every N words; consecutive chunks share K words | Simple, predictable cost | May cut mid-sentence or mid-idea |
| Sentence-based | Split on sentence boundaries; group sentences until size limit | Preserves sentence integrity | Chunk sizes vary; needs NLP library |
| Semantic | Embed sentences; split where cosine similarity drops | Topically coherent chunks | Requires embedding model; slower |
Overlap tip: 10–15% word overlap between consecutive chunks prevents losing context at boundaries. For 500-word chunks, use ~50-word overlap.
Two battle-tested architectures for summarising documents that exceed your context window.
[chunk 1] → summary 1 ↘ [chunk 2] → summary 2 → COMBINE → final [chunk 3] → summary 3 ↗
All chunks processed independently (parallelisable). Fast, but may lose cross-chunk relationships.
[chunk 1] → running summary [chunk 2] + running → updated [chunk 3] + updated → final
Sequential — cannot parallelise. Better narrative coherence but O(n) calls; early errors propagate.
Choosing: use map-reduce when speed matters or chunks are independent. Use refine when narrative coherence is critical (fiction, legal briefs, medical notes).
Not every long-document problem needs chunking. Sometimes stuffing the whole document in is simpler and better; sometimes it's overkill or impossible.
| Signal | Full context | RAG / chunking |
|---|---|---|
| Document size | < 100 K tokens, cost acceptable | Exceeds model limit or too expensive per call |
| Update frequency | Static document, queried rarely | Frequently updated corpus |
| Query type | Whole-document synthesis or comparison | Specific fact lookup or search |
| Latency | Single call, predictable | Can be faster with small retrieval |
| Cost | Pays full context on every call | Only pay for retrieved chunks |
Decision rule: if the document fits in context and you query it infrequently, go full context. If it's a knowledge base queried hundreds of times a day, build RAG.
Live walkthrough of map-reduce summarisation on demo/sample_document.txt.
demo/sample_document.txt — a ~4 000-word history of AIchunk_text(text, chunk_size=500, overlap=50) — inspect chunk count and first chunk boundarymap_reduce_summarize(chunks) — watch intermediate summaries appearchunk_text(text, chunk_size, overlap); verify chunk count, boundary quality, and that overlapping words appear in consecutive chunkssummarize_chunk() and map_reduce_summarize(); run on the sample document and print intermediate summariesrefine_summarize(chunks) — start with a summary of chunk 0, then for each subsequent chunk ask Claude to refine the running summary. Compare result quality with map-reduce.