Homework

Objective

Empirically determine the best chunk size for summarising the sample document by comparing quality and cost at three different settings.

Instructions

1. Run your map_reduce_summarize function on demo/sample_document.txt with chunk sizes of 256 words, 512 words, and 1024 words (use overlap = 10% of chunk size in each case).
2. For each run, record:
   • Number of chunks created
   • Total input tokens (sum across all API calls)
   • Total output tokens
   • Estimated cost (use Haiku pricing: $0.25 / 1M input tokens, $1.25 / 1M output tokens)
   • Quality of the final summary (your subjective rating 1–5, plus a brief explanation)
3. Write a short paragraph (100–150 words) explaining which chunk size you would choose for a production system and why.

Deliverable

A comparison table + written recommendation. Add it as a Markdown cell at the end of your lab.pdf.

Objective

Implement the "refine" summarization pattern and compare it to map-reduce on quality and cost.

Instructions

1. Implement refine_summarize(chunks: list[str]) -> str:
   • Start with a summary of chunks[0]
   • For each subsequent chunk, call Claude with a prompt like: "Here is a running summary of a document so far: {running_summary}. Here is the next section: {chunk}. Please update and refine the summary to incorporate the new information."
   • Return the final running summary after processing all chunks
2. Run both map_reduce_summarize and refine_summarize on the same chunked document.
3. Compare the two outputs on:
   • Narrative coherence (does it read as a unified summary?)
   • Coverage (does it mention the main eras / events?)
   • Total cost
   • Total latency (wall-clock time)

Deliverable

Add refine_summarize to your lab notebook with a comparison table and a paragraph explaining when you would prefer each approach.

Objective

Derive algebraically at what document token length map-reduce becomes cheaper than single-pass, and verify your answer empirically.

Setup

Assume the following (adjust to match Haiku's current pricing if different):

• Input cost: $0.25 per million tokens
• Output cost: $1.25 per million tokens
• Chunk size: C = 500 tokens (after tokenization)
• Each chunk summary: ~100 output tokens
• Combine step: input = number_of_chunks × 100 tokens; output = 200 tokens
• Single-pass: input = D (document tokens) + 50 (prompt overhead); output = 200 tokens

Instructions

1. Write the cost formula for single-pass as a function of document tokens D.
2. Write the cost formula for map-reduce as a function of D (assuming number of chunks = D / C).
3. Set the two formulas equal and solve for D — this is the crossover point.
4. Verify: calculate both costs at D = 10 000, D = 50 000, and D = 200 000. Which method is cheaper at each size?
5. Show all algebra clearly in a Markdown cell with LaTeX or plain text equations.

Deliverable

A Markdown cell in your lab notebook showing the derivation, the crossover value of D, and the verification table.