Lesson 06 Structured Outputs
Theme:

Homework

Three tasks that push structured extraction into your own domain and stress-test the reliability of each approach.

Complete all three lab exercises before starting the homework. Tasks 2 and 3 are extensions of the lab work.

📥 Reference Solution

⬇ lab_answer.pdf

Task 1 — Build Your Own Domain Extractor

Objective

Apply structured extraction to a domain you care about. The goal is to demonstrate that the pattern generalises beyond job postings.

Instructions

  1. Choose one of the following domains (or propose your own with instructor approval):
    • Invoice: vendor name, invoice number, date, line items (description + amount), total, currency
    • Meeting notes: meeting title, date, attendees, decisions made, action items (owner + task + due date), next meeting date
    • Bug report: title, severity (critical/high/medium/low), environment, steps to reproduce, expected behaviour, actual behaviour, reporter
    • Research paper abstract: title, authors, year, problem statement, methodology, key findings, limitations
  2. Define a Pydantic model with at least 5 fields. Include at least one optional field and one list field.
  3. Implement an extraction function using whichever approach (prompt-based or tool use) you prefer.
  4. Collect or write 5 realistic examples in your chosen domain. Run the extractor on all 5.
  5. Document: what extraction quality did you get? Were there any systematic failures?

Deliverable

A Jupyter notebook (or Python script) with the Pydantic model, extraction function, 5 test inputs, outputs, and a brief written evaluation.

Task 2 — Retry with ValidationError Feedback

Objective

Extend your retry implementation to send the full ValidationError back to Claude. Empirically test whether the error message helps Claude fix the specific problem.

Instructions

  1. Take your extract_with_retry function from the lab and enhance it:
    • On ValidationError, format the error as: f"Validation failed:\n{e}\n\nPlease fix the JSON and return only the corrected JSON."
    • On json.JSONDecodeError, format as: f"JSON parse error: {e}\n\nReturn valid JSON only, no markdown."
  2. Create 3 deliberately tricky inputs:
    • Input that causes a type error (e.g., salary as an integer instead of a string)
    • Input with a missing required field (e.g., no company name)
    • Input that is ambiguous enough that the model might hallucinate a field
  3. Run the retry function on each input with logging enabled (print attempt number and the error message sent back to Claude).
  4. Document: did the error feedback help? How many retries were needed? Were there cases where the model still failed after max_retries?

Deliverable

An extension to your lab notebook with the 3 test inputs, log output, and a paragraph of findings.

Task 3 — Reliability Benchmarking

Objective

Empirically measure the failure rate of prompt-based vs tool-use extraction on a standardised sample set.

Instructions

  1. Create or collect a set of 20 text samples in your chosen domain. They should vary in quality and completeness — include some that are well-structured and some that are minimal or ambiguous.
  2. Run prompt-based extraction on all 20 samples. Record a failure for each sample that raises json.JSONDecodeError or pydantic.ValidationError.
  3. Run tool-use extraction on all 20 samples. Record failures (Pydantic validation errors only — JSON parse errors should not occur).
  4. Build a results table: Sample ID | Prompt-based result (pass/fail) | Tool-use result (pass/fail) | Notes
  5. Calculate failure rate for each approach. Note any patterns in which samples caused failures.

Deliverable

A results table (as Markdown or CSV) plus a 150–200 word write-up covering: which approach was more reliable, by how much, what types of inputs caused failures, and whether the reliability difference justifies the extra complexity of tool use for your domain.

Efficiency tip: to avoid excessive API costs, use claude-haiku-4-5 for all 40 calls. Consider setting a spending limit in your Anthropic console before starting this task.

Submission Checklist