Theory: Structured Outputs & JSON Mode

Why reliable structure is hard to get from LLMs, and three progressively robust ways to get it.

Introduction

Large language models are trained to produce fluent, helpful text. They are not trained to produce valid JSON. These goals frequently conflict. A model that wants to be helpful will add a polite preamble ("Sure! Here is the JSON you requested:") before the JSON, rendering the raw output unparseable. A model that learned from diverse text data may silently invent field names that weren't in your schema. A model generating a long JSON object may truncate it mid-key if it hits the max_tokens limit.

Production applications that consume LLM outputs downstream — databases, APIs, UIs, analytics pipelines — need a contract: a guaranteed shape, type, and completeness of data. This lesson covers the toolbox for enforcing that contract.

Why Structured Output Matters

Consider a pipeline that extracts information from 10 000 customer emails per day and stores results in a PostgreSQL database. The schema requires five fields: sender, intent, urgency (1–5), product_mentioned, and action_required. If 1% of extractions fail to parse, that is 100 failed records per day — leading to missing data, failed foreign key constraints, and silent corruption.

The problem is compounded by the nondeterministic nature of LLMs. Even with temperature=0, model outputs can vary across versions, across deployment configurations, and across prompt phrasings that seem equivalent. A system that worked yesterday may fail today after a model update.

Structured output approaches address this by either constraining the model's output generation (tool use, grammar-constrained decoding) or by validating and retrying when the output doesn't conform (Pydantic + retry loops).

Approach 1: Prompt-Based JSON

The simplest approach is to instruct the model in the prompt to return only valid JSON in a specified schema. This works surprisingly well most of the time — modern large models understand JSON schemas and follow instructions faithfully. But "most of the time" is insufficient for production systems.

Common failure modes include:

• Model adds a prose introduction before the JSON
• Model wraps JSON in markdown code fences (```json ... ```)
• Model invents field names not in the schema
• Model uses wrong types (string where integer expected)
• Model truncates the JSON if output approaches max_tokens

Defensive coding helps: strip code fences, use try/except json.JSONDecodeError, and implement a retry on parse failure. But the fundamental issue — that the model can produce anything — is not resolved by prompting alone.

Approach 2: Tool Use

Anthropic's tool use API (also called function calling) was designed partly to solve the structured output problem. When you define a tool with a JSON Schema input specification and force the model to call that tool using tool_choice={"type": "tool", "name": "..."}, the API enforces the schema at the level of the message contract.

The model must return a tool_use content block whose input field matches the declared schema. If a required field is absent from the source text, the model fills it with a type-appropriate default or null. Invalid types are coerced or raise an error before leaving the API.

This eliminates the most common failure modes. The trade-off is slightly higher latency (the tool call protocol has overhead), slightly higher token count, and a more complex response parsing path — you must handle ToolUseBlock rather than a plain text string.

# The response content is a list that may contain ToolUseBlock items:
for block in response.content:
    if block.type == "tool_use":
        data = block.input  # guaranteed to match the schema
        break

Pydantic v2

Pydantic is the de facto standard for data validation in Python. Version 2, rewritten in Rust for performance, is significantly faster than v1 and has a cleaner API. Define your expected structure as a BaseModel subclass, and Pydantic handles type coercion, validation, and serialisation.

Key patterns:

• model_validate(dict) — validate a dict, raise ValidationError on failure
• model_validate_json(str) — parse JSON string and validate in one step
• model_dump() — serialise to a plain dict
• model_json_schema() — generate the JSON Schema (useful for building tool definitions)
• @field_validator — define custom validation logic per field

from pydantic import BaseModel

class JobPosting(BaseModel):
    title: str
    company: str
    location: str
    salary_range: str | None = None  # Optional field with default
    remote: bool
    requirements: list[str]

# Generate JSON Schema for use as a tool definition:
print(JobPosting.model_json_schema())

Retry and Recovery

Even with tool use, edge cases can cause validation failures — particularly when source data is ambiguous or incomplete. A retry loop that feeds the error back to Claude is highly effective:

1. Make the initial extraction request
2. Attempt to parse and validate the response
3. On failure, append the assistant's response and a user message containing the error details to the messages list
4. Make another request with the updated messages list
5. Repeat up to max_retries times

The key insight is that Claude can understand Python exception messages. A ValidationError from Pydantic describes exactly which field failed and why. Claude can use that information to fix the specific problem. Empirically, the first retry resolves approximately 80% of failures, and the second retry resolves most of the remainder.

Choosing an Approach

Further Reading

• Anthropic Tool Use documentation
• Pydantic v2 documentation
• JSON Schema — Getting Started
• Instructor library — a popular open-source wrapper that automates tool use + retry for structured extraction