Cost, Caching & Rate Limits

Production LLM applications must manage costs, leverage caching for efficiency, handle rate limits gracefully, and use the Batch API for high-volume workloads. This lesson covers the economics and operational patterns of Claude at scale.

Learning Objectives

• Understand Claude’s token pricing model (input, output, cache read/write)
• Implement prompt caching to reduce cost and latency
• Know the cache TTL rules and gotchas
• Use the Batch API for 50% cost savings on async workloads
• Handle rate limits with exponential backoff

Token Pricing

Claude uses per-token pricing with separate rates for input and output tokens.

MTok = million tokens. Output tokens cost 5× more than input tokens because generation is more compute-intensive than processing input.

Prompt Caching

Prompt caching lets you cache frequently reused prefixes (system prompts, long documents, tool definitions) so you pay a reduced rate on subsequent requests.

response = client.messages.create(
    model="claude-sonnet-4-20250514",
    max_tokens=1024,
    system=[{
        "type": "text",
        "text": LONG_SYSTEM_PROMPT,      # 2000+ tokens
        "cache_control": {"type": "ephemeral"}
    }],
    messages=[{"role": "user", "content": "Hello!"}]
)

Cache TTL & Gotchas

Important rules for prompt caching:

• Minimum size: Cached content must be at least 1,024 tokens (Sonnet/Opus) or 2,048 tokens (Haiku)
• TTL: 5 minutes, refreshed on each cache hit
• Prefix matching: The cache matches from the beginning. If the cached prefix changes even by one token, the cache misses
• Multiple breakpoints: You can set up to 4 cache breakpoints in a single request

Batch API

For workloads that do not need real-time responses, the Batch API offers 50% cost savings with a 24-hour SLA.

batch = client.messages.batches.create(
    requests=[
        {
            "custom_id": f"task-{i}",
            "params": {
                "model": "claude-sonnet-4-20250514",
                "max_tokens": 1024,
                "messages": [{"role": "user", "content": task}],
            }
        }
        for i, task in enumerate(tasks)
    ]
)
print(f"Batch ID: {batch.id}")  # Poll for results

Rate Limits

Claude enforces rate limits at two levels:

When you hit a limit, the API returns 429 Too Many Requests with a retry-after header.

import time

def call_with_backoff(fn, max_retries=5):
    for attempt in range(max_retries):
        try:
            return fn()
        except anthropic.RateLimitError as e:
            wait = 2 ** attempt  # 1, 2, 4, 8, 16 seconds
            print(f"Rate limited. Waiting {wait}s...")
            time.sleep(wait)
    raise Exception("Max retries exceeded")

Demo

Live demonstration: prompt caching savings and batch API usage.

Lab & Homework