Theory: Cost, Caching & Rate Limits

Managing the economics and operational constraints of production LLM applications.

Pricing Model

Claude’s API uses per-token pricing. Every API call is billed based on the number of tokens in the input (system prompt + messages + tool definitions) and the number of tokens in the output (the model’s response). Output tokens are significantly more expensive than input tokens — typically 5× more — because generating text requires more computation than processing it.

Understanding this pricing structure is essential for building cost-effective applications. A chatbot that generates long, verbose responses costs much more than one tuned for concise output. Similarly, a system prompt that includes 5,000 tokens of instructions sent with every message creates a large recurring input cost.

The response.usage object returned by every API call tells you exactly how many tokens were consumed:

print(response.usage)
# Usage(input_tokens=1523, output_tokens=287,
#       cache_creation_input_tokens=0, cache_read_input_tokens=0)

Prompt Caching

Prompt caching is Claude’s most impactful cost-optimization feature. When you mark a portion of your prompt with cache_control, the API caches it server-side. Subsequent requests that share the same prefix pay a drastically reduced rate for those cached tokens.

The savings are substantial: cached tokens cost 90% less than regular input tokens. The first request pays a 25% premium to write the cache, but every subsequent request within the TTL window reads from cache at 1/10th the cost.

response = client.messages.create(
    model="claude-sonnet-4-20250514",
    max_tokens=1024,
    system=[{
        "type": "text",
        "text": long_system_prompt,  # 2000+ tokens
        "cache_control": {"type": "ephemeral"}
    }],
    messages=[{"role": "user", "content": "What can you help me with?"}]
)

# First call: cache_creation_input_tokens > 0
# Second call: cache_read_input_tokens > 0 (90% cheaper)

The best candidates for caching are: long system prompts, tool definition blocks, few-shot example collections, and large documents that are referenced across multiple requests in a conversation.

Cache Breakpoints

You can place up to 4 cache breakpoints in a single request. A cache breakpoint marks the end of a cacheable prefix. The system caches everything up to that point.

Breakpoints must be placed strategically. The cache uses prefix matching: the cached content must match exactly from the beginning of the request. If anything before a breakpoint changes, the cache misses entirely.

response = client.messages.create(
    model="claude-sonnet-4-20250514",
    max_tokens=1024,
    system=[
        {"type": "text", "text": system_prompt,
         "cache_control": {"type": "ephemeral"}},  # breakpoint 1
    ],
    messages=[
        {"role": "user", "content": [
            {"type": "text", "text": long_document,
             "cache_control": {"type": "ephemeral"}},  # breakpoint 2
        ]},
        {"role": "assistant", "content": previous_response},
        {"role": "user", "content": new_question},
    ]
)

Batch API

The Message Batches API lets you submit a collection of requests that are processed asynchronously within 24 hours. In return, you get a 50% discount on both input and output tokens. This is ideal for workloads where latency is not critical.

batch = client.messages.batches.create(
    requests=[
        {
            "custom_id": f"item-{i}",
            "params": {
                "model": "claude-sonnet-4-20250514",
                "max_tokens": 1024,
                "messages": [{"role": "user", "content": prompt}],
            }
        }
        for i, prompt in enumerate(prompts)
    ]
)
# Returns immediately with batch.id
# Poll with: client.messages.batches.retrieve(batch.id)

Batch results are retrieved by polling the batch status. When the batch is complete, you can iterate over the results, each tagged with the custom_id you assigned. Results may complete in any order.

Common batch use cases include: classifying thousands of documents, generating summaries for a corpus, running evals across a test suite, extracting structured data from a large dataset, and translating content libraries.

Rate Limits

The API enforces rate limits to ensure fair usage and system stability. Limits vary by tier (determined by your usage history and spend) and by model. The three key limits are:

• Requests per minute (RPM): The number of API calls you can make per minute.
• Input tokens per minute (ITPM): The total input tokens across all requests per minute.
• Output tokens per minute (OTPM): The total output tokens generated per minute.

When you exceed a limit, the API returns HTTP 429 with headers indicating when you can retry. The retry-after header tells you how many seconds to wait.

Backoff

The standard approach to handling rate limits is exponential backoff with jitter. This means waiting progressively longer between retries, with a random component to prevent thundering herds when many clients are rate-limited simultaneously.

import time
import random
import anthropic

def call_with_backoff(fn, max_retries=5):
    for attempt in range(max_retries):
        try:
            return fn()
        except anthropic.RateLimitError:
            base_wait = 2 ** attempt  # 1, 2, 4, 8, 16
            jitter = random.uniform(0, base_wait * 0.5)
            wait = base_wait + jitter
            print(f"Rate limited. Retry {attempt + 1}/{max_retries} in {wait:.1f}s")
            time.sleep(wait)
    raise Exception("Max retries exceeded")

Optimization Patterns

Beyond caching and batching, several patterns help control costs in production:

• Model routing: Use Haiku for simple tasks (classification, extraction) and Sonnet/Opus only for complex reasoning. A router can save 60–80% by sending 80% of requests to the cheapest model.
• Output length control: Set max_tokens conservatively. A chatbot that only needs 200-word answers should not have max_tokens=4096.
• Conversation pruning: In multi-turn conversations, summarise older messages rather than sending the full history. This reduces input tokens linearly with conversation length.
• Token counting: Use the anthropic.count_tokens() method to estimate costs before making a call. Gate expensive operations behind budget checks.
• Monitoring: Track token usage per user, per feature, and per day. Set alerts for anomalies (a bug that generates 10× normal output is expensive).

Further Reading

• Prompt Caching Documentation
• Message Batches API
• Claude Model Pricing
• Rate Limits Documentation
• Token Counting