Theory

Reading time: ~20 minutes (~1500 words) · Prerequisites: Familiarity with the Claude API, basic Python, understanding of production deployment concepts

1. Why Observability Matters for LLM Applications

Traditional software is deterministic: given the same input, you get the same output. LLM applications are fundamentally different. The same prompt can produce different outputs depending on temperature, model version, and even server-side load balancing across replicas. This non-determinism means that conventional monitoring — checking whether the server returns 200 OK — is insufficient. You need observability: the ability to understand the internal state of your system from its external outputs.

Observability for LLM applications rests on three pillars:

Logs — structured records of individual requests and responses
Traces — end-to-end timelines linking multiple steps (retrieval, LLM calls, tool use) into a single user interaction
Metrics — aggregated numerical summaries computed from logs and traces (latency percentiles, error rates, cost)

Without these, debugging a production issue becomes guesswork. With them, you can answer questions like: "Why did this user get a bad answer at 3:42 PM?" or "Which prompt version is 15% cheaper without hurting quality?"

2. Structured Logging

The first step is capturing structured data for every LLM call. A structured log entry is a JSON object with well-defined fields, as opposed to a freeform string like print("Got response from Claude"). Structured logs enable programmatic querying, aggregation, and alerting.

Essential Fields

Every log entry should include:

request_id: a UUID that uniquely identifies this call, used for correlation
timestamp: ISO 8601 format with timezone (e.g., 2026-04-04T14:32:01.045Z)
model: the exact model string (e.g., claude-sonnet-4-20250514)
input_tokens and output_tokens: from the API response usage field
latency_ms: wall-clock time from request send to response complete
status: success, error type, or timeout
prompt_version: an identifier for which prompt template was used
cost_usd: computed from token counts and model pricing

Privacy Warning

Logging full prompts and responses may capture personally identifiable information (PII), medical data, or other sensitive content. In production, log only hashes or redacted summaries of prompt/response content. Store full payloads in a separate, access-controlled trace store with appropriate retention policies and encryption.

Implementation Pattern

The most maintainable approach is a wrapper function that intercepts every LLM call, records timing and metadata, and emits the log entry automatically. This avoids scattering logging code throughout your application:

import time, uuid, json, logging
import anthropic

logger = logging.getLogger("llm.calls")
client = anthropic.Anthropic()

def call_llm(messages, model="claude-sonnet-4-20250514", **kwargs):
    request_id = str(uuid.uuid4())
    start = time.perf_counter()
    try:
        response = client.messages.create(
            model=model, messages=messages, max_tokens=1024, **kwargs
        )
        elapsed = (time.perf_counter() - start) * 1000
        logger.info(json.dumps({
            "request_id": request_id,
            "model": model,
            "input_tokens": response.usage.input_tokens,
            "output_tokens": response.usage.output_tokens,
            "latency_ms": round(elapsed),
            "status": "success",
        }))
        return response
    except Exception as e:
        elapsed = (time.perf_counter() - start) * 1000
        logger.error(json.dumps({
            "request_id": request_id,
            "model": model,
            "latency_ms": round(elapsed),
            "status": "error",
            "error": str(e),
        }))
        raise

3. Distributed Tracing

Logging captures individual calls, but modern LLM applications involve chains of operations: a user query triggers a retrieval step, then an LLM call, possibly a tool invocation, then another LLM call to synthesise the final answer. Distributed tracing connects these steps into a single timeline.

Core Concepts

Trace: represents the complete user interaction from start to finish
Span: a single operation within a trace (e.g., one LLM call, one database query)
Parent-child relationships: spans can be nested to show that one operation triggered another

Platform Comparison

Langfuse is an open-source observability platform designed specifically for LLM applications. It provides a Python SDK that wraps your existing code with minimal changes. Key features include prompt management (versioned templates stored server-side), cost tracking, and a web UI for exploring traces.

LangSmith is LangChain's hosted tracing platform. If you are already using LangChain or LangGraph, LangSmith provides deep integration with automatic tracing of chains and agents. It also includes a dataset and evaluation pipeline for testing prompt changes before deployment.

Phoenix by Arize is an open-source tool focused on LLM evaluation and observability. It excels at embedding-level analysis — you can visualise how your retrieval embeddings drift over time, which is valuable for RAG applications. Phoenix uses OpenTelemetry under the hood, making it compatible with existing APM infrastructure.

For teams with existing monitoring infrastructure (Datadog, Grafana, New Relic), OpenTelemetry provides a vendor-neutral approach. You instrument your code with OpenTelemetry spans and export them to whatever backend you already use. This avoids adding yet another dashboard to your stack.

4. Key Metrics

Once you have logs and traces flowing, you need to aggregate them into metrics that tell you whether your system is healthy. The four most important metric categories for LLM applications are:

Latency Percentiles

Average latency is misleading because a few very slow requests can hide behind a good average. Use percentiles instead:

p50 (median): what the typical user experiences
p95: the experience for 1 in 20 users — this is usually what you set SLAs against
p99: the worst-case tail, often caused by retries, cold starts, or rate-limit backoff

For streaming applications, also track time to first token (TTFT) separately. Users perceive a fast TTFT as responsive even if the total generation takes several seconds.

Token Throughput

Track total input and output tokens per minute. This metric correlates directly with cost and helps you detect anomalies — a sudden spike in output tokens might indicate a prompt regression causing verbose responses, while a drop in throughput might indicate rate limiting.

Error Rates

Categorise errors by type: HTTP 429 (rate limited), HTTP 500 (provider outage), timeouts, and semantic errors — cases where the response was technically valid HTTP but the content was wrong, empty, or unparseable. Semantic error detection often requires custom validation logic specific to your application.

Cost Tracking

Compute cost per request by multiplying token counts by the model's per-token pricing. Aggregate this into cost per user, cost per feature, and daily/monthly burn rate. Cost anomalies are often the first signal that something has changed — a prompt regression that doubles output length doubles your bill.

5. Debugging: Conversation Replay and Prompt Versioning

When a user reports a bad response, you need to reproduce it. This requires two things:

Conversation replay means storing the complete message history (system prompt, user messages, assistant messages, tool results) for every trace. When a bug is reported, you look up the trace by ID, retrieve the exact inputs, and replay them against the same model. This confirms the bug. Then you replay against your fixed prompt or a different model to verify the fix.

Prompt versioning means treating your prompts like code: every change gets a version number, every trace is tagged with the prompt version it used, and you can compare metrics across versions. Langfuse provides built-in prompt management; alternatively, you can version prompts in your own codebase using git tags or a simple version table in your database.

6. Feedback Loops

Automated metrics tell you about system health, but they cannot tell you whether the user was satisfied with the response. For that, you need explicit user feedback.

The simplest pattern is a thumbs up / thumbs down button on every AI-generated response. Each vote is linked to a trace ID, so you can correlate feedback with the exact prompt, model, and response. Even a small feedback rate (5-10% of responses) provides statistically meaningful signal at scale.

More nuanced feedback includes user corrections (when users edit the AI output, the diff is implicit negative feedback), multi-dimensional ratings (accuracy, tone, completeness), and escalation signals (clicking "talk to a human" is strong negative signal).

The real value of feedback comes from closing the loop: aggregate feedback by prompt version and feature, identify patterns in low-scoring responses, curate those into evaluation datasets, use them to improve prompts or fine-tuning data, deploy the improvement, and measure whether feedback scores improve. This creates a continuous improvement cycle that compounds over time.

7. Building Your Observability Stack

For most teams, the recommended starting point is:

Week 1: Add structured logging to every LLM call (the wrapper pattern above). Ship logs to your existing log aggregator.
Week 2: Integrate Langfuse (or your preferred tracing platform) for distributed traces. Set up the web UI and explore a few real traces.
Week 3: Build a metrics dashboard with latency percentiles, error rates, and daily cost. Set up basic alerts.
Week 4: Add feedback collection (thumbs up/down) and link it to traces. Begin building your first evaluation dataset from real user interactions.

This incremental approach delivers value at each step without requiring a large upfront investment. The key principle is: start with logging, graduate to tracing, aggregate into metrics, close with feedback.