LLM Observability in Production

You cannot improve what you cannot measure. Observability turns your LLM application from a black box into a system you can understand, debug, and optimise under real-world load.

Learning Objectives

• Implement structured request/response logging with token counts and latency
• Integrate distributed tracing with Langfuse, LangSmith, or Phoenix/Arize
• Define and track key metrics: latency percentiles, throughput, error rates, cost
• Build alerts and dashboards for anomaly detection
• Set up feedback loops to close the gap between monitoring and improvement

Logging: The Foundation

Every LLM call should produce a structured log entry. Unstructured print() statements do not scale. Structured JSON logs let you query, filter, and aggregate.

What to Log

import json, time, hashlib, logging

logger = logging.getLogger("llm")

def log_llm_call(request_id, model, prompt, response, start_time):
    entry = {
        "request_id": request_id,
        "model": model,
        "prompt_hash": hashlib.sha256(prompt.encode()).hexdigest()[:12],
        "input_tokens": response.usage.input_tokens,
        "output_tokens": response.usage.output_tokens,
        "latency_ms": round((time.time() - start_time) * 1000),
        "status": "success",
    }
    logger.info(json.dumps(entry))

Tracing: Following the Thread

A single user action may trigger multiple LLM calls, retrieval steps, and tool invocations. Tracing links these into a single timeline so you can see where time and tokens are spent.

Tracing Platforms

Trace Anatomy

Metrics: What to Measure

Raw logs become actionable through aggregated metrics. Focus on the four golden signals adapted for LLM systems.

Monitoring & Alerting

Metrics without alerts are just numbers. Define thresholds that trigger action before users notice problems.

Alert Strategy

Dashboard Essentials

• Real-time panel: request rate, active errors, p50/p95 latency (last 15 min)
• Trends panel: daily cost, token usage, error rate over 30 days
• Model comparison: latency and cost side-by-side across model versions
• Anomaly detection: statistical deviation from rolling baselines (z-score or IQR)

Debugging: Conversation Replay & Prompt Versioning

When something goes wrong, you need to reproduce the exact conditions. This requires two capabilities:

# Langfuse prompt versioning example
from langfuse import Langfuse

langfuse = Langfuse()

# Fetch a versioned prompt
prompt = langfuse.get_prompt("summariser", version=3)
compiled = prompt.compile(topic="quarterly earnings")

# The trace automatically links to prompt name + version
trace = langfuse.trace(name="summarise-earnings")
generation = trace.generation(
    name="llm-call",
    model="claude-sonnet-4-20250514",
    prompt=prompt,        # links prompt version to trace
    input=compiled,
)

Feedback Loops

Observability is incomplete without user signal. Automated metrics tell you what happened; user feedback tells you whether it was good.

Feedback Collection Patterns

• Thumbs up / down — lowest friction, highest volume. Attach to trace ID.
• User corrections — when users edit AI output, capture the diff as implicit negative feedback.
• Explicit ratings — 1-5 stars on specific dimensions (accuracy, helpfulness, tone).
• Escalation signals — "talk to a human" button clicks indicate failure.

Closing the Loop

# Sending feedback to Langfuse
langfuse.score(
    trace_id="trace-abc-123",
    name="user-thumbs",
    value=1,             # 1 = positive, 0 = negative
    comment="User clicked thumbs-up"
)

Lab & Homework