Lesson 12 Cost, Latency & Evals Across Frameworks
Theme:

Cost, Latency & Evals Across Frameworks

Lesson 12 · Module 6 · Choosing & Shipping · ~12 min read

Three pillars of operating an agent

Building an agent that runs once is one thing; operating it at scale is another. The difference between a demo and a production system isn’t the model or the framework — it’s the discipline of measuring what you spend, what you wait for, and whether the answer is right. Three pillars matter:

Lesson 11 covered the cross-framework comparison of cost + latency + observability with the scorecard. This lesson zooms into each pillar in turn, plus an explicit eval rig sketch that’s framework-agnostic. The MCP-bridge sidebar at the end is the bridge between this course’s agents and the Creating MCP Servers course’s tool ecosystem — the moment where a tool stops belonging to one agent and starts belonging to your whole stack.

Cost accounting in practice

Three layers of cost stack up in every agent run:

  1. Token cost (the majority). Input tokens + output tokens × per-million rates. Every framework returns usage data; the shape differs:
    • Claude SDK: result.usage.input_tokens, output_tokens, total_cost_usd (the SDK does the multiplication for you).
    • OpenAI Agents SDK: RunResult.usage is a dict with per-model breakdown — you compute USD yourself from the published pricing.
    • LangGraph: subscribe to on_llm_end callbacks; aggregate per node yourself.
    • CrewAI: the Crew.kickoff() returned object has usage_metrics; you sum across the role-played sub-agents.
    The L11 eval harness normalises this — each agent.py exposes a single cost_usd field in its return dict, so the scorecard treats all four frameworks the same way regardless of the underlying SDK’s shape.
  2. Tool execution cost. Usually zero (your code, your CPU). Non-zero when a tool calls a paid API: Tavily ($0.005/request), Serper (free tier then paid), a vector DB on a SaaS plan, etc. A tool-heavy agent making 10 paid API calls per task is sometimes more expensive in tool cost than in tokens — check both.
  3. Framework overhead. Approximately zero in dollars; non-zero in engineer-time. The “cost” of LangGraph is the hour you spent learning the DAG mental model — paid once. Don’t double-count this against running costs.

Three caps every production agent should have on day one:

Latency budgeting

Latency = LLM round-trip × turns × (tool wait factor). A useful back-of-envelope formula: 

budget_s = max_turns × per_turn_s × (1 + tool_serial_factor)

For the L04 research assistant, plausible defaults:

That’s ~45s for a worst-case run, ~18s for a typical run. If your UX needs <10s, cut max_turns first, then drop to a faster model (Haiku, GPT-4o-mini). Don’t switch frameworks for latency — L11’s scorecard shows frameworks vary by ~10–30% on latency, but the agent design (how many turns, how many tools per turn) varies by 200%+. The wrong knob to turn is the framework choice; the right knob is the loop budget.

Streaming changes the feel of latency without changing the wall-clock. Time-to-first-token drops to ~500ms on most frameworks. For chat UX, streaming is the single biggest perceived-performance lever. For batch jobs, streaming buys you nothing — the consumer is waiting for the final answer anyway, not the prefix.

Per use case, not per framework

Latency budgets should be set per use case, not per framework. A back-office summariser can take 60s; a customer-support reply can’t. Negotiate the number with whoever owns the UX, then build backwards from it — not the other way around.

Evals — beyond “did it crash?”

Four eval flavours, increasing in complexity and signal:

A minimum rig for the L04 research assistant: 

def eval_one(q, expected_bullets):
    out = run_research_agent(q)
    bullets = out["report"].get("bullets", [])
    hits = sum(
        1 for expected in expected_bullets
        if any(expected.lower() in b.lower() for b in bullets)
    )
    return {
        "q": q,
        "score": hits / len(expected_bullets),
        "cost_usd": out["cost_usd"],
        "n_bullets": len(bullets),
    }

Run this across 5–10 known questions. Track the average score over time. When it drops, investigate — don’t just bump the threshold.

Evals are a forcing function, not a substitute

Don’t let “we have an eval suite” become a substitute for “we know what right looks like.” Evals are a forcing function for clarity, not a replacement for judgment. A 90% quality score on a question set that doesn’t cover the real production traffic is worse than no eval — it gives false confidence.

Observability sketch

Every framework emits per-step events you can log:

Pipe the events to a structured logger (structlog, loguru) keyed by agent_id, turn, tool_name, and cost_so_far. That’s the minimum production setup — you can query it in any log aggregator without further investment. Step up to OpenTelemetry / Datadog / Honeycomb when the agent volume justifies it (typically >10k runs/day or multi-tenant traffic).

MCP-bridge sidebar

The Model Context Protocol (MCP) lets an agent call tools hosted in a separate process — possibly written in a different language, possibly running on a different host. The pitch: write a tool once as an MCP server; any MCP-aware agent (Claude SDK, OpenAI SDK, LangGraph via adapter, CrewAI via adapter) can call it without re-implementation.

In 2026 Q2 the wins are:

The course bundles a stub MCP server at agents/_eval/mcp_stub/server.py that exposes pkb_search over stdio (the simplest MCP transport). The L12 lab connects to it from a Claude SDK agent — the same agent shape as L04, but with the tool now hosted in a separate process. You launch the server as a subprocess, the agent speaks MCP over the subprocess’s stdin/stdout, and the rest of the loop is unchanged.

What’s not covered here:

For all of the above, see the Creating MCP Servers course (mcp-servers/). This lesson’s sidebar is the bridge; that course is the destination.

Production checklist

A reader-takeaway list. Treat it as the floor, not the ceiling — if any item is missing in your deployment, fix it before adding new features:

  1. Cost cap per task. max_cost_usd= or equivalent. Always.
  2. Token + cost logging per turn. Structured, indexable, queryable.
  3. Latency budget documented. Agreed with whoever owns the UX.
  4. Smoke + structural evals in CI. Every PR runs them; red blocks merge.
  5. Quality evals in a slower lane. Nightly run on a fixed question set; trend the average.
  6. Streaming on user-facing endpoints. Buys you perceived-latency for free.
  7. MCP for any tool used by >1 agent. Reuse beats re-implementation.
  8. An obvious kill switch. Env var or feature flag that disables the agent on the inevitable bad day — no code change, no deploy.

References

LAST-REVIEWED: 2026-05-25