Lesson 12 — Cost, Latency & Evals

Operating agents in production: knowing what you spend, what you wait for, and whether the answer is right.

Agents Module 6 Operations MCP Bridge

Three pillars of operating an agent

• Cost — you have a budget (per task, per user, per day).
• Latency — your user has patience (~3s for chat, ~30s for batch).
• Evals — you need to know if the agent is right, not just whether it ran.
• Plus an implicit fourth — observability — which L11's scorecard already covered.

Cost — what actually drives the bill

• LLM tokens (input + output) — almost always the majority of cost.
• Tool execution costs — usually negligible unless the tool calls another paid API (Tavily, Serper, a SaaS database).
• Framework overhead — usually zero (most frameworks are free / self-hosted).
• Vendor lock-in tax — switching cost when you want to leave.

Token accounting in code

total_in = total_out = total_cost = 0.0
for event in agent.stream(question):
    if event.type == "message_stop":
        total_in += event.usage.input_tokens
        total_out += event.usage.output_tokens
        total_cost += event.usage.cost_usd     # if the SDK provides it

Each framework exposes usage differently — Claude SDK on the streaming event, OpenAI SDK on Runner.run_result.usage, LangGraph via callbacks, CrewAI by aggregating its sub-agents. The L11 eval harness abstracts this.

Cost caps you should always have

Latency — what your user actually waits for

• LLM round-trip — usually 1–3s per turn for Sonnet, 3–8s for Opus.
• Tool execution — variable. Real web search adds 200–800ms; a stubbed tool adds <1ms.
• Network — 50–200ms each direction; matters less for long-context turns.
• Framework overhead — usually <50ms; framework choice rarely moves latency by more than a few percent.

Latency budgeting

Streaming makes latency feel different

• agent.stream(...) lets you render token-by-token.
• Total wall-clock is the same, but time-to-first-token drops to ~500ms.
• For chat UX, this is the single biggest perceived-performance lever.
• All four frameworks support streaming; the L03 lab covered the Claude SDK form.

Evals — beyond “did it crash?”

• Smoke evals: does the agent return something on a representative input? (What the L11 harness does.)
• Structural evals: does the output have the expected fields (intro / bullets / sources)? Easy — parse the write_report tool call.
• Quality evals: is the answer good? Hard — usually LLM-as-judge or a held-out test set with human-rated answers.
• Regression evals: did today's commit make any task worse than yesterday's?

A minimum eval rig

def eval_one(question: str, expected_bullets_substring: list[str]) -> dict:
    out = run_research_agent(question)
    bullets = out["report"].get("bullets", [])
    hits = sum(
        1 for substr in expected_bullets_substring
        if any(substr.lower() in b.lower() for b in bullets)
    )
    return {"q": question, "score": hits / len(expected_bullets_substring),
            "cost": out["cost_usd"], "latency_s": out["turns"]}

Substring-based scoring is crude but reproducible. Good first eval — replace with LLM-as-judge if you need precision.

MCP-bridge sidebar — what it is

• MCP (Model Context Protocol) = standard way for an agent to call tools hosted in a separate process / server.
• Why it matters here: you can write a tool once (as an MCP server) and have any MCP-aware agent call it — Claude SDK, OpenAI SDK, LangGraph, CrewAI all support MCP today.
• This is the bridge between this course (agents) and the Creating MCP Servers course (servers).

MCP-bridge sidebar — the demo

# Launch the stub server as a subprocess; connect via stdio.
from mcp.client.stdio import stdio_client, StdioServerParameters
async def demo():
    params = StdioServerParameters(
        command="python", args=["-m", "agents._eval.mcp_stub.server"]
    )
    async with stdio_client(params) as (read, write):
        async with ClientSession(read, write) as sess:
            await sess.initialize()
            result = await sess.call_tool(
                "pkb_search", {"query": "BIG TCP"}
            )
            print(result.content[0].text)

This is the client side; the server side is the stub we ship in agents/_eval/mcp_stub/server.py.

What the sidebar does NOT cover

Recap & what's next

• Token accounting is per-framework — shape differs across Claude SDK / OpenAI SDK / LangGraph / CrewAI; the eval harness normalises it.
• Latency budget = turns × per-turn time × (1 + tool_serial_factor); cut turns or model before changing frameworks.
• Eval at minimum the structural shape — quality evals can come later, but smoke + structural belong in CI.