Solutions — Structural Eval, MCP Second Tool, Haiku Budget

Lesson 12 · Choosing & Shipping · worked answers with commentary

Below is one clean answer per homework task, plus the why-this-not-that commentary that distinguishes a passing solution from one that’s actually useful in production. Compare your attempt against these — the numbers in Task 3 in particular will look different on your machine, but the shape of the trade-off should match.

Solution to Task 1 — a structural eval for the L04 report

Write eval_structural.py next to the eval harness (or inside agents/_eval/ if you prefer to keep eval code co-located). Import the agent entrypoint, run it against three different questions, and assert on the shape of the returned report dict — not on the prose itself.

"""Structural eval for the L04 research assistant."""
from agents.lesson04_tool_heavy_agent.agent import run_research_agent

QUESTIONS = [
    "What changed in Linux kernel 6.14's networking stack?",
    "What's notable in 6.14's filesystem changes?",
    "What scheduler improvements landed in 6.14?",
]

def evaluate(report: dict) -> dict:
    return {
        "has_intro": len(report.get("intro", "")) >= 20,
        "n_bullets": len(report.get("bullets", [])),
        "n_sources": len(report.get("sources", [])),
    }

for q in QUESTIONS:
    out = run_research_agent(q)
    e = evaluate(out["report"])
    ok = e["has_intro"] and 2 <= e["n_bullets"] <= 5 and e["n_sources"] >= 1
    print(f"[{'OK ' if ok else 'FAIL'}] {q!r}  -> {e}")

Run it three times; you should see at least one OK line and any FAIL lines should print the offending counts so you can see why they failed (zero sources? six bullets? a one-word intro?).

Lesson. Structural evals are the cheapest, fastest eval rung. They don’t need a judge LLM, they don’t need a reference answer, and they catch the failure modes that matter most in production: the agent didn’t return what it promised to return. Stack them at the front of any eval pipeline — only run the expensive LLM-as-judge stages on the runs that pass the structural gate.

Solution to Task 2 — add an MCP-hosted pkb_count tool

Two changes: one server-side (the MCP stub gains a new tool), one client-side (the agent declares it so the LLM knows it’s available). Both edits are surgical.

Server-side — in agents/_eval/mcp_stub/server.py:

# In list_tools():
Tool(
    name="pkb_count",
    description="Return the number of entries in the PKB.",
    inputSchema={"type": "object", "properties": {}, "required": []},
),

# And in call_tool():
elif name == "pkb_count":
    return [TextContent(type="text", text=str(len(PKB)))]

Client-side — add a second @tool in agent_with_mcp.py:

@tool
async def pkb_count() -> str:
    """Return the number of entries currently in the PKB."""
    result = await session.call_tool("pkb_count", {})
    return result.content[0].text

Then ask the agent “How many entries are in the knowledge base?”. With the tool wired in, the agent should call pkb_count exactly once, get back a string like "7", and quote that number in its answer rather than hallucinating a count or refusing to answer.

Lesson. MCP tools cross three boundaries (server, transport, client) and any of those boundaries can silently fail to wire up. When debugging, work outwards from the server: confirm the tool appears in list_tools() output (you can dump it in the connect handshake), then confirm the client sees it, then confirm the agent has it in its tool list. The most common failure is the last hop, because the others usually raise on misconfiguration and the last one just shrugs.

Solution to Task 3 — latency budget for a slimmer model

Re-run agent.py with model="claude-3-5-haiku-latest", time three runs each for Sonnet and Haiku, take the median, and price each run from the harness’s token counters. A representative result — your numbers will differ, but the shape of the trade-off should match:

Roughly: Haiku is ~4× cheaper and ~2× faster, with a noticeable quality drop on this research-assistant shape. The cost / latency / quality triangle is tight — you almost never pick up two cleanly without giving back on the third.

Recommendation. Use Sonnet for any production agent where answer quality matters (research, customer-facing summaries, anything a human will read end to end). Use Haiku for triage / classification / “quick first pass” agents where the cost & latency win dwarfs the quality delta — routing, intent detection, draft generation that a Sonnet pass will polish.

Lesson. Model choice is its own decision, made after the framework choice and run on the same task suite. Re-run your evals when you switch models, the way you’d re-run them when you switch frameworks. The numbers above are illustrative; your numbers, on your tasks, are the ones that matter.

What you’ve built

Across three small extensions you’ve added structural rigour (cheap, fast, no-judge evals at the front of the pipeline), MCP-hosted tool composition (a second tool added in two surgical edits), and multi-axis model selection (cost × latency × quality measured on the same task suite) to the cross-framework operations toolkit. That’s the production template for cost / latency / eval discipline in three pieces:

• Structural evals first — gate everything else on shape, only spend judge tokens on runs that pass.
• MCP for tool reuse — tools written once, exposed to every framework, swappable without touching the agent runtime.
• Models as a separate axis — framework choice and model choice are independent; measure both, don’t conflate them.

Closing tip: You’ve now built the operations layer that sits across all four frameworks. Lesson 13 closes the course with where the field is going — what’s settled, what’s still unsettled, and one warning about the half-life of any framework pick.

← Back to homework