Homework — Structural Evals, an MCP Second Tool, and a Haiku Budget

Task 1 — Write a structural eval for the L04 research assistant

Create eval_structural.py at the repo root (or inside agents/_eval/ — either is fine, just be consistent with your imports). Import run_research_agent from L04 and assert that the returned report dict has the following structural shape:

• intro — a string of length ≥ 20 chars.
• bullets — a list of length 2–5.
• sources — a list of length ≥ 1.

Run the eval on three different questions — pick ones with different shapes (a factual lookup, an open-ended “what changed” question, and a comparison-style question). Print a one-line OK/FAIL per question with the structural counts so a failure is obvious at a glance:

[OK ] 'What changed in Linux kernel 6.14...'  -> {'has_intro': True, 'n_bullets': 3, 'n_sources': 2}
[FAIL] 'Compare ext4 and btrfs for the 6.14 era' -> {'has_intro': True, 'n_bullets': 1, 'n_sources': 2}

✓ Self-check: at least one question passes; failures (if any) emit a clear assertion-style message that names the dimension that failed (intro / bullets / sources) and the actual count. Do not hard-fail the whole script on the first failure — the point of a structural eval is to surface all the failures in one run, not just the first one.

Task 2 — Add an MCP-hosted second tool (pkb_count)

Extend the stub MCP server at agents/_eval/mcp_stub/server.py with a second tool, pkb_count() -> int, that returns the number of entries in the in-process PKB dict. Then update the client-side agent_with_mcp.py (from the Lab 12 MCP-bridge sidebar) so the agent sees the new tool — remember the agent only knows about tools you wire into its tools=[...] list.

# In agents/_eval/mcp_stub/server.py, inside list_tools():
Tool(
    name="pkb_count",
    description="Return the number of entries in the PKB.",
    inputSchema={"type": "object", "properties": {}, "required": []},
),

# And in call_tool():
elif name == "pkb_count":
    return [TextContent(type="text", text=str(len(PKB)))]

On the client side, add a second @tool wrapper in agent_with_mcp.py that proxies the MCP call:

@tool
async def pkb_count() -> str:
    """Return the number of entries currently in the PKB."""
    result = await session.call_tool("pkb_count", {})
    return result.content[0].text

Run the agent with the prompt “How many entries are in the knowledge base?” and inspect the tool-call trace.

✓ Self-check: the agent calls pkb_count exactly once (not twice, not zero times) and quotes the returned number verbatim in its reply. If it calls pkb_search first to “look around” before counting, that's a tool-selection failure worth noting — add a sentence to the tool's description telling the model to use it directly for “how many” questions and re-run.

Task 3 — Latency & cost budget for a slimmer model (Haiku)

Re-run agents/lesson04-tool-heavy-agent/agent.py with model="claude-3-5-haiku-latest" instead of the default Sonnet. Measure two numbers across three runs of the same question (pick one from Task 1):

• Cost per task — average across the three runs.
• Median latency — wall-clock seconds from run_research_agent entry to return.

Record the Sonnet baseline (from your Lab 12 numbers) alongside the Haiku numbers in a small markdown table:

| Model                       | Cost/run | Median latency | Quality (vibe) |
|-----------------------------|----------|----------------|----------------|
| claude-3-5-sonnet-latest    | $0.034   | 18s            | sources cited, bullets specific |
| claude-3-5-haiku-latest     | $0.008   | 9s             | bullets sometimes vague |

Then write a 2–3 sentence recommendation: for which class of workload would you pick Haiku, and for which class would you pick Sonnet?

✓ Self-check: Haiku should land roughly 3–5× cheaper and 2× faster than Sonnet on the same question. The answer quality may degrade (sometimes substantially) for the research-assistant shape — Haiku is great at transformation, weaker at multi-source synthesis. Your recommendation should name the dimension that wins for each workload (e.g., “Haiku for triage / classification / first-pass; Sonnet for any agent where source attribution matters”).