Lab 11 — Running the cross-framework scorecard

Lesson 11 · Module 6 · Choosing & Shipping · ~30 min hands-on, ~$0.30 budget

Goal: by the end of this lab you will have (a) run agents/_eval/run_all.py end-to-end (~5 min, ~$0.30), (b) opened results.md and read each row, (c) modified the question set in run_all.py to add a question that matters to you, and (d) re-run the harness and observed which dimensions shift the most. By the end you’ll have your own apples-to-apples scorecard for the four sweet-spot agents shipped in Modules 2–5.

Prereqs

All four sweet-spot agents work end-to-end. If any of L04 / L06 / L08 / L10 labs are broken in your checkout, fix those first — the harness imports each agent.py and calls its top-level entry point, so a single broken module blocks the whole table.
agents/requirements.txt installed. It pins all four framework SDKs plus tabulate for the Markdown table renderer. From the repo root:
```
pip install -r agents/requirements.txt
```
ANTHROPIC_API_KEY and OPENAI_API_KEY exported in your shell. The Claude Agent SDK and CrewAI default to Anthropic; the OpenAI Agents SDK and LangGraph default to OpenAI. The harness runs all four, so you need both.
About $0.30 of API budget for the harness’s default run (3 questions × 4 frameworks × 1 run each).
About 5–8 minutes of wall-clock time for the default run.
Working directory at the repo root (the paths below assume that).

Step 1 — Sanity-check the harness imports cleanly

Before paying for any API calls, make sure all four agent modules can be imported. The cheapest way is to ask the runner for its argparse help text — that exercises every import without firing any LLM calls.

python agents/_eval/run_all.py --help

Expected: argparse help text listing --questions, --runs, and --out-dir (or similar) flags. No traceback.

If you see ModuleNotFoundError, one of the four agent modules failed to import — most likely an __init__.py missing from a lesson directory. Fix by running:

for d in agents/lesson0{4,6,8}-*/ agents/lesson10-*/; do touch "$d/__init__.py"; done

✓ Self-check: --help prints flags and exits 0.

Step 2 — Run the harness with defaults

Now the expensive part. From the repo root:

python agents/_eval/run_all.py

Expected:

4 === measuring NAME (path) === lines on stderr as each framework runs.
Roughly 5–8 minutes wall-clock total.
A Markdown table printed to stdout at the end.
Two new files written: agents/_eval/results.json and agents/_eval/results.md.

If a framework fails mid-run

If any framework raises during its run, the harness logs the error and keeps going — the failed framework’s row will show $0.0000 / 0.0s latency / its real LoC and rubric scores. Don’t claim victory until all four ran. Re-run the failing framework’s standalone lab (e.g. python agents/lesson06-graph-shaped-agent/agent.py) to debug it in isolation, then re-run the harness.

✓ Self-check: results.json and results.md both exist; all 4 frameworks have non-zero cost_usd.

Step 3 — Read `results.md`

cat agents/_eval/results.md

You should see:

One table row per framework (4 rows total).
Numeric columns — cost per task (USD), median latency (seconds), lines of code — populated with real numbers.
Rubric columns — observability, eval-surface area, vendor lock-in, failure-recovery — rendered as 0–3 integers (or thin / moderate / rich-style strings, depending on which fields the harness chose to render).
An “excerpts” section below the table showing the first ~120 chars of each agent’s final_text — a quick sanity-check that they all produced something coherent, not just an empty string.

Step 4 — Pick a winner per dimension

Walk through the table column by column and name a winner for each. You should see roughly this shape (exact numbers vary with model + question set):

Cheapest per task? Probably the OpenAI Agents SDK or the Claude Agent SDK — both are single-loop frameworks with no orchestration overhead.
Fastest median latency? Same suspects — fewer turns means fewer round-trips.
Fewest LoC? Almost certainly the Claude Agent SDK — the L04 agent.py is the tightest of the four.
Best observability? LangGraph — you get per-node state-machine traces for free.
Most portable (lowest vendor lock-in)? CrewAI — swap the underlying LLM in one config line.
Best failure-recovery primitives? LangGraph — conditional edges + checkpointing give you real retry/rollback semantics.

The whole point of this lab

No single framework wins everything. That’s by design. Pick a winner per the dimension that matters for your hypothetical project. If cost-per-task is dominant, you pick one column’s winner. If you need audit-grade observability, you pick a different one. The scorecard exists so you can have that conversation with numbers in hand instead of vibes.

Step 5 — Modify the question set

Open agents/_eval/run_all.py in your editor. Find the DEFAULT_QUESTIONS list (or similar — check the runner’s docstring for the exact name). Replace one of the default questions with something realistic for your domain. Examples:

Internal-tools: “Which on-call engineer owns the payments service this week?”
Customer-support: “A customer says their invoice from last month is wrong — walk through the resolution.”
Research: “Summarise the top three security advisories filed in the last 30 days.”

Then re-run with a smaller question set so iteration is fast:

python agents/_eval/run_all.py --questions 3

Expected behaviour

The rubric scores don’t change — observability, lock-in, and LoC are per-framework, not per-task. But cost and latency probably shift, possibly by 30%+. That’s the lesson: the scorecard is a function of your tasks, not just your framework. If you’re picking a framework for production, use production-realistic questions in the harness — not the demo defaults.

✓ Self-check: results.md shows different cost/latency numbers than Step 2; rubric columns are unchanged.

Step 6 — Bump `--runs` and check stability

A single run per question is noisy — especially for latency. If you’re going to present these numbers to a team, re-run with more samples:

python agents/_eval/run_all.py --runs 5

Expected:

Median latencies are more stable. The swing between two back-to-back runs should be much smaller than the swing with --runs 1.
Cost numbers barely move. Token counts per task are roughly deterministic for the same question (modulo sampling). Use --runs 5 when you need confidence intervals you’d defend in a design review.

Step 7 — Optional: install `cloc` for tighter LoC counts

The harness has a Python fallback for line-counting that excludes the mocked PKB / WEB_INDEX data dicts using a regex heuristic. It’s good but not perfect. For more accurate numbers, install cloc:

brew install cloc                    # macOS
sudo apt install cloc                # Debian / Ubuntu
python agents/_eval/run_all.py --questions 1 --runs 1   # quick re-run

Expected: LoC numbers may shift by ±10 lines per framework. If you’re going to cite LoC publicly (a blog post, a conference talk), use cloc so nobody can nitpick the heuristic.

What you should leave with

agents/_eval/results.md checked locally — your own apples-to-apples scorecard for the four sweet-spot agents.
The instinct that no framework wins everything — the right choice is task-shape-dependent, not religion-dependent.
The harness itself as a reproducible artefact you can re-run on any commit — e.g. after bumping a framework version in agents/requirements.txt, to catch regressions in cost or latency that the version bump introduced.
A populated results.json you can hand to downstream tooling (a notebook, a dashboard, a regression-detection script).

Troubleshooting

Common stumbles

ModuleNotFoundError: No module named 'agents.lesson04_tool_heavy_agent' — the lesson directory’s __init__.py is missing, or the directory name uses hyphens that Python can’t import as a module path. The harness expects the underscore-form package alias; see the runner’s docstring for the exact import shape it uses.
results.md shows $0.0000 for a framework — that framework’s run_* function didn’t put cost_usd in its return dict. Re-check the relevant lesson’s agent.py; all four were specified to return {"final_text": ..., "cost_usd": ..., "turns": ...} at minimum.
Wall-clock blew past 15 minutes — your network is slow OR one framework is misbehaving (rate-limited, retrying, etc.). Cancel with Ctrl-C, then re-run with --questions 1 --runs 1 to isolate the slow framework. The stderr === measuring NAME === lines will tell you which one is the offender.
Two consecutive runs disagree wildly on cost. Check that temperature is the same across runs and that you haven’t hit a provider-side cache (some providers cache identical prompts — the second hit can be 90% cheaper). Bump the question wording slightly to defeat the cache if you need a clean apples-to-apples measurement.