Lesson 11 — Choosing the Right Framework

One scorecard, four sweet-spot agents, real numbers.

Agents Module 6 Scorecard Cross-Framework

Where we are

• You've now built four agents — one per framework — each at that framework's natural sweet spot.
• L04 (Claude Agent SDK), L06 (LangGraph), L08 (OpenAI Agents SDK), L10 (CrewAI).
• This lesson is the apples-to-apples lens you couldn't have had earlier — you needed all four agents to exist before any comparison was meaningful.
• The output: a scorecard you can re-run on your own machine, with your own questions.

The four agents (recap)

Each agent was built at the shape its framework was designed for — no contortions.

The scorecard

Why these seven (not five, not nine)

• Cost + latency + LoC are measured — the harness records them per run.
• Observability + eval + lock-in + failure-recovery are editorial-but-defensible — the eval harness emits a rubric score 0–3 for each.
• Seven is what fits a comparison table without scrolling sideways.
• Adding more dimensions (DX, docs quality, community size) starts to bias toward whoever writes the rubric — we leave those to taste.

The eval harness

agents/_eval/
  __init__.py
  scorecard.py       # ScorecardRow dataclass + render helpers
  run_all.py         # main runner: imports all 4 agents, measures, emits JSON+MD
  results.json       # generated artefact (gitignored)
  results.md         # generated artefact (gitignored)
  mcp_stub/          # used by L12's sidebar, not L11

The lab walks you through running it — one command, one minute, your own numbers.

What the harness measures (live)

• Wraps run_research_agent / run_pipeline_agent / run_handoff_agent / run_crew_agent with a timer.
• Reads cost_usd from each agent's returned dict — every sweet-spot agent exposes this field.
• Runs cloc (or a Python LoC counter fallback) on the underlying agent.py.
• Repeats across 3 questions to compute a median, not a single sample.

What the harness scores (rubric)

• Observability (0–3): does the framework emit per-step events? Does it ship with a tracing UI? Is the trace easily exportable?
• Eval surface (0–3): can you intercept tool calls / final outputs? Is there a documented eval mode?
• Lock-in (0–3, higher = more portable): how many places does the code mention “OpenAI” / “Anthropic” / “LangSmith”?
• Failure-recovery (0–3): built-in retries? Timeouts? Prompt-injection defences? (See the AI Prompting Security course for the last point.)

A real scorecard (2026 Q2 snapshot)

Example numbers — the lab produces the real ones for your machine and model choices.

What jumps out

• OpenAI SDK wins on latency + cost for this sweet-spot suite. Why? Handoffs are the cheapest shape — mostly text, fewer tools.
• LangGraph wins on observability + eval — because you can intercept any node.
• CrewAI is highest LoC + cost — three role-played agents talk a lot. That's the price of the role-play frame.
• Claude SDK is the median across all dimensions. Solid default unless one of the others' wins matter for your case.

What the numbers DON'T tell you

Decision matrix

What the harness CAN'T be replaced by

• Hand-rolled spreadsheets bias toward the framework the author already likes.
• The harness is deliberately not a vibes-check — it runs the same task set against every agent, in the same loop, with the same timer.
• If you disagree with a number, change the harness — don't lobby for editorial overrides.
• This is the same discipline as benchmarks in any other engineering domain: reproducible > opinionated.

Recap & what's next

• Seven dimensions — three measured, four rubric-scored 0–3.
• An eval harness produces the numbers — not editorial taste.
• Decision matrix as a starting point, not as a verdict — your context matters.