Choosing the Right Framework

Lesson 11 · Module 6 · Choosing & Shipping · ~12 min read

Why this lesson exists

You’ve now seen four agent frameworks at each one’s sweet spot — the Claude Agent SDK on a tool-heavy assistant (L04), LangGraph on a graph-shaped workflow (L06), the OpenAI Agents SDK on a triage-with-handoff (L08), and CrewAI on a role-played multi-agent crew (L10). The first instinct when comparing them is to argue from taste: “CrewAI is just LangChain with extra roles,” “the OpenAI SDK is too tied to OpenAI,” “LangGraph is overkill for anything under five nodes.” That conversation is unwinnable in the abstract because every framework has a real strength somewhere. This lesson does the comparison from numbers — a small eval harness runs each sweet-spot agent against the same task set and records seven dimensions of behaviour. The reader’s job in the lab is to run the harness, look at the table, and form their own opinion. The scorecard isn’t the answer — it’s the apparatus that gives you the answer for your own context.

The seven scorecard dimensions, defined precisely

Three of the seven are measured directly by the harness from the live run. Four are scored against a short rubric encoded in scorecard.py. Defined one by one:

• Cost per task (USD). Sum of input + output token cost across all LLM calls for one task, averaged over the harness’s question set. Pricing comes from each framework’s own returned usage data — the harness doesn’t recompute it. If a framework under-reports usage, the scorecard reflects that, which is itself useful information.
• Median latency (seconds). Wall-clock per task, measured around the agent’s run(...) call. Median across runs (not mean — one slow run shouldn’t dominate). Includes tool execution time; for our sweet-spot agents tools are mocked so this is mostly LLM round-trip plus framework overhead.
• Lines of code (LoC). cloc count on the agent’s agent.py file, excluding mocked data dicts (which are domain content, not framework wiring). If cloc isn’t installed, the harness falls back to a Python-only counter that skips blank lines and comment-only lines.
• Built-in observability (0–3). Editorial rubric scored by the harness based on three checks: (a) does the framework emit per-step structured events the reader can subscribe to? (b) does it ship a tracing UI or dashboard out of the box? (c) is the trace easily exportable as JSON for downstream observability tools? One point per yes.
• Eval surface area (0–3). Editorial rubric: (a) can you intercept individual tool calls? (b) can you intercept the final output before it returns to the caller? (c) is there a documented “eval mode”, dry-run, or replay primitive? One point per yes.
• Vendor lock-in (0–3, higher = MORE portable). Counts how easy it is to swap the model provider out: (a) does the agent’s config explicitly name a provider? (b) does the underlying SDK support multiple providers? (c) is the rewrite cost low (single import change vs. wholesale refactor)? Read this column with the polarity in mind — high is good (portable).
• Failure-recovery primitives (0–3). Built-in retries (1pt) + timeouts (1pt) + prompt-injection defences (1pt). For the third sub-dimension, “defences” means anything past “we hope the system prompt is enough” — input filtering, structured-output constraints, tool-call validation. See the AI Prompting Security course for what “real” defences look like in 2026 Q2.

Why “measured” vs “scored” matters

Cost, latency, and LoC are measured — the harness reads them and reports them, and a different reviewer running the same harness on the same day should get the same numbers within noise. Observability, eval surface, lock-in, and failure-recovery are scored 0–3 by rubric — the harness encodes a defensible heuristic in roughly twenty lines of Python per dimension.

The rubrics are short on purpose: anyone can read agents/_eval/scorecard.py, see the rules, and disagree (or replace them). What the rubrics are not: arbitrary editorial assertions. The lesson explicitly invites the reader to challenge a rubric by changing the code — that’s the whole point of having the harness under version control rather than as a static PDF chart. If you think LangGraph deserves a 3 on failure-recovery instead of a 2, edit the rubric, re-run, and ship a PR. The scorecard improves with use.

How the harness works

agents/_eval/run_all.py is the entry point. Its job, in six steps:

1. Import the four sweet-spot agents’ run_* functions — one per framework, from lesson04-tool-heavy-agent/agent.py, lesson06-graph-shaped-agent/agent.py, lesson08-handoff-agent/agent.py, and lesson10-multi-agent-crew/agent.py.
2. Define a small question set (3 questions in the default config; configurable at the top of the file).
3. For each (agent, question) pair: run the agent under a timer, capture the returned dict’s cost_usd and final_text, repeat 3 times for a median latency.
4. Run cloc (or the Python fallback) on each agent’s agent.py to get LoC.
5. Score the four rubric dimensions per agent based on hardcoded knowledge of each framework’s primitives — the rubric scores don’t depend on the live run, they depend on the framework.
6. Build a ScorecardRow per framework. Render the rows as JSON (results.json) and Markdown (results.md).

The Markdown output is what this lesson includes as a snapshot. The JSON is what downstream tooling (e.g., a CI dashboard) would consume.

Reading the 2026 Q2 snapshot

Slide 9 shows the populated table; the same table renders into agents/_eval/results.md when you run the harness. Three patterns are worth calling out before you skim the cells:

• The cheapest framework varies by task shape. The OpenAI Agents SDK comes out cheapest on the L08 handoff sweet spot because handoffs use fewer tools per turn and the triage agent often resolves the question in one or two turns. The Claude Agent SDK or LangGraph could easily be cheaper on a tool-heavy or graph-shaped task — the scorecard reflects each framework at its sweet spot, not its universal cost profile. Don’t generalise the cost column past the row it lives in.
• Latency tracks LoC less than you’d think. CrewAI (L10) has the most LoC of the four, but the latency penalty is mostly role-play conversation overhead — agents handing off to other agents adds turns, not framework boilerplate. If you optimised the L10 agent to use fewer roles, you’d cut latency without cutting LoC much. The two columns measure different things.
• Observability and vendor lock-in are anti-correlated. LangGraph (lock-in score 2, observability 3) is more portable and more observable than the Claude SDK (lock-in 1, observability 2). That’s not a coincidence: frameworks that expect to span vendors invest in cross-cutting concerns like traces, because they can’t lean on a single vendor’s native tooling.

Decision matrix in prose

Tie the scorecard to actionable recommendations. The five short cases below cover >90% of new-project decisions:

• “I want the simplest path that works.” → Claude Agent SDK. Median across all dimensions; the SDK lives inside the anthropic package you’d install anyway. Rewrite later if a specific need surfaces. This is the right default when nothing else screams.
• “My workflow looks like a drawn graph.” → LangGraph. Pay the upfront cost of thinking in nodes; reap the observability and eval-surface scores. Worth it when you have hard ordering constraints or explicit branch/merge points the LLM shouldn’t be inventing on the fly.
• “I need to swap models per-call, or I’m doing triage / handoff.” → OpenAI Agents SDK. Model-routing is first-class; the Runner.handoff() primitive is what we exercised in L08. The lock-in score is misleading here — the SDK name says OpenAI, but the runtime supports multiple providers via the model-routing layer.
• “I want explicit roles (a Researcher, a Critic, a Writer) talking to each other.” → CrewAI. Highest LoC and cost of the four, but the only one that prescribes role-played coordination as a primitive rather than as a pattern you assemble yourself.
• “None of the above are obviously right.” → start with the Claude SDK; the scorecard’s “median” behaviour means you’re unlikely to lock yourself out of a future move. Migrate when a specific row of the scorecard becomes the thing that’s costing you.

What the lab gives you

By the end of the lab you’ll have:

• Run python agents/_eval/run_all.py once (~5 min, ~$0.30) and seen the live scorecard populate row by row in the terminal.
• Opened agents/_eval/results.md to confirm the table populated correctly, with all four framework rows present and all seven numeric columns filled.
• Modified the default question set in run_all.py — e.g., added a fourth question that’s deliberately tool-heavy — and re-run to see the cost and latency columns shift in response.
• Optionally swapped the Python LoC counter for cloc if installed, and observed the LoC numbers move slightly (different counting rules for blank lines and docstrings).

You’ll leave with a runnable, reproducible artefact — not a static editorial table. The homework asks you to (a) add a question to the set, (b) add a fifth scorecard dimension of your own with a documented rubric, and (c) write a two-paragraph recommendation for a hypothetical project based on your re-run numbers. The exercise closes the loop: you don’t just read the comparison, you own it.

References

• agents/SDK-VERSIONS.md — exact framework versions the snapshot was measured against. Pin these when you reproduce.
• agents/_eval/scorecard.py and agents/_eval/run_all.py — the harness source. ~250 lines combined; read both before modifying a rubric.
• AI Prompting Security course (ai-prompt-security/) — depth on the prompt-injection sub-dimension of failure-recovery.
• The four sweet-spot agents this scorecard compares: agents/lesson04-tool-heavy-agent/agent.py (Claude SDK), agents/lesson06-graph-shaped-agent/agent.py (LangGraph), agents/lesson08-handoff-agent/agent.py (OpenAI Agents SDK), agents/lesson10-multi-agent-crew/agent.py (CrewAI).

LAST-REVIEWED: 2026-05-25