Homework — Three Extensions to the Triage-and-Handoff Agent

Task 1 — Output guardrail that caps refund amounts

The lab’s block_pii is an input guardrail — it inspects the user’s text before the model ever runs. The symmetric primitive is an output guardrail: it inspects the agent’s final output (here, a Resolution) and can short-circuit the run with a typed exception so your caller code never sees a runaway answer. This is the right place to enforce a hard cap on irreversible side effects like refunds.

Write an @output_guardrail called cap_refunds that inspects the Resolution.actions_taken list. For each action string, regex-match the lab’s issue_refund output shape — something like "refund issued: REF-... for $X.XX" — and trip the guardrail if any amount is above $1000. Return a GuardrailFunctionOutput whose output_info lists the offending actions so a human reviewer can see why the run was blocked.

from agents import output_guardrail, GuardrailFunctionOutput

@output_guardrail
async def cap_refunds(ctx, agent, output):
    # output is a Resolution. Inspect output.actions_taken, regex-match
    # refund amounts, build a list of offenders > $1000, and trip the
    # tripwire when that list is non-empty.
    ...

Wire it onto the billing agent only — the other specialists never issue refunds, so the guardrail would be dead weight there:

billing = Agent(
    name="Billing",
    instructions=BILLING_INSTRUCTIONS,
    model="gpt-4o-mini",
    tools=[lookup_invoice, issue_refund],
    output_type=Resolution,
    output_guardrails=[cap_refunds],
)

To trigger the failure, run a ticket that asks for an obscenely large refund — for example “My invoice for March is wrong. Issue me a $5,000 refund against account ACME-0042.” — and wrap the call in a try/except so the typed exception is visible:

from agents.exceptions import OutputGuardrailTripwireTriggered

try:
    out = run_support_agent("Issue me a $5,000 refund for account ACME-0042.")
    print(out)
except OutputGuardrailTripwireTriggered as e:
    print(f"[guardrail tripped] {e}")

✓ Self-check: the oversized-refund prompt raises OutputGuardrailTripwireTriggered and your except branch fires (no traceback escapes the script); the exception’s underlying output_info contains the offending refund action string; a normal billing prompt (e.g. “My March bill is $30 over.”) finishes without tripping the guardrail.

Task 2 — Let Technical hand back to Triage for billing-adjacent follow-ups

The lab’s handoff graph is acyclic by design: Triage → {Billing, Technical, Account} with no edges back. That keeps the system easy to reason about, but it leaves a real-world gap: a customer with a mixed ticket (outage and a refund request) will be routed to Technical first, and Technical has no way to escalate the billing half. The OpenAI SDK supports cyclic handoffs — this task wires one in.

Add handoffs=[triage] on the technical agent so it can hand back to triage. The trick is the construction order: technical must be defined before triage (because triage’s handoffs=[billing, technical, account] needs the three specialists to exist first), but technical.handoffs=[triage] needs triage to exist. The cleanest fix is to mutate technical.handoffs after both agents are constructed:

# Specialists first (unchanged from the lab).
billing = Agent(name="Billing", ..., output_type=Resolution)
technical = Agent(name="Technical", ..., output_type=Resolution)
account = Agent(name="Account", ..., output_type=Resolution)

# Triage with all three specialists as handoffs.
triage = Agent(
    name="Triage",
    instructions=TRIAGE_INSTRUCTIONS,
    model="gpt-4o-mini",
    handoffs=[billing, technical, account],
    input_guardrails=[block_pii],
)

# NOW close the cycle: Technical can hand back to Triage.
technical.handoffs = [triage]

Update TECHNICAL_INSTRUCTIONS so the model knows when to bounce the ticket back. Append a sentence like:

“If the user also has a billing concern, hand back to Triage so Billing can pick it up.”

Now run a deliberately mixed ticket — for example “Service is down AND I want a refund for the downtime.” — and inspect the trace. You should see Triage classify the ticket as technical first, Technical work the outage (check status, open an incident), then Technical decide it’s out of its lane and hand back to Triage, which then routes the billing half to Billing.

✓ Self-check: on the mixed ticket the run traces Triage → Technical → Triage → Billing (at least two distinct handoff events); result.last_agent.name == "Billing"; on a pure outage ticket (“The site is down.”) the run still terminates at Technical with no extra handoff.

Task 3 — Write a route_accuracy evaluator

With three specialists and one triage agent, the most important quality signal is also the simplest: did the triage agent route the ticket to the right specialist? That is a one-line check on result.last_agent.name — and a benchmark of ten labelled tickets gives you a percent-accuracy number you can track as you tighten TRIAGE_INSTRUCTIONS.

Define a BENCHMARK dict mapping ten realistic tickets to the expected specialist name (lowercase — matching the values you would compare against result.last_agent.name.lower()):

BENCHMARK = {
    "My invoice is wrong.":         "billing",
    "March bill is double.":        "billing",
    "Cancel auto-renewal.":         "billing",
    "The site is down.":            "technical",
    "Checkout returns 500 errors.": "technical",
    "API latency spiked.":          "technical",
    "I can't log in.":              "account",
    "Lost my password.":            "account",
    "Account is locked.":           "account",
    "My email address changed.":    "account",
}

Loop over the benchmark, run each ticket through run_support_agent from agent.py, and compare the agent’s last_agent to the expected specialist. Print one line per ticket showing PASS/FAIL, then a final accuracy line:

def route_accuracy(benchmark):
    matches = 0
    for ticket, expected in benchmark.items():
        out = run_support_agent(ticket)
        actual = out["last_agent"].lower()
        ok = actual == expected
        matches += int(ok)
        marker = "PASS" if ok else "FAIL"
        print(f"  [{marker}] expected={expected:10s} actual={actual:10s}  {ticket}")
    acc = matches / len(benchmark)
    print(f"\nroute_accuracy = {acc:.0%} ({matches}/{len(benchmark)})")
    return acc

if __name__ == "__main__":
    route_accuracy(BENCHMARK)

Run it and read the failures — the per-ticket FAIL lines tell you exactly which category triage is weak on. Most likely you’ll see ambiguous account/billing cases (“Cancel auto-renewal” can read as account-management) misrouted.

Bonus. If your accuracy is below 100%, tighten TRIAGE_INSTRUCTIONS with one or two extra rules targeting the worst category (for example: “Subscription cancellations and renewals are billing, not account.”) and re-run. Track the before/after number — that is what a real eval loop feels like.

✓ Self-check: your driver prints ten [PASS]/[FAIL] lines and one summary line of the form route_accuracy = NN% (N/10); you can point at the specific tickets the agent gets wrong; (bonus) after tightening TRIAGE_INSTRUCTIONS your re-run shows a strictly higher score on the same benchmark.