Homework

Task 1: Model Routing Strategy for Customer Support

Goal: Design a multi-model routing strategy that balances cost and quality for a high-volume customer support system.

Scenario

You are the ML engineer at a SaaS company. Your customer support system handles 50,000 tickets per day. Based on historical data, tickets break down as follows:

Tier	Percentage	Volume/Day	Description
Simple	60%	30,000	FAQ-style questions, status checks, basic instructions
Medium	30%	15,000	Account-specific issues, troubleshooting, multi-step solutions
Complex	10%	5,000	Billing disputes, escalations, edge cases requiring reasoning

Instructions

1. Assign a model to each tier. Justify your choice based on the six selection criteria.
2. Calculate the monthly cost for your routing strategy. Assume average token usage per tier:
   • Simple: 150 input tokens, 50 output tokens
   • Medium: 400 input tokens, 200 output tokens
   • Complex: 800 input tokens, 500 output tokens
3. Calculate what the monthly cost would be if you used a single model (the one assigned to your Complex tier) for all 50,000 tickets. What is the savings from routing?
4. Describe how you would implement the router: what signal(s) would you use to classify ticket complexity?

Deliverable

A structured document (markdown, text, or notebook) with your model assignments, cost calculations, and router design.

Self-check

• ✓ You assigned a specific model (with version) to each tier.
• ✓ Your cost calculations account for both input and output tokens separately.
• ✓ You computed the single-model baseline and the routing savings percentage.
• ✓ Your router design describes concrete signals (not just "complexity").

Task 2: Benchmark Limitations Analysis

Goal: Develop critical thinking about benchmark scores by finding where they fail to predict real-world performance.

Instructions

1. Go to a public benchmark leaderboard. Good options:
   • LMSYS Chatbot Arena
   • Open LLM Leaderboard (Hugging Face)
   • Klu.ai LLM Leaderboard
2. Identify a model that ranks in the top 5 on at least one benchmark.
3. Design a practical task (something you might use in production) where you believe that model would underperform a lower-ranked model. For example:
   • A model that tops HumanEval but struggles with a complex multi-file refactoring task
   • A model with the highest MMLU score but poor instruction-following on structured output
   • A model with the best Chatbot Arena Elo but highest latency, making it impractical for real-time use
4. If possible, test your hypothesis by running the task on both models and comparing results.
5. Write a 200–300 word analysis explaining:
   • Which benchmark the model leads on and what that benchmark measures
   • What your practical task requires that the benchmark does not test
   • Why the benchmark score was misleading for your use case
   • What this implies about how benchmarks should be used in model selection

Self-check

• ✓ You identified a specific model and a specific benchmark where it excels.
• ✓ Your practical task is concrete and realistic (not a toy example).
• ✓ Your analysis explains the gap between benchmark performance and practical performance.
• ✓ You stated an actionable takeaway about how to use benchmarks.

Task 3: Model Recommendation Memo

Goal: Write a professional recommendation memo that a non-technical stakeholder could use to make a model selection decision.

Scenario

Choose one of the following business scenarios (or create your own):

• Scenario A: A healthcare startup needs to summarise patient visit notes for doctors. 10,000 notes/day, each 500–2000 words. HIPAA compliance required. Summaries must be accurate — errors could affect patient care.
• Scenario B: An e-commerce company wants to generate product descriptions from structured data (specs, features, category). 100,000 products to process initially, then 500 new products/day. Cost sensitivity is high.
• Scenario C: A law firm needs a research assistant that can answer questions about uploaded legal documents (20–100 pages each). 200 queries/day. Accuracy is paramount; the firm will accept higher cost for fewer errors.

Memo Structure

Your memo should be approximately one page and include:

1. Executive summary (2–3 sentences): What model do you recommend and why?
2. Requirements analysis: Map the business requirements to the six selection criteria.
3. Options evaluated: List the 3–4 models you considered with a one-line rationale for each.
4. Recommendation: Your chosen model with justification across all six criteria.
5. Cost projection: Monthly cost estimate at the stated volume.
6. Risks and mitigations: What could go wrong with your choice and how to address it.

Self-check

• ✓ Your memo is understandable by a non-technical reader.
• ✓ You considered at least 3 candidate models before recommending one.
• ✓ Your cost projection uses realistic token estimates and current pricing.
• ✓ You identified at least two risks with your recommendation.
• ✓ Your memo is approximately one page (400–600 words).