Homework

Task 1: Decision Framework — Your Own Use Case

Goal: Apply the 5-question decision framework to a real task from your work or a personal project.

Instructions

Identify a task where you currently use (or would like to use) an LLM. It could be anything: generating reports, classifying tickets, writing documentation, answering user questions, producing structured data, etc.

Walk through the decision framework, answering each question explicitly:

1. Does the base model understand the task? Test with a clear prompt. What happens?
2. Is the output format/style acceptable? Where does it deviate from what you need?
3. Does the model need external knowledge? Does the information exist in the model's training data, or does it need your documents?
4. Do you have 500+ labelled examples? If not, how would you create them? What would each example look like?
5. Is this a high-volume production task? Estimate calls per day. Would fine-tuning pay for itself?

Deliverable

A written analysis (300–500 words) structured around the 5 questions, ending with a clear recommendation: prompt engineering, RAG, fine-tuning, or a combination. Include your reasoning at each step.

Self-Check

• ✓ You answered all 5 framework questions with specific details about your use case.
• ✓ Your recommendation follows logically from your answers.
• ✓ You considered at least one alternative approach and explained why you ruled it out.
• ✓ You included a rough cost or volume estimate to support your recommendation.

Task 2: Fine-Tuning Proposal

Goal: Practice the planning process for a fine-tuning project, even if you ultimately decide not to fine-tune.

Instructions

Choose one of the following scenarios (or use your own from Task 1 if fine-tuning was your recommendation):

• Scenario X: A SaaS company wants all AI-generated help articles to match their documentation style: concise, second-person ("you"), always includes a code example, and ends with a "Next steps" section.
• Scenario Y: A financial services firm needs to generate risk assessment summaries that follow a strict 4-section template (Overview, Risk Factors, Mitigations, Rating) using industry-standard terminology.

Write a fine-tuning proposal that includes:

1. Task description: What the model needs to do, what "success" looks like.
2. Data plan: How many examples you need, how you would collect them, what each example looks like (write 2 sample input/output pairs).
3. Approach selection: Which fine-tuning method (API-based, LoRA, QLoRA, full) and why.
4. Cost estimate: Training cost, per-call inference cost, and break-even point at the expected volume.
5. Risks: What could go wrong and how you would mitigate it.

Deliverable

A structured document (500–800 words) covering all 5 sections above. Include the 2 sample input/output pairs.

Self-Check

• ✓ Your proposal has all 5 sections.
• ✓ Your sample input/output pairs are realistic and clearly demonstrate the desired behaviour.
• ✓ Your cost estimate uses specific numbers (not "it would be cheap").
• ✓ You identified at least 2 risks with concrete mitigation strategies.
• ✓ A teammate could read your proposal and implement the project without further guidance.

Task 3: Provider Research

Goal: Familiarise yourself with one API-based fine-tuning provider so you can make informed choices in later lessons.

Instructions

Choose one of the following providers:

• OpenAI Fine-Tuning
• Together AI
• Fireworks AI
• Mistral AI

Research and summarise the following:

1. Data format: What format does the training data need to be in? (JSONL, CSV, etc.) Show a 2-line example of the expected format.
2. Supported models: Which base models can you fine-tune?
3. Pricing: What does training cost? What does inference on the fine-tuned model cost? Is there a storage fee?
4. Limitations: Maximum dataset size, maximum epochs, any restrictions on use cases?
5. Unique features: Anything this provider offers that others do not (e.g., evaluation tools, hosted inference, model merging).

Deliverable

A summary (200–400 words) covering the 5 points above. Include the 2-line data format example as a code block.

Self-Check

• ✓ You covered all 5 research areas.
• ✓ Your data format example is valid (you could paste it into the provider's API).
• ✓ Pricing information includes both training and inference costs.
• ✓ You identified at least one unique feature or limitation that would affect your provider choice.