Homework

Task 1: Fine-Tune for a New Domain

Goal: Apply the full fine-tuning pipeline to a domain of your choice, reinforcing each step from the lab.

Instructions

Choose a domain different from the customer-support scenario used in the lab. Good options include:

• Technical documentation writer: The model generates API docs, function references, or README sections
• Product description generator: Given product attributes, the model writes marketing copy in a specific brand voice
• Code review assistant: The model provides structured code review feedback (issues, suggestions, praise)
• Email drafter: The model writes professional emails given a brief and tone directive

For your chosen domain:

1. Write a clear system prompt that defines the assistant's role, tone, and output format.
2. Create a JSONL file with at least 30 examples (more is better). Split 80/20 into training and validation sets.
3. Run your validation script from Exercise 2 and fix any issues.
4. Upload and fine-tune using the OpenAI API (or simulate with a dry-run if you want to avoid the cost).
5. Evaluate on 5+ held-out test prompts, comparing fine-tuned vs. base model.

Deliverable

Your JSONL files (train.jsonl, val.jsonl) and a short written summary (~200 words) covering:

• What domain you chose and why
• How you sourced or created the training examples
• Validation results (token stats, any issues found)
• Evaluation results: did fine-tuning improve performance? On which dimensions?

Self-check

• ✓ Your system prompt is clear and would make sense to someone unfamiliar with the domain.
• ✓ Your JSONL passes all validation checks (valid JSON, correct schema, no duplicates).
• ✓ All assistant responses in your training data follow the same style and format.
• ✓ Your evaluation compares fine-tuned vs. base model on the same prompts.

Task 2: Hyperparameter Experiment

Goal: Build intuition for how hyperparameters affect fine-tuning outcomes.

Instructions

Using either your lab dataset (TechFlow support) or your Task 1 dataset, run three fine-tuning jobs with different epoch counts:

Job	n_epochs	Other params
A	1	defaults
B	3	defaults
C	6	defaults

For each job, record:

1. Final training loss
2. Final validation loss (if you provided a validation file)
3. Training time
4. Evaluation scores on 5 held-out test prompts (use the same scorecard from Exercise 4)

Create a comparison table and answer these questions:

• At which epoch count does validation loss start increasing (overfitting)?
• Does the model with the lowest training loss also have the best evaluation scores?
• Is there a meaningful quality difference between 3 epochs and 6 epochs?

Deliverable

A comparison table (markdown or printed output) showing metrics for each job, plus a written paragraph (~150 words) explaining what you learned about epoch selection.

Self-check

• ✓ You used the exact same training data for all jobs.
• ✓ You recorded both training loss and validation loss for each job.
• ✓ Your evaluation used identical test prompts across all three models.
• ✓ You can explain in your own words what overfitting looks like in fine-tuning metrics.

Task 3: Cost Analysis — Fine-Tuning vs. Few-Shot

Goal: Make a data-driven decision about when fine-tuning is worth the investment compared to few-shot prompting.

Instructions

Consider a production workload with the following characteristics:

• Volume: 10,000 API calls per day
• Base prompt (few-shot approach): system message (100 tokens) + 5 few-shot examples (400 tokens total) + user query (50 tokens) = 550 input tokens per call
• Fine-tuned prompt: system message (100 tokens) + user query (50 tokens) = 150 input tokens per call (no few-shot examples needed)
• Expected output: 100 tokens per call for both approaches

Calculate and compare:

1. Monthly inference cost for few-shot approach on gpt-4o-mini
2. Monthly inference cost for fine-tuned gpt-4o-mini (remember: fine-tuned inference rates are higher)
3. One-time training cost for the fine-tuned model
4. Break-even point: after how many days does the fine-tuning investment pay for itself?

Use the following rates (approximate, check current pricing):

Model	Input (per 1M tokens)	Output (per 1M tokens)
gpt-4o-mini (base)	$0.15	$0.60
gpt-4o-mini (fine-tuned inference)	$0.30	$1.20
gpt-4o-mini (training)	$3.00 / 1M tokens

Deliverable

A written analysis (~200–300 words) that includes:

• A cost comparison table showing monthly costs for both approaches
• The break-even calculation with your work shown
• Your recommendation: should this team fine-tune or stick with few-shot?
• At least one scenario where your recommendation would change (e.g., different volume, different model, different prompt length)

Self-check

• ✓ Your calculations account for both input and output token costs.
• ✓ You used the correct (higher) rates for fine-tuned model inference.
• ✓ Your break-even calculation includes the one-time training cost.
• ✓ Your recommendation considers factors beyond pure cost (quality improvement, latency, maintenance).