Lab Guide

Format: Python script (lab.py) · Time estimate: 45–60 minutes · Prerequisites: Python 3.10+, openai, tiktoken, python-dotenv installed, OpenAI API key with fine-tuning access

📥 Lab Files

⬇ lab.py (student) ⬇ lab_answer.py (solution)

Setup

Before starting, ensure your environment is ready:

# From the project root
pip install openai tiktoken python-dotenv

# Ensure your .env file contains your OpenAI API key
echo "OPENAI_API_KEY=sk-your_key_here" >> .env

Verify setup by running:

python -c "from openai import OpenAI; print('OpenAI SDK ready')"

Cost Warning

Fine-tuning incurs real charges on your OpenAI account. The lab dataset is small (~50 examples), so training cost should be under $1. However, always check your usage dashboard after running fine-tuning jobs.

Exercise 1: Prepare JSONL Training Data

Goal

Create a JSONL training dataset of 50+ examples in the chat-completions message format for a customer-support domain.

Steps

Define a system prompt that describes the assistant's role and style:

SYSTEM_PROMPT = (
    "You are a customer-support agent for TechFlow, a SaaS project management tool. "
    "Be helpful, concise, and professional. If you don't know the answer, say so honestly. "
    "Always end with asking if there's anything else you can help with."
)

Create a list of 50+ (user_question, assistant_answer) pairs covering common support topics:
- Account management (password reset, billing, plan upgrades)
- Feature usage (creating projects, inviting team members, setting deadlines)
- Troubleshooting (login issues, sync errors, notification problems)
- Edge cases (unknown questions, inappropriate requests)

Write each example as a JSONL line with the messages array format:

import json

def create_example(user_msg, assistant_msg, system_msg=SYSTEM_PROMPT):
    """Create a single JSONL training example."""
    return json.dumps({
        "messages": [
            {"role": "system", "content": system_msg},
            {"role": "user", "content": user_msg},
            {"role": "assistant", "content": assistant_msg}
        ]
    })

Save all examples to training_data.jsonl, reserving 10 examples for validation_data.jsonl.

Expected Output

Created training_data.jsonl with 40 examples
Created validation_data.jsonl with 10 examples
Sample example:
  System: You are a customer-support agent for TechFlow...
  User: How do I reset my password?
  Assistant: To reset your password, go to app.techflow.io/reset and enter...

Tips for Quality Data

Write assistant responses the way you want the model to respond — these are your gold standard.
Keep the system prompt identical across all examples.
Include a few examples where the model should say "I don't know" or redirect to human support.
Vary the phrasing of similar questions (e.g., "reset password", "forgot my password", "can't log in").

Exercise 2: Validate the Dataset

Goal

Build and run automated validation checks on your JSONL file to catch errors before uploading.

Steps

Write a validate_jsonl(filepath) function that checks each line for:
- Valid JSON parsing
- Presence of messages key
- Each message has role and content keys
- Roles are one of: "system", "user", "assistant"
- Last message has role "assistant"
- No empty content strings

Add token counting using tiktoken:

import tiktoken

def count_tokens(messages, model="gpt-4o-mini"):
    """Count tokens for a list of messages."""
    enc = tiktoken.encoding_for_model(model)
    total = 0
    for msg in messages:
        # Every message has overhead: role tokens + content tokens + separators
        total += 4  # message overhead
        total += len(enc.encode(msg["content"]))
    total += 2  # assistant reply priming
    return total

Check for duplicate examples (exact match on user + assistant content).
Print a summary report: total examples, token statistics (min/max/mean), any errors found.

Expected Output

=== Validation Report ===
File: training_data.jsonl
Total examples: 40
Valid examples: 40
Errors: 0

Token statistics:
  Min tokens per example: 87
  Max tokens per example: 342
  Mean tokens per example: 156.4
  Total training tokens: 6,256

Duplicates: 0
Estimated training cost (3 epochs): $0.056

All checks passed!

Exercise 3: Create a Fine-Tuning Job

Goal

Upload your validated data to OpenAI and start a fine-tuning job.

Steps

Upload both training and validation files:

from openai import OpenAI
client = OpenAI()

# Upload training file
train_file = client.files.create(
    file=open("training_data.jsonl", "rb"),
    purpose="fine-tune"
)
print(f"Training file ID: {train_file.id}")

# Upload validation file
val_file = client.files.create(
    file=open("validation_data.jsonl", "rb"),
    purpose="fine-tune"
)
print(f"Validation file ID: {val_file.id}")

Create the fine-tuning job with explicit hyperparameters:

job = client.fine_tuning.jobs.create(
    training_file=train_file.id,
    validation_file=val_file.id,
    model="gpt-4o-mini-2024-07-18",
    hyperparameters={
        "n_epochs": 3
    },
    suffix="techflow-support-v1"
)
print(f"Job ID: {job.id}")
print(f"Status: {job.status}")

Poll for job completion:

import time

while True:
    job = client.fine_tuning.jobs.retrieve(job.id)
    print(f"Status: {job.status}")
    if job.status in ("succeeded", "failed", "cancelled"):
        break
    time.sleep(30)

if job.status == "succeeded":
    print(f"Fine-tuned model: {job.fine_tuned_model}")
else:
    print(f"Job failed: {job.error}")

List recent training events to inspect the training progress:

events = client.fine_tuning.jobs.list_events(job.id, limit=10)
for event in events.data:
    print(f"[{event.created_at}] {event.message}")

Expected Output

Training file ID: file-abc123...
Validation file ID: file-def456...
Job ID: ftjob-xyz789...
Status: validating_files
Status: running
Status: running
Status: succeeded
Fine-tuned model: ft:gpt-4o-mini-2024-07-18:personal:techflow-support-v1:ABC123

Training Time

Fine-tuning typically takes 5–30 minutes for small datasets. Larger datasets or more epochs take longer. The job runs asynchronously — you can close your terminal and check back later.

Exercise 4: Evaluate the Fine-Tuned Model

Goal

Compare the fine-tuned model against the base model on held-out test prompts.

Steps

Define 5–10 test prompts that were not in your training data:

test_prompts = [
    "Can I export my project data to Excel?",
    "How many users can I add on the free plan?",
    "The Gantt chart isn't loading properly.",
    "Can you help me write a poem?",          # out-of-domain
    "I want to delete my account permanently.",
]

Run each prompt through both the base model and the fine-tuned model:

def get_response(model, user_msg, system_msg=SYSTEM_PROMPT):
    response = client.chat.completions.create(
        model=model,
        messages=[
            {"role": "system", "content": system_msg},
            {"role": "user", "content": user_msg}
        ],
        temperature=0,
        max_tokens=300
    )
    return response.choices[0].message.content

for prompt in test_prompts:
    base_resp = get_response("gpt-4o-mini", prompt)
    ft_resp = get_response(job.fine_tuned_model, prompt)
    print(f"\nUser: {prompt}")
    print(f"Base model: {base_resp[:200]}...")
    print(f"Fine-tuned:  {ft_resp[:200]}...")

Create a simple evaluation scorecard:

criteria = [
    "Follows the TechFlow brand voice",
    "Ends with 'anything else' question",
    "Stays concise (under 100 words)",
    "Handles out-of-domain gracefully",
]
# Score each response 0-2 for each criterion
# (0 = fails, 1 = partial, 2 = fully meets)

Summarise results: which model scored higher overall? On which criteria did fine-tuning help most?

Expected Output

=== Evaluation Summary ===
                          Base Model    Fine-Tuned
Brand voice               1.2 / 2.0    1.8 / 2.0
Closing question          0.4 / 2.0    1.9 / 2.0
Conciseness               1.6 / 2.0    1.7 / 2.0
Out-of-domain handling    1.0 / 2.0    1.6 / 2.0
---
Overall average           1.05 / 2.0   1.75 / 2.0

Fine-tuning improved scores by 67% on average.

Finishing Up

Once you have completed all four exercises:

Run the full script end-to-end: python lab.py
Review your evaluation scorecard — does fine-tuning provide meaningful improvement for your use case?
Compare your implementation with lab_answer.py.
Optionally, delete the fine-tuned model if you do not plan to use it further:
```
client.models.delete(job.fine_tuned_model)
```

Self-Check

✓ Exercise 1: Your JSONL file has 50+ examples with consistent system prompts.
✓ Exercise 2: Validation passes with zero errors and you can explain the token statistics.
✓ Exercise 3: Your fine-tuning job completed successfully and you have a model ID.
✓ Exercise 4: You compared both models on at least 5 test prompts and can articulate where fine-tuning helped.