Lab Guide

Format: Python script (lab.py)  ·  Time estimate: 45–60 minutes  ·  Prerequisites: pandas, datasets, tiktoken, anthropic, scikit-learn

Overview

In this lab you will build a complete dataset preparation pipeline for fine-tuning. You will collect data using three strategies, clean it, convert it to the instruction-response format, validate it, and split it into train/validation sets. The domain is Python programming Q&A — a concrete and testable domain that does not require specialised knowledge.

Setup

# Install dependencies
pip install pandas datasets tiktoken anthropic scikit-learn python-dotenv

# Ensure your .env has ANTHROPIC_API_KEY set
# The lab uses Claude for synthetic data generation

Step 1: Create Seed Examples (Manual Collection)

Goal

Write 10 high-quality instruction-response pairs by hand. These serve as your gold-standard seed set and as few-shot examples for synthetic generation.

Steps

1. Define a style guide for your dataset. For this lab, use:
   • Instruction: A clear Python programming question (10–50 words)
   • Input: Optional code snippet or context (can be empty)
   • Output: A concise, correct answer with a code example (50–200 words)
2. Write 10 examples covering different Python topics: data structures, string manipulation, file I/O, error handling, list comprehensions, classes, decorators, etc.
3. Store them in a list of dictionaries with keys instruction, input, output, and source (set to "manual").

Expected Output

seed_examples = [
    {
        "instruction": "How do I reverse a list in Python without modifying the original?",
        "input": "",
        "output": "Use slicing with `[::-1]` to create a reversed copy:\n\n```python\noriginal = [1, 2, 3, 4, 5]\nreversed_copy = original[::-1]\nprint(reversed_copy)  # [5, 4, 3, 2, 1]\nprint(original)       # [1, 2, 3, 4, 5] (unchanged)\n```\n\nAlternatively, use `list(reversed(original))` which returns a new list from the reversed iterator.",
        "source": "manual"
    },
    # ... 9 more examples
]
print(f"Seed examples: {len(seed_examples)}")

Step 2: Generate Synthetic Data

Goal

Use Claude to generate 40 additional examples, using your seed set as few-shot context.

Steps

1. Build a prompt that includes 3 of your seed examples as few-shot demonstrations, plus a schema definition and instructions for generating new examples.
2. Call the Claude API to generate 10 examples at a time (4 calls total).
3. Parse the JSON responses and add "source": "synthetic" to each.
4. Combine with your seed examples into a single list.

Prompt Template

GENERATION_PROMPT = """Generate {n} Python programming Q&A examples.
Each example must have these fields:
- instruction: A clear Python question (10-50 words)
- input: Optional code context (empty string if not needed)
- output: Concise answer with a code example (50-200 words)

Here are reference examples showing the expected format and quality:

{few_shot_examples}

Requirements:
- Cover diverse Python topics (do not repeat topics from the examples above)
- Answers must be technically correct
- Include runnable code snippets in the output
- Vary the question style (how-to, what-is, debug-this, compare-these)

Return ONLY a JSON array of {n} objects. No other text."""

Step 3: Convert Existing Documentation

Goal

Convert a small set of Python documentation excerpts into instruction-response pairs.

Steps

1. Define 5–10 documentation snippets as strings (e.g., from Python docs on enumerate, zip, with statement, dataclasses, etc.).
2. Write a function that converts each snippet into an instruction-response pair:
   • Extract the topic from the snippet
   • Generate a natural instruction like "Explain how to use X in Python"
   • Use the documentation content as the basis for the output
3. Tag these with "source": "docs".
4. Add them to your combined dataset.

def convert_doc_to_example(topic: str, doc_text: str) -> dict:
    """Convert a documentation snippet to an instruction-response pair."""
    # TODO: Implement this function
    # - Create a natural question about the topic
    # - Format the doc_text as a clear response
    # - Return a dict with instruction, input, output, source keys
    pass

Step 4: Clean the Dataset

Goal

Apply deduplication, length filtering, and quality scoring to your combined dataset.

Steps

1. Load all examples into a pandas DataFrame.
2. Add token counts: Use tiktoken to count tokens for each instruction and output field.
3. Deduplicate: Remove exact duplicates based on instruction + output hash. Then use fuzzy matching (Jaccard similarity on word sets) to flag near-duplicates.

   import hashlib
   
   def compute_hash(row):
       text = row["instruction"] + row["output"]
       return hashlib.md5(text.encode()).hexdigest()
   
   df["hash"] = df.apply(compute_hash, axis=1)
   before = len(df)
   df = df.drop_duplicates(subset=["hash"])
   print(f"Dedup: {before} -> {len(df)} ({before - len(df)} removed)")

4. Length filter: Keep examples where instruction tokens are between 5–512 and output tokens are between 20–1024.

   df = df[df["instruction_tokens"].between(5, 512)]
   df = df[df["output_tokens"].between(20, 1024)]
   print(f"After length filter: {len(df)} examples")

5. Quality score (optional): If you have API access, send a sample of examples to Claude for scoring on a 1–5 scale. Otherwise, manually review 10 examples and remove any that are clearly wrong or incomplete.

Step 5: Validate the Dataset

Goal

Run automated checks to catch structural problems before training.

Steps

1. Write a validation function that checks each example for:
   • All required fields are present and non-empty (instruction, output)
   • No field exceeds maximum token length
   • Output contains at least one code block (for this Python Q&A domain)
   • No obvious placeholder text ("TODO", "INSERT HERE", etc.)
2. Run the validator and print a summary of passing/failing examples.
3. Fix or remove failing examples.

def validate_example(example: dict) -> list[str]:
    """Return a list of validation errors (empty = valid)."""
    errors = []
    # TODO: Check required fields
    # TODO: Check token lengths
    # TODO: Check for code blocks in output
    # TODO: Check for placeholder text
    return errors

# Run validation
for i, row in df.iterrows():
    errs = validate_example(row.to_dict())
    if errs:
        print(f"Example {i}: {errs}")

Step 6: Split into Train/Validation

Goal

Create train and validation sets with proper leakage prevention.

Steps

1. Assign a group_id to each example. For this lab, examples from the same source and topic get the same group ID. (In practice, all augmented variants of one example share a group.)
2. Use scikit-learn's GroupShuffleSplit to split on groups, not individual examples.
3. Verify no group appears in both splits.
4. Save the splits as JSONL files.

from sklearn.model_selection import GroupShuffleSplit
import json

splitter = GroupShuffleSplit(n_splits=1, test_size=0.2, random_state=42)
train_idx, val_idx = next(splitter.split(df, groups=df["group_id"]))

train_df = df.iloc[train_idx]
val_df = df.iloc[val_idx]

print(f"Train: {len(train_df)} examples")
print(f"Val:   {len(val_df)} examples")

# Verify no group leakage
train_groups = set(train_df["group_id"])
val_groups = set(val_df["group_id"])
assert train_groups.isdisjoint(val_groups), "Data leakage detected!"

# Save as JSONL
def save_jsonl(df, path):
    with open(path, "w") as f:
        for _, row in df.iterrows():
            example = {
                "instruction": row["instruction"],
                "input": row["input"],
                "output": row["output"]
            }
            f.write(json.dumps(example) + "\n")

save_jsonl(train_df, "train.jsonl")
save_jsonl(val_df, "val.jsonl")
print("Saved train.jsonl and val.jsonl")

Finishing Up

Once you have completed all six steps:

1. Run the full script end-to-end: python lab.py
2. Verify that train.jsonl and val.jsonl are written and contain valid JSONL.
3. Compare your implementation with lab_answer.py — focus on the cleaning and validation logic.

Self-Check

• ✓ Your seed set has 10 diverse, high-quality manual examples.
• ✓ Synthetic generation produced additional examples with proper source tagging.
• ✓ Deduplication removed at least some examples (if not, check your dedup logic).
• ✓ Length filtering removed examples outside the defined thresholds.
• ✓ Validation catches structural problems (missing fields, no code blocks).
• ✓ Train/validation split has zero group overlap (the assert passes).
• ✓ Both JSONL files are valid and parseable.
• ✓ You can explain why splitting before augmentation prevents data leakage.