Curating high-quality training data for fine-tuning — the single biggest factor in whether your fine-tuned model succeeds or fails.
A fine-tuned model learns the patterns in your dataset — including the bad ones. 500 carefully crafted examples consistently outperform 10,000 noisy ones.
Supervised fine-tuning (SFT) datasets follow a consistent structure. The model learns to map instructions to responses.
{"instruction": "Summarise the following customer complaint in one sentence.",
"input": "I ordered a blue widget on March 3rd and received a red one...",
"output": "Customer received incorrect colour (red instead of blue) for widget ordered on March 3rd."}
{"instruction": "Classify the sentiment of this review.",
"input": "The battery life is amazing but the screen is too dim outdoors.",
"output": "mixed"}
{"messages": [
{"role": "system", "content": "You are a concise customer support agent."},
{"role": "user", "content": "My order #4521 hasn't arrived."},
{"role": "assistant", "content": "I'll look into order #4521 right away. Could you confirm your shipping address?"}
]}Choose the format that matches your target API. Anthropic fine-tuning uses the messages array; many open-source tools accept the instruction/input/output format.
Domain experts write instruction-response pairs by hand. Highest quality, but expensive and slow. Best for: seed datasets, edge cases, establishing gold-standard style.
Tip: Create a style guide first. Define tone, length, format, and common patterns before writing any examples.
Use a strong model (e.g., Claude Sonnet) to generate training data for a smaller model. Provide few-shot examples and a clear schema. Fast and scalable, but requires validation.
Tip: Always have a human review a random sample (10-20%) of synthetic data before training.
Transform existing assets — FAQs, docs, support tickets, wikis — into instruction-response pairs. Leverages existing knowledge, but needs reformatting.
Tip: Extract Q&A pairs from documentation headings. Each heading becomes an instruction; the section content becomes the response.
Raw collected data is never ready for training. Every dataset needs a cleaning pass — ideally automated with spot-checks.
Exact and near-duplicate examples cause the model to memorise rather than generalise. Use hash-based exact dedup and MinHash/LSH for fuzzy matching.
Remove examples that are too short (likely incomplete) or too long (exceed your training context). Set min/max token thresholds for both instructions and responses.
Use a strong model to score each example on a 1-5 scale for correctness, completeness, and style adherence. Filter out examples scoring below your threshold.
import pandas as pd
from datasets import Dataset
def clean_dataset(df: pd.DataFrame) -> pd.DataFrame:
# 1. Dedup on instruction+output hash
df = df.drop_duplicates(subset=["instruction", "output"])
# 2. Length filter (token counts)
df = df[df["output_tokens"].between(20, 1024)]
df = df[df["instruction_tokens"].between(5, 512)]
# 3. Quality filter
df = df[df["quality_score"] >= 3]
return df.reset_index(drop=True)| Split | Proportion | Purpose |
|---|---|---|
| Train | 80-90% | Model learns from these examples |
| Validation | 10-20% | Monitors for overfitting during training |
| Test (optional) | 5-10% | Final held-out evaluation |
If you generated variations of the same base example, all variations must go in the same split. Otherwise the model has effectively "seen" the validation data during training, and your metrics will be misleadingly optimistic.
Validation examples that overlap with training data give false confidence. Always split before augmentation. Check for near-duplicates across splits.
If 90% of your examples are about topic A and 10% about topic B, the model will over-index on A and underperform on B. Audit your topic distribution and balance deliberately.
If every response starts with "Sure! Here's..." the model will always start that way — even when inappropriate. Vary your response openings and structure.
If your training data includes examples from standard eval benchmarks, your model will appear better than it is. Screen training data against common benchmarks.