Homework

Task 1: Build a Domain-Specific Dataset (200+ Examples)

Goal: Apply all three collection strategies to build a real dataset for a domain you care about.

Instructions

Choose a domain that is different from the Python Q&A domain used in the lab. Good options include:

• Customer support for a product or service you know well
• Technical documentation for a framework or tool
• Medical/legal/financial FAQ (use public information only)
• Code review feedback for a specific language or framework

Build your dataset using all three strategies:

1. Manual creation (30+ examples): Write a style guide first. Then create at least 30 seed examples that define the quality bar. These should cover the core topics and edge cases in your domain.
2. Synthetic generation (120+ examples): Use your seed examples as few-shot context to generate at least 120 additional examples with Claude or another strong model. Vary the few-shot examples across generation batches to increase diversity.
3. Document conversion (50+ examples): Find existing documentation, FAQs, or knowledge base articles in your chosen domain. Convert them into instruction-response pairs. Cite your sources.

Run the full cleaning pipeline from the lab (dedup, length filter, validation) on your combined dataset. Save the final result as homework_dataset.jsonl.

Deliverable

A JSONL file with 200+ cleaned examples and a brief README (5–10 lines) describing your domain, style guide decisions, and how many examples came from each source.

Self-check

• ✓ At least 200 examples after cleaning.
• ✓ All three collection strategies are represented.
• ✓ Each example has instruction, input, output, and source fields.
• ✓ No exact duplicates remain.
• ✓ Token lengths fall within your defined thresholds.
• ✓ You can articulate why you chose each style guide decision.

Task 2: Quality Audit with Inter-Rater Agreement

Goal: Develop a quality rubric and measure how consistently it can be applied.

Instructions

1. Define a scoring rubric with 3 dimensions, each scored 1–5:

   Dimension	1 (Poor)	3 (Acceptable)	5 (Excellent)
   Correctness	Factually wrong or would cause errors	Mostly correct with minor issues	Completely accurate and verified
   Completeness	Does not address the instruction	Addresses the core question	Thorough, covers edge cases
   Style	Wrong tone/format for the domain	Acceptable but inconsistent	Matches style guide perfectly

2. Randomly sample 50 examples from your Task 1 dataset.
3. Score all 50 examples yourself using the rubric.
4. Have a second person (colleague, friend, or use Claude as a proxy rater) independently score the same 50 examples.
5. Compute agreement metrics:

   import numpy as np
   
   def compute_agreement(rater1_scores, rater2_scores):
       """Compute exact agreement and within-1 agreement."""
       exact = np.mean(np.array(rater1_scores) == np.array(rater2_scores))
       within_1 = np.mean(np.abs(np.array(rater1_scores) - np.array(rater2_scores)) <= 1)
       return {"exact_agreement": exact, "within_1_agreement": within_1}

6. Identify examples where raters disagreed by 2+ points. Examine what caused the disagreement and refine your rubric.

Deliverable

A summary report (text or script output) showing: agreement percentages per dimension, the 5 most-disagreed examples with analysis, and your revised rubric if you made changes.

Self-check

• ✓ You scored exactly 50 examples on all 3 dimensions.
• ✓ You have two independent sets of scores.
• ✓ Within-1 agreement is above 70% (if not, your rubric needs clearer definitions).
• ✓ You analysed the highest-disagreement examples and can explain why they were ambiguous.

Task 3: Distribution Analysis and Visualisation

Goal: Identify potential biases and imbalances in your dataset before training.

Instructions

1. For your Task 1 dataset, compute and visualise the following distributions:
   • Topic distribution: Categorise each example by topic (manually or using a classifier). Plot a bar chart. Are any topics over- or under-represented?
   • Length distribution: Plot histograms of instruction token counts and output token counts. Are there unusual clusters or gaps?
   • Source distribution: What proportion came from manual, synthetic, and doc conversion? Plot a pie chart.
2. Write a function that flags potential problems:

   def audit_distribution(df):
       """Flag distribution issues in the dataset."""
       issues = []
       # Check if any topic has less than 5% of examples
       topic_counts = df["topic"].value_counts(normalize=True)
       rare_topics = topic_counts[topic_counts < 0.05]
       if len(rare_topics) > 0:
           issues.append(f"Under-represented topics: {list(rare_topics.index)}")
       # Check if any topic has more than 30% of examples
       dominant = topic_counts[topic_counts > 0.30]
       if len(dominant) > 0:
           issues.append(f"Over-represented topics: {list(dominant.index)}")
       # Check output length variance
       length_cv = df["output_tokens"].std() / df["output_tokens"].mean()
       if length_cv > 1.0:
           issues.append(f"High output length variance (CV={length_cv:.2f})")
       return issues

3. Based on your analysis, propose specific actions to improve balance (e.g., "add 20 more examples about topic X", "remove 15 near-duplicate examples about topic Y").

Deliverable

Three visualisations (saved as PNG or displayed inline) and a written analysis (200–300 words) of what the distributions reveal and what you would change before training.

Self-check

• ✓ You produced all three visualisations (topic, length, source).
• ✓ Your audit function runs and reports at least one finding.
• ✓ Your analysis is specific — it names actual topics and numbers, not just "some topics are underrepresented".
• ✓ You proposed at least 2 concrete actions to improve the dataset.