Homework

Task 1: Map a Model's Training Pipeline

Goal: Research and document the full training pipeline for a published model to reinforce your understanding of the pre-train → SFT → alignment stages.

Instructions

Choose one of the following models (or another model with a published technical report):

• Llama 3 — Meta technical report
• Mistral 7B — Mistral AI paper
• Claude — Anthropic research page (Constitutional AI paper + model card)

For your chosen model, create a structured document covering:

1. Pre-training: What data was used? How many tokens? What was the model architecture (parameter count, layers, attention heads)? What hardware was used? How long did training take?
2. SFT: What kind of demonstration data was used? How many examples? Was any specific chat template adopted?
3. Alignment: Which alignment method was used (RLHF, DPO, Constitutional AI, or a combination)? What preference data was collected? Were there any novel techniques?
4. Key decisions: What trade-offs did the team make? What is notable or unusual about their approach compared to the "standard" pipeline?

Deliverable

A 300–500 word summary structured under the four headings above. Include specific numbers where available (parameter count, token count, training compute).

Self-check

• ✓ You cited a specific technical report or paper.
• ✓ All three training stages are covered with specific details (not just "they did SFT").
• ✓ You identified at least one notable decision or trade-off the team made.
• ✓ Numbers are included for at least parameter count, training tokens, and dataset size for SFT/alignment.

Task 2: Create Preference Pairs

Goal: Build intuition for what makes good preference data by writing your own pairs.

Instructions

Choose a domain you know well — it could be your professional field, a hobby, or a technical area. Write 10 preference pairs: for each, provide a prompt, a "chosen" (better) response, and a "rejected" (worse) response.

Your pairs should cover different reasons for preference:

• At least 3 pairs where the chosen response is more helpful (more complete, better structured, more actionable)
• At least 2 pairs where the chosen response is more honest (acknowledges uncertainty, corrects misconceptions, avoids overconfidence)
• At least 2 pairs where the chosen response is more harmless (avoids bad advice, includes safety caveats, declines inappropriate requests)
• At least 1 pair where the preference is ambiguous — annotate why reasonable people might disagree

Format

Use this format for each pair:

--- Pair 4 / Category: More Honest ---
Prompt: "Is it safe to take ibuprofen every day?"

Chosen: "Taking ibuprofen daily is generally not recommended without
medical supervision. While it can be effective for short-term pain
relief, regular use can increase the risk of gastrointestinal issues,
kidney problems, and cardiovascular events. If you need daily pain
management, I'd recommend discussing this with your doctor who can
evaluate your specific situation and suggest safer long-term options."

Rejected: "Yes, ibuprofen is very safe. Millions of people take it
every day with no problems. The recommended dose is 200-400mg every
4-6 hours. Just don't exceed 1200mg per day and you'll be fine."

Why chosen is better: The chosen response acknowledges complexity and
recommends professional consultation. The rejected response is
overconfident and could lead to harm by normalising daily NSAID use
without medical oversight.

Deliverable

10 preference pairs in the format above, with category labels and "why chosen is better" annotations.

Self-check

• ✓ You have exactly 10 pairs with all three components (prompt, chosen, rejected).
• ✓ Each pair has a category label and a "why chosen is better" annotation.
• ✓ You covered all four categories (helpful, honest, harmless, ambiguous).
• ✓ Your "chosen" and "rejected" responses are both plausible — the rejected response is not obviously terrible.
• ✓ You were mindful of length bias: chosen responses are not always longer than rejected ones.

Task 3: Constitutional AI vs DPO Comparison

Goal: Articulate when you would choose each alignment approach and why.

Instructions

Write approximately 200 words comparing Constitutional AI (CAI) and Direct Preference Optimization (DPO). Your comparison should address:

1. When to use CAI: What kind of organisation, use case, or constraint makes CAI the better choice?
2. When to use DPO: What scenarios favour DPO over CAI or traditional RLHF?
3. Trade-offs: What does each approach sacrifice compared to the other?

Consider factors such as:

• Available compute and infrastructure
• Access to human annotators
• Need for explicit, auditable safety criteria
• Model size and training budget
• Open-source vs proprietary deployment

Deliverable

A 200-word (±20) comparison that a technical colleague could use to make an informed decision between the two approaches.

Self-check

• ✓ Your answer is 180–220 words.
• ✓ You identified at least one scenario where CAI is clearly preferable.
• ✓ You identified at least one scenario where DPO is clearly preferable.
• ✓ You mentioned at least two concrete trade-offs (not just "one is better").
• ✓ Your comparison is specific enough to be actionable, not generic.