Lesson 17 Open Source Models
Theme:

Homework

Three tasks exploring model comparison, quantization research, and use-case analysis for open vs closed models.

Complete all three lab exercises before starting the homework. These tasks extend the practical skills from the lab into deeper analysis and decision-making.

📥 Reference Solution

⬇ lab_answer.ipynb

Task 1 — Open vs Closed Model Comparison Matrix

Objective

Build a comprehensive comparison matrix evaluating open-source and closed models across multiple dimensions.

Instructions

  1. Select 3 open-source models (e.g., Llama 3.1 8B, Mistral 7B, Phi-3 Mini) and 2 closed models (Claude Sonnet, GPT-4o-mini)
  2. Design 5 test prompts covering: factual accuracy, code generation, instruction following, creative writing, and structured output (JSON)
  3. Run each prompt on every model (use Ollama for open models, APIs for closed models)
  4. Score each response on a 1–5 scale for: accuracy, completeness, formatting, and response time
  5. Create a comparison table and write a 200-word analysis of the patterns you observe

Deliverable

A notebook containing the comparison table, all model outputs, and your written analysis.

Self-check: At least 25 model responses collected (5 models x 5 prompts). Scores are justified. Analysis identifies clear patterns about which model types excel at which tasks.

Task 2 — Quantization Deep Dive

Objective

Research and document the practical impact of quantization on model quality, speed, and memory usage.

Instructions

  1. Pick one model (e.g., Llama 3.1 8B) and pull multiple quantization levels in Ollama:
    • ollama pull llama3.1:8b (default Q4_K_M)
    • Find the same model in Q2, Q5, and Q8 variants (check Ollama library or HuggingFace for GGUF variants)
  2. For each quantization level, record: file size, RAM usage during inference (use ollama ps), and tokens per second
  3. Run the same 3 prompts on each variant and compare output quality
  4. Research and write a 300-word explanation of how GGUF quantization works: what does Q4_K_M mean? What is the difference between Q4_0 and Q4_K_M? Why do K-quant methods preserve quality better?
  5. Create a chart or table showing the quality-size-speed trade-off

Deliverable

A notebook with quantization benchmarks, quality comparison, and your written explanation of GGUF quantization methods.

Self-check: At least 3 quantization levels compared. File sizes and speed recorded. Written explanation covers K-quant methods and the quality-size trade-off.

Task 3 — Use-Case Decision Framework

Objective

Build a decision framework that recommends open-source or closed models based on project requirements.

Instructions

  1. Define 5 fictional but realistic project scenarios:
    • A healthcare startup summarising patient notes (privacy-critical)
    • A consumer chatbot handling 1M messages/day (cost-sensitive, high volume)
    • A legal firm analysing contracts (accuracy-critical, moderate volume)
    • A mobile app doing on-device text completion (offline, edge deployment)
    • A research lab experimenting with novel architectures (customisation needed)
  2. For each scenario, evaluate: data privacy needs, volume and cost, required capability level, latency requirements, customisation needs, and team expertise
  3. Recommend a specific model (or combination) for each scenario with justification
  4. Create a flowchart or decision tree that generalises your reasoning into a reusable framework
  5. Write a brief cost estimate for each scenario (API costs vs hardware costs for self-hosting)

Deliverable

A notebook or document with all 5 scenario analyses, a decision framework, and cost estimates.

Self-check: All 5 scenarios analysed. Each recommendation includes specific model names and justification. Decision framework could be applied to a new scenario. Cost estimates are realistic.

Submission Checklist