Lab Guide — Open Source Models

Install Ollama, run a model locally, compare outputs with Claude, and explore HuggingFace model cards.

Prerequisites: A modern laptop with at least 8 GB RAM. An ANTHROPIC_API_KEY in your .env file for Exercise 2. No GPU required — Ollama handles quantization automatically for CPU/Apple Silicon.

📥 Lab Notebooks

⬇ lab.ipynb ⬇ lab_answer.ipynb

Exercise 1 — Install Ollama and Run a Local Model

EX 1

Get a model running on your machine

Install Ollama and pull a small open-source model to run locally.

Install Ollama from ollama.com/download (macOS, Linux, or Windows)
Open a terminal and verify the installation: ollama --version
Pull a small model: ollama pull llama3.2:3b (about 2 GB download)
Start an interactive chat: ollama run llama3.2:3b
Try several prompts: ask it to explain a concept, write code, and summarise text
Exit the chat with /bye
List your downloaded models: ollama list
Check model details: ollama show llama3.2:3b

# Install (macOS / Linux)
curl -fsSL https://ollama.com/install.sh | sh

# Pull a model
ollama pull llama3.2:3b

# Interactive chat
ollama run llama3.2:3b

# Check what's installed
ollama list

Tip: If you have 16 GB+ RAM, try ollama pull llama3.1:8b for better quality. On 8 GB machines, stick with the 3B variant.

Checkpoint: Ollama is installed, a model is pulled, and you can have an interactive conversation in the terminal.

Exercise 2 — Local Model vs Claude API

EX 2

Compare Ollama output with Claude on the same prompts

Use Ollama’s OpenAI-compatible API to send the same prompts to both a local model and Claude, then compare quality, speed, and cost.

Ensure Ollama is running (ollama serve — it starts automatically after install)
Set up two clients: one for Ollama (local) and one for Claude (API)
Define 3 test prompts:
- A factual question: “What are the three laws of thermodynamics?”
- A coding task: “Write a Python function to find the longest common subsequence of two strings.”
- A creative task: “Write a haiku about machine learning.”
Send each prompt to both models, measuring response time
Print the results side by side and note quality differences

import time
from openai import OpenAI
import anthropic

# Local model via Ollama
local_client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")

# Claude API
claude_client = anthropic.Anthropic()

prompts = [
    "What are the three laws of thermodynamics?",
    "Write a Python function to find the longest common subsequence of two strings.",
    "Write a haiku about machine learning.",
]

for prompt in prompts:
    # Local model
    start = time.time()
    local_resp = local_client.chat.completions.create(
        model="llama3.2:3b",
        messages=[{"role": "user", "content": prompt}],
        max_tokens=500,
    )
    local_time = time.time() - start

    # Claude
    start = time.time()
    claude_resp = claude_client.messages.create(
        model="claude-sonnet-4-20250514",
        messages=[{"role": "user", "content": prompt}],
        max_tokens=500,
    )
    claude_time = time.time() - start

    print(f"\n{'='*60}")
    print(f"PROMPT: {prompt[:50]}...")
    print(f"\nLocal ({local_time:.1f}s): {local_resp.choices[0].message.content[:200]}")
    print(f"\nClaude ({claude_time:.1f}s): {claude_resp.content[0].text[:200]}")

Checkpoint: Both models return responses for all 3 prompts. You observe differences in quality, speed, and detail.

Exercise 3 — Explore HuggingFace Model Cards

EX 3

Navigate HuggingFace Hub and understand model metadata

Learn to read model cards and understand the quantization variants available for popular models.

Go to huggingface.co/models and filter by “Text Generation”
Find meta-llama/Meta-Llama-3.1-8B-Instruct and read its model card:
- What training data was used?
- What is the licence?
- What benchmarks are reported?
- What are the intended use cases and limitations?
Search for GGUF quantized versions (search for “Llama-3.1-8B GGUF”). Compare file sizes across quantization levels (Q2_K, Q4_K_M, Q5_K_M, Q8_0)
Find the Open LLM Leaderboard and compare scores for:
- Llama 3.1 8B vs Mistral 7B vs Phi-3 Mini
- Note which model wins on which benchmark
Document your findings in the notebook: create a comparison table with model name, size, quantized sizes, licence, and benchmark highlights

# You can also explore programmatically with the huggingface_hub library
from huggingface_hub import HfApi
api = HfApi()

# Search for Llama 3 models
models = api.list_models(search="llama-3.1", sort="downloads", direction=-1, limit=10)
for m in models:
    print(f"{m.modelId:50s}  downloads: {m.downloads:>10,}")

Checkpoint: You have a comparison table documenting at least 3 models with their sizes, quantization options, licences, and benchmark scores.

Important: Some HuggingFace models require accepting a licence agreement before download. You will need to create a free HuggingFace account and accept the Llama licence on the model page before you can download Llama weights programmatically.