Lesson 01 — Python ETL Fundamentals

From a URL on the internet to a CSV on your disk in 50 lines.

Extract → Transform → Load.

Every data pipeline you'll ever write is some version of those three verbs. Today's pipeline is the simplest possible one — three Python functions and a __main__.

One hour of GitHub, every hour, since 2011.

• Hourly .json.gz files
• ~50 MB each; ~5 GB/day
• No auth required
• URL pattern: https://data.gharchive.org/YYYY-MM-DD-H.json.gz

One event looks like:

{
  "id": "12345",
  "type": "PushEvent",
  "actor": {"login": "octocat"},
  "repo":  {"name": "octocat/hello"},
  "created_at": "2024-01-01T15:00:00Z"
}

Count events by type for one hour.

• Download one hour of gharchive data
• Parse the gzipped JSON-lines
• Group by type
• Write a CSV

50 lines of Python. We'll keep coming back to this script for the rest of Module 1.

Extract — the download

import requests, gzip, json, pathlib

URL = "https://data.gharchive.org/{hour}.json.gz"

def fetch_hour(hour: str, out_dir: pathlib.Path) -> pathlib.Path:
    out_dir.mkdir(parents=True, exist_ok=True)
    dest = out_dir / f"{hour}.json.gz"
    if dest.exists():
        return dest                      # poor man's caching
    resp = requests.get(URL.format(hour=hour), timeout=30)
    resp.raise_for_status()
    dest.write_bytes(resp.content)
    return dest

Transform — parse + count

from collections import Counter

def count_by_type(path: pathlib.Path) -> Counter:
    counts = Counter()
    with gzip.open(path, "rt") as f:
        for line in f:
            ev = json.loads(line)
            counts[ev["type"]] += 1
    return counts

Use Counter, not dict. Use a streaming reader (for line in f) not f.read() — gharchive hours can spike to 200 MB.

Load — write a CSV

import csv

def write_csv(counts: Counter, path: pathlib.Path) -> None:
    with path.open("w", newline="") as f:
        w = csv.writer(f)
        w.writerow(["type", "count"])
        for evtype, n in counts.most_common():
            w.writerow([evtype, n])

Wire it together

if __name__ == "__main__":
    import sys
    hour = sys.argv[1]                    # e.g. "2024-01-01-15"
    raw = fetch_hour(hour, pathlib.Path("data/raw"))
    counts = count_by_type(raw)
    write_csv(counts, pathlib.Path(f"data/out/{hour}.csv"))
    print(f"wrote {sum(counts.values())} events across {len(counts)} types")

What you'd see on a real hour

The 6 things our 50 lines did right

• Explicit input/output paths
• Cached the download
• Streamed instead of slurping
• Raised on HTTP errors
• Used Counter (constant-memory grouping)
• Had a __main__ guard

Each of those is a lesson on its own — we'll deepen them across Module 1.

What our 50 lines did wrong (and we'll fix)

Where it lives in the repo

data-engineering/sample-pipeline/
  extract/gharchive.py        # fetch_hour
  transform/events.py         # count_by_type, write_csv
  requirements.txt

Today you'll fill the NotImplementedError stubs that Plan A scaffolded.

A word on "ETL vs ELT vs streaming"

We do batch ETL today. The modern lakehouse prefers ELT (load first, transform on the warehouse) — that pattern is covered in Module 3 when we introduce dbt and a cloud warehouse.

Streaming (Kafka, Flink) is a sibling discipline and intentionally not in this course. It requires a different runtime model; treating it as "just faster batch" is one of the classic mistakes in data engineering. Our gharchive pipeline is batch by design — the data is already hourly.

Recap & what's next

• ETL = Extract, Transform, Load — the three verbs behind every data pipeline.
• gharchive.org gives us real, public, big-enough data to build against.
• 50 lines is enough to learn idempotency, streaming reads, and explicit paths — and to see exactly what's missing.