Solutions — Homework 01

Lesson 01 · Data Engineering with Python

Solution to Task 1 — 24 hours combined

The key insight is that Counter supports .update(), which merges two counters by summing counts for matching keys. This means the day-total loop is three lines:

from collections import Counter
import pathlib
from extract.gharchive import fetch_hour
from transform.events import count_by_type, write_csv

day = "2024-01-01"
combined = Counter()
for h in range(24):
    hour = f"{day}-{h}"
    raw = fetch_hour(hour, pathlib.Path("data/raw"))
    combined.update(count_by_type(raw))

write_csv(combined, pathlib.Path(f"data/out/{day}-combined.csv"))
print(f"day total: {sum(combined.values()):,} events across {len(combined)} types")

Why Counter.update() and not manual addition? You could write for evtype, n in hourly.items(): combined[evtype] += n and get the correct answer. But Counter.update(another_counter) does the same thing in one call, is more readable, and preserves the most_common() and elements() convenience methods on the merged result. It also handles zero-initialisation — if an event type appears in hour 15 but not in hour 0, Counter.update starts that key at zero automatically.

Common mistake: building a dict[str, int] and manually incrementing with += 1 in the inner loop rather than delegating to Counter. This is technically correct but loses the streaming and most_common convenience, and becomes painful when you add Task 2's actor counting on top.

Solution to Task 2 — Top-N actors

Add a top_actors function to transform/events.py:

def top_actors(path: pathlib.Path, n: int = 10) -> list:
    counts = Counter()
    with gzip.open(path, "rt") as f:
        for line in f:
            ev = json.loads(line)
            counts[ev["actor"]["login"]] += 1
    return counts.most_common(n)

Update the main() function to accept the --top-actors flag:

def main() -> None:
    import argparse
    parser = argparse.ArgumentParser(description="Transform gharchive events")
    parser.add_argument("--input", required=True, help="Path to .json.gz input file")
    parser.add_argument("--output", default=None, help="Path to .csv output file")
    parser.add_argument("--top-actors", type=int, default=None,
                        help="Print top-N actor logins instead of writing CSV")
    args = parser.parse_args()

    if args.top_actors:
        for login, n in top_actors(pathlib.Path(args.input), args.top_actors):
            print(f"{n:>8}  {login}")
    else:
        if not args.output:
            raise SystemExit("--output is required unless --top-actors is set")
        counts = count_by_type(pathlib.Path(args.input))
        write_csv(counts, pathlib.Path(args.output))
        print(f"wrote {sum(counts.values())} events across {len(counts)} types")

Common mistake: assuming actor.login exists on every event. It does — for now. But gharchive events occasionally include null actors for deleted accounts, or a bot actor object where the login key has a different name. Lesson 03 will show you why "I checked once" is not the same as "I validated with an expectation suite". For now, you can add a defensive ev.get("actor", {}).get("login", "unknown") if you're seeing KeyErrors on unusual hours.

Solution to Task 3 — Predict the Lesson 02 win

"Disk footprint shrinks roughly 5–10×: Parquet's columnar layout lets Snappy or Zstd compress runs of the same type very tightly, and partitioning by type means the partition column is not stored in the row data at all. The PushEvent count query becomes nearly free: with partition pruning, the engine only reads the PushEvent=* directory's metadata — no row scan at all. The top-actors query is the case where columnar still helps but partitioning hurts: you have to read every partition (every event type) but only the actor column from each file. With predicate pushdown and column projection, that is still 10–100× faster than scanning a CSV that must materialise every column for every row."

Common mistake: assuming Parquet always wins on disk size. It does not — for tiny files (under about 1 MB), the metadata overhead can make Parquet larger than the equivalent CSV. Parquet wins at the scale where any of this matters: millions of rows, many columns, repeated values in categorical fields. Our one-hour gharchive file is right on the boundary; the full-day 24-hour file is solidly in "Parquet wins" territory.

Notice all three solutions reused the Lesson 01 functions verbatim — fetch_hour, count_by_type, and write_csv were imported, not copy-pasted. That is the payoff for keeping those functions as plain, importable functions instead of glueing everything into one script with no module boundary. When Lesson 02 replaces write_csv with write_parquet, the extract and transform functions do not change at all.