Homework 01 — Python ETL Fundamentals

Task 1 — Process 24 hours

Loop over all 24 hours of 2024-01-01, run the Lesson 01 pipeline for each (extract + transform), and write one combined CSV (type,count) that sums the event counts across the whole day. The output file should be named data/out/2024-01-01-combined.csv.

You must import the functions you wrote in the lab — fetch_hour, count_by_type, and write_csv — rather than copy-pasting the code. Write this as a standalone Python script or as an addition to transform/events.py.

✓ Self-check: the day's PushEvent total is in the millions. The total events across all types is in the tens of millions. The file has a type,count header and roughly 30 rows.

Task 2 — Add a CLI flag for the actor

Extend transform/events.py with a --top-actors N option. When this flag is passed, instead of counting by event type the script should print the top-N most active actor logins for that hour, sorted descending by event count.

Example usage:

python -m transform.events \
  --input data/raw/2024-01-01-15.json.gz \
  --top-actors 5

Expected output (values will differ by hour):

  12453  dependabot[bot]
   8901  renovate[bot]
   4312  github-actions[bot]
   2100  octocat
    987  some-human

✓ Self-check: for any single hour in 2024, the #1 actor is almost always a bot account — look for [bot] in the login. If your top result is a human, double-check that you are iterating over ev["actor"]["login"], not ev["actor"].

Task 3 — Predict the Lesson 02 win

Without writing any code, estimate: if you stored the same 24 hours as Parquet partitioned by event type instead of one combined CSV, how would each of the following change?

1. The total disk footprint.
2. The time to answer "how many PushEvents on 2024-01-01?"
3. The time to answer "top 10 actors across all types on 2024-01-01?"

Write 80–120 words. Defend your numbers in one sentence each.

✓ Self-check: your answer mentions columnar, compression, and at least one of predicate pushdown or partition pruning. (Don't worry if those terms aren't in your vocabulary yet — peek at Slide 2 of Lesson 02 if needed, then come back.)