Homework 03 — Detect drift, extend the suite, classify severity

Task 1 — Detect schema drift across years

Pick two hours from very different years — for example, 2015-01-01-12 (hour 12 of January 1, 2015) and 2024-01-01-12 (same slot in 2024). Run your full L01–L02 pipeline against both hours separately to produce two Parquet datasets.

Then run your L03 GE checkpoint against each in turn. Point the checkpoint at the 2015 output first, then the 2024 output.

To download a specific hour from GHArchive, the URL pattern is:

https://data.gharchive.org/2015-01-01-12.json.gz
https://data.gharchive.org/2024-01-01-12.json.gz

Adapt your L01 extract script to fetch these specific URLs, then run the L02 transform on the resulting files.

✓ Self-check: at least one expectation fails on the 2015 output that passes on the 2024 output (the event-type vocabulary has grown significantly between 2015 and 2024). Record: which expectation failed, what were the unexpected values, and how many rows were affected.

Task 2 — Add two more expectations

Choose two of the following expectations to add to your suite. For each, write one sentence explaining which category of data breakage (schema / value / volume / freshness drift) it guards against.

• expect_column_value_lengths_to_be_between("actor", min_value=1, max_value=39) — GitHub login names are 1–39 characters by platform rules.
• expect_column_values_to_match_regex("repo", r"^[^/]+/[^/]+$") — every repository name must be in owner/name format.
• expect_column_pair_values_A_to_be_greater_than_B — over two derived timestamp fields (requires adding a computed column to your transform; challenging).
• expect_table_row_count_to_be_between with bounds you calibrated from your own downloaded hours — use df.shape[0] to measure several real hours and pick a reasonable [min, max] range.

Update quality/expectations.py to include both new expectations. Re-save the suite and re-run the checkpoint against a valid hour to confirm both pass.

Then manually corrupt the input to make each new expectation fail. For example: write a row with a 50-character actor login, or a repo name with no slash. Confirm GE reports the failure and shows the offending values.

✓ Self-check: both new expectations pass on at least one real downloaded hour, and fail when you manually corrupt the input. Capture the failure output (unexpected count + partial unexpected list) for each.

Task 3 — Decide a severity policy

You now have approximately 8 expectations in your suite (6 from the lab + 2 from Task 2). For each expectation, classify it as block / warn / info. Write one sentence per classification explaining your reasoning.

Format your answer as a table:

Expectation	Severity	Reason
expect_column_to_exist("type")	block	Your reasoning here.
... one row per expectation ...

✓ Self-check: no more than 3 expectations are in the block tier. Pipelines with too many blockers get muted — on-call learns to ignore alerts when they fire constantly. If you find yourself wanting to block on more than 3, ask whether those expectations truly catch data that downstream consumers cannot tolerate.