Solutions — Homework 03

Lesson 03 · Data Engineering with Python · Worked solutions

Solution to Task 1 — Schema drift across years

Running the L01–L02 pipeline against 2015-01-01-12 and 2024-01-01-12 and then applying the L03 GE checkpoint to each reveals that the event-type vocabulary has grown significantly over the nine years.

The 2015 hour passes all expectations. The event types present are a subset of the known set: PushEvent, WatchEvent, CreateEvent, IssuesEvent, PullRequestEvent, ForkEvent, DeleteEvent, GollumEvent, IssueCommentEvent, CommitCommentEvent, MemberEvent.

The 2024 hour, by contrast, contains event types that did not exist in 2015 — most notably PullRequestReviewEvent (introduced 2016), PullRequestReviewCommentEvent (introduced 2016), and DiscussionEvent (introduced 2020). If the known-types set in the suite was calibrated to the 2015 vocabulary, those three types would all fail the expect_column_values_to_be_in_set check on the 2024 data.

The expected GE failure output looks like this:

expect_column_values_to_be_in_set
  column: type
  success: False
  observed_value: ["PullRequestReviewEvent", "PullRequestReviewCommentEvent", "DiscussionEvent", ...]
  unexpected_count: 4,212
  unexpected_percent: 3.8
  partial_unexpected_list: ["DiscussionEvent", "DiscussionEvent",
                             "PullRequestReviewEvent", "DiscussionEvent", ...]

Common mistake: "fixing" the failure by immediately adding DiscussionEvent to the known-types set without any process. The correct response is:

1. Decide whether to accept the new type (does your downstream consumer understand it? does it need special handling?).
2. Document the decision — who approved it, when, and why.
3. Update the expectation and commit the change with an explanatory message.
4. Backfill if necessary: if historical data has the new type and needs to be reprocessed, run the pipeline on affected partitions.
5. Notify downstream consumers that a new event type is now flowing through the system.

Lesson 13 turns this informal procedure into a formal data contract with automated schema-change notifications.

Solution to Task 2 — Two more expectations

The two recommended expectations and their implementations:

Option A — Actor login length

# Guards against: value drift (malformed actor data, e.g., from early gharchive rows
# where deleted-account placeholders were longer than 39 chars)
validator.expect_column_value_lengths_to_be_between(
    "actor", min_value=1, max_value=40
)
# Note: GitHub's stated limit is 39 chars, but we use 40 as the max to allow
# for edge cases. Tighten to 39 once you've validated against real data.

To make this fail, write a row with a 50-character actor login:

import pandas as pd, pathlib

df = pd.read_parquet("data/parquet/date=2024-01-01/")
bad = df.iloc[0:1].copy()
bad["actor"] = "a" * 50   # 50-char login
df_bad = pd.concat([df, bad], ignore_index=True)
out = pathlib.Path("data/parquet/date=2024-01-01-bad-actor")
out.mkdir(parents=True, exist_ok=True)
df_bad.to_parquet(out / "events.parquet", index=False)

Expected failure output:

expect_column_value_lengths_to_be_between
  column: actor
  success: False
  unexpected_count: 1
  partial_unexpected_list: ["aaaaa...aaaaa"]  # 50 chars

Option B — Repository name format

# Guards against: value drift (malformed repo names; some early gharchive rows
# had bot-generated entries with no slash separator)
validator.expect_column_values_to_match_regex(
    "repo", r"^[^/]+/[^/]+$"
)

To make this fail, write a row with a repo name that has no slash:

bad["repo"] = "no-slash-here"

Expected failure output:

expect_column_values_to_match_regex
  column: repo
  success: False
  unexpected_count: 1
  partial_unexpected_list: ["no-slash-here"]

Common mistake: choosing an expectation that does not actually catch real bugs in the dataset. The two above do catch real bugs: early gharchive rows (2011–2013) occasionally had malformed actor data from deleted accounts, and a handful of archived events had repo names without a slash due to a GitHub API bug in 2013. Expectations should be grounded in observed failure modes, not hypothetical ones.

Solution to Task 3 — Severity policy

Model three-tier classification for the full suite of 8 expectations:

Common mistake: putting expect_column_values_to_be_in_set("type", KNOWN_TYPES) in the block tier. New event types appear — GitHub added DiscussionEvent, PullRequestReviewEvent, and others over the years. If the pipeline halts every time GitHub adds a new verb, the on-call will mute the alert within a week. The warn tier is correct: it surfaces the problem for human review without stopping the load.

Note the block tier has exactly 3 items — the minimum number of invariants that downstream consumers truly cannot tolerate. This keeps the block tier credible. When a block-the-pipeline alert fires, engineers take it seriously because false positives are rare.