Data Quality with Great Expectations

Lesson 03 · Data Engineering with Python · ~12 min read

Section 1 — The four kinds of broken data

Your pipeline can run without errors, your unit tests can pass, and your data can still be wrong. The reason is that data bugs do not look like code bugs: they arrive silently, as changed values in a file the extract step downloaded successfully. Understanding how data breaks is the first step toward catching it.

Schema drift happens when the structure of your incoming data changes. A column you rely on disappears. A column you never expected appears. A field that was always an integer starts arriving as a string. In the gharchive context, this is exactly what happened when GitHub added the DiscussionEvent event type in 2020: pipelines that had hardcoded column expectations started failing silently, loading partial data without realising it. Schema drift is insidious because the code does not raise an exception — it just produces subtly wrong output.

Value drift is subtler. The schema is fine, but the distribution of values has shifted. Imagine that bot accounts on GitHub doubled overnight: your actor-name cardinality explodes, your aggregations look wrong, your dashboards spike. Or a new event type appears in the type column that your downstream consumers have never seen and do not know how to handle. The column exists, it has a string value, but the string means something your code was never designed for.

Volume drift is the one that most obviously points to an outage. An hourly pipeline that normally processes 40,000–60,000 rows suddenly delivers 50. This almost always means the upstream source had an incident — the GitHub API was throttling, the archiving job missed a window, or a network partition dropped packets. Without a volume check, the downstream tables quietly shrink and no one notices until a dashboard looks suspiciously quiet.

Freshness drift means the data arrived, but late. Your hourly pipeline expected the 14:00 file by 14:15. It landed at 20:00. If downstream jobs depend on that data being present before they run, they will either fail noisily or — worse — succeed silently on stale data from yesterday. A freshness expectation captures the SLA: "this data must be no older than 90 minutes." When it is violated, alert before downstream jobs run, not after.

Section 2 — Code tests vs data tests

Unit tests verify that your transform function produces the correct output for a given input. They are fast, deterministic, and run in CI. They are excellent at catching regressions in logic: if you change how events.py parses the created_at field, a unit test tells you immediately.

What unit tests cannot do is tell you whether today's actual data is good. Your transform function can be perfectly correct and the data it transforms can still be broken. The function does not produce the data — GitHub does. You do not control what GitHub publishes. You do not control whether the archiving tool captured it faithfully. You do not control what changes GitHub makes to its event schema over the lifetime of your pipeline.

Data-quality checks live at a different point in the lifecycle. They run after your pipeline produces output, or between pipeline stages, and they assert properties of the actual rows and columns in the actual file. They are not about whether the code is correct — they are about whether the data meets the contract your downstream consumers rely on. This parallel test surface — the data itself — lives and breathes and changes long after the code is "done."

Section 3 — Great Expectations in 5 minutes

Great Expectations (GE) is a Python library that turns data assertions into a first-class, testable, documented artifact. The core vocabulary:

The course pins great-expectations>=0.18,<1 (the v0 API). The v1 API was introduced in 2024 and involves a substantially different object model. The v0 API is more widely deployed and the concepts transfer directly; the v1 upgrade path is noted in Lesson 13.

Here is a minimal working example that builds and runs a suite against the L02 Parquet output:

import great_expectations as gx

# Get or create a local GE context (creates gx/ directory on first run)
context = gx.get_context()

# Add a Pandas datasource pointing at a Parquet directory
datasource = context.sources.add_pandas("gharchive")
asset = datasource.add_parquet_asset(
    name="events",
    batching_regex=r".*\.parquet",
)
batch_request = asset.build_batch_request()

# Get a batch and run expectations
validator = context.get_validator(batch_request=batch_request)
validator.expect_column_to_exist("type")
validator.expect_column_values_to_not_be_null("type")
validator.expect_column_values_to_be_in_set(
    "type",
    ["PushEvent", "PullRequestEvent", "IssuesEvent",
     "WatchEvent", "ForkEvent", "CreateEvent"]
)

# Save the suite and run a checkpoint
validator.save_expectation_suite(discard_failed_expectations=False)
checkpoint = context.add_or_update_checkpoint(
    name="gharchive-events",
    validations=[{"batch_request": batch_request,
                  "expectation_suite_name": "default"}],
)
result = checkpoint.run()
print("All passed:", result.success)

The pipeline diagram below shows where validation gates the data flow:

Section 4 — Three severities of failure

Not every failed expectation should wake someone up at 3 a.m. The most important discipline in data quality is matching the severity of the response to the severity of the problem.

The alerting story: every block-the-pipeline failure is an on-call page. Warns go to a Slack channel that gets reviewed daily. Infos go to dashboards that get reviewed weekly (or when investigating a bug). The discipline that keeps this working is keeping the block tier small. A pipeline with 20 block-the-pipeline expectations will generate false positives constantly, on-call will start ignoring the pages, and one day a genuine critical failure will be missed. Block the pipeline only for invariants that downstream systems truly cannot tolerate.

Section 5 — A starter suite for the gharchive events table

The starter suite for Module 1 contains six expectations. Each maps to a specific category of data breakage:

expect_column_to_exist("type"), expect_column_to_exist("actor"), expect_column_to_exist("repo"), expect_column_to_exist("created_at") — catches schema drift. If GitHub changes a column name or your transform renames a field, this fires. These should be in the block tier: downstream queries that reference type will break immediately if the column is gone.

expect_column_values_to_not_be_null("type"), expect_column_values_to_not_be_null("actor"), etc. — catches value drift of the worst kind: complete loss of a field. In real gharchive data, a handful of events have null actors due to deleted accounts, but type should never be null. This is block-tier for type, warn-tier for actor.

expect_column_values_to_be_in_set("type", KNOWN_TYPES) — catches value drift in the vocabulary of event types. The known set as of 2024 includes PushEvent, PullRequestEvent, IssuesEvent, WatchEvent, ForkEvent, CreateEvent, DeleteEvent, ReleaseEvent, PublicEvent, MemberEvent, GollumEvent, CommitCommentEvent, IssueCommentEvent, PullRequestReviewEvent, PullRequestReviewCommentEvent, and DiscussionEvent. This should be warn-tier: new event types appear slowly and should be reviewed, not silently rejected.

expect_column_values_to_match_strftime_format("created_at", "%Y-%m-%dT%H:%M:%SZ") — catches freshness-related and schema-related timestamp malformation. Early gharchive data has slightly different timestamp formats; this expectation will fire on old hours and give you visibility into where format assumptions break.

expect_table_row_count_to_be_between(min_value=1000, max_value=500000) — catches volume drift. Real bounds should be calibrated from your actual data. 1,000 is a conservative floor (any hour with fewer rows probably has a pipeline problem); 500,000 is a generous ceiling (most hours are 40,000–100,000 rows). This is info-tier: a spike is worth knowing about, but probably does not warrant blocking the pipeline.

Section 6 — Common pitfalls

References

• Great Expectations documentation — the authoritative source; use the v0.18 docs to match the pinned version in this course.
• Great Expectations v0.18 docs (direct link)
• Data Quality Fundamentals (O'Reilly, Barr Moses et al.) — the most comprehensive book on the topic.
• dbt blog: "What are dbt tests and why should I use them?" — a useful comparison of dbt tests (transform-layer assertions) vs GE expectations (data-layer assertions).
• OpenLineage — data lineage standard used in Lesson 13 to trace which pipeline version produced which data.