Lesson 03 · Slide 1

Lesson 03 — Data Quality with Great Expectations

Catching the moment your data lies to you.

Data Engineering Module 1

Lesson 03 · Slide 2

The four kinds of broken data

Schema drift — a column appears, disappears, or changes type.
Value drift — the shape of values changes (e.g., a new event type, or a sudden spike in nulls).
Volume drift — yesterday's hour had 50 K rows; today's has 50.
Freshness drift — the file landed 6 hours late.

Lesson 03 · Slide 3

Test in code vs test in data

Unit tests (code)

Check the code that processes data — gates the CI build.

Data-quality checks (data)

Check the data the code processed — gates the pipeline run.

Different lifecycles, different alerting. A unit test passing does not mean today's data is good.

Lesson 03 · Slide 4

The Great Expectations vocabulary

Term	What it is
Expectation	A single assertion about data (e.g., "column X is never null").
Suite	A named bundle of expectations applied to one table.
Checkpoint	A scheduled run of a suite against fresh data.
Data docs	Generated HTML reports of run results.

Lesson 03 · Slide 5

A first expectation

import great_expectations as gx
context = gx.get_context()
dx = context.sources.add_pandas("gh").read_parquet("data/parquet/date=2024-01-01/")
result = dx.expect_column_values_to_not_be_null("type")
print(result.success)   # True

One call. One assertion. Returns a result object with success, observed value, and a sample of unexpected rows.

Lesson 03 · Slide 6

The Module 1 starter suite

Expectations to add to the gharchive events table:

expect_column_to_exist for type, actor, repo, created_at
expect_column_values_to_not_be_null for all four
expect_column_values_to_be_in_set for type (the known event types)
expect_column_values_to_match_strftime_format for created_at (%Y-%m-%dT%H:%M:%SZ)
expect_table_row_count_to_be_between (sanity bounds)

Lesson 03 · Slide 7

Drift detection — `expect_column_values_to_be_in_set`

The set of GitHub event types changes slowly but does change (e.g., DiscussionEvent added in 2020). Pin the known set; let the expectation fail loudly when a new value appears; then decide whether to accept or reject the new type.

This is not a bug to fix in code — it is a data contract decision. Document it. Update the expectation. Tell downstream consumers.

Lesson 03 · Slide 8

What to do when validation fails

Severity	Example	Action
block-the-pipeline	required column missing	fail loudly; alert on-call; do not load downstream.
warn-but-load	new event type appeared	annotate the load; surface in dashboards; review in standup.
info	row count up 10% week-over-week	dashboard only; no alert.

Lesson 03 · Slide 9

Where it fits in the pipeline

extract -> transform -> [VALIDATE] -> load -> downstream

The validate step is between transform and load — sometimes also right after extract. On failure, the load step is skipped.

Lesson 03 · Slide 10

The L03 sample-pipeline shape

sample-pipeline/
  extract/
  transform/
  quality/
    expectations.py        # the suite
    checkpoints/           # GE-generated
    data_docs/             # GE-generated HTML

Lesson 03 · Slide 11

Common pitfalls

Adding 100 expectations at once. Start with 5–10; let them stabilise.
Pinning a value set to "the last 30 days" without a process for updating it.
Treating every failure as block-the-pipeline. You'll silence the alerts within a week.

Lesson 03 · Slide 12

What this course is not doing today

Full schema contracts (Lesson 13), automated profiling (Lesson 12), CI integration of suites (Lesson 06). Today: a hand-written suite, running locally, against the L02 output.

One step at a time. The primitives you learn here are the building blocks for all of those advanced patterns.

Lesson 03 · Slide 13

Recap & what's next

Data breaks in four ways — schema, value, volume, and freshness drift.
Great Expectations gives you assertions, suites, checkpoints, and HTML reports.
Severity tiers (block / warn / info) keep your on-call sane.

Coming up: Module 2 — wrap all of this in Airflow.