Lab 03 — Build a Great Expectations suite for gharchive events

Lesson 03 · Data Engineering with Python · ~45 min

By the end of this lab you will have a runnable GE checkpoint that reads the L02 Parquet output, runs eleven expectations, and writes an HTML report. You will also intentionally break the data and observe what the failure report looks like — the most important thing you can do to build intuition for a new tool.

What you need

• The L02 Parquet output at data-engineering/sample-pipeline/data/parquet/date=2024-01-01/ — at least one .parquet file from the previous lab.
• The virtual environment from L01/L02 (already has pandas and pyarrow).
• great-expectations>=0.18,<1 — already listed in data-engineering/sample-pipeline/requirements.txt. If you skipped the install step, run pip install "great-expectations>=0.18,<1" now.

Step 1 — Initialise GE in the sample-pipeline

Navigate to the sample-pipeline directory and run the GE init command:

cd data-engineering/sample-pipeline
source .venv/bin/activate       # or however you activate the venv
great_expectations init

GE will ask you a few questions about your data source. Answer "no" to the interactive wizard prompts for now — we will configure the datasource in code. After the command completes you should have a gx/ folder containing:

gx/
  great_expectations.yml
  expectations/
  checkpoints/
  uncommitted/
    config_variables.yml
    data_docs/

Step 2 — Implement build_suite in quality/expectations.py

Open data-engineering/sample-pipeline/quality/expectations.py. Replace the NotImplementedError body with the following implementation:

"""Great Expectations suite for the gharchive events table."""
from __future__ import annotations

import pathlib
import great_expectations as gx
from great_expectations.core.batch import RuntimeBatchRequest

KNOWN_EVENT_TYPES = [
    "PushEvent", "PullRequestEvent", "IssuesEvent",
    "WatchEvent", "ForkEvent", "CreateEvent", "DeleteEvent",
    "ReleaseEvent", "PublicEvent", "MemberEvent", "GollumEvent",
    "CommitCommentEvent", "IssueCommentEvent",
    "PullRequestReviewEvent", "PullRequestReviewCommentEvent",
    "DiscussionEvent",
]

SUITE_NAME = "gharchive_events"


def build_suite(parquet_dir: str | pathlib.Path) -> None:
    """Build and save the starter expectation suite.

    Args:
        parquet_dir: Path to a directory of .parquet files to validate against.
    """
    context = gx.get_context()

    # Create a Pandas datasource
    datasource = context.sources.add_or_update_pandas("gharchive")
    asset = datasource.add_parquet_asset(
        name="events",
        batching_regex=r".*\.parquet",
    )
    batch_request = asset.build_batch_request()

    # Get a validator to build the suite
    validator = context.get_validator(
        batch_request=batch_request,
        expectation_suite_name=SUITE_NAME,
    )

    # --- Schema expectations (block-the-pipeline tier) ---
    for col in ("type", "actor", "repo", "created_at"):
        validator.expect_column_to_exist(col)
        validator.expect_column_values_to_not_be_null(col)

    # --- Value expectations (warn-but-load tier) ---
    validator.expect_column_values_to_be_in_set("type", KNOWN_EVENT_TYPES)
    validator.expect_column_values_to_match_strftime_format(
        "created_at", strftime_format="%Y-%m-%dT%H:%M:%SZ"
    )

    # --- Volume expectation (info tier) ---
    validator.expect_table_row_count_to_be_between(
        min_value=1_000, max_value=500_000
    )

    # Save the suite
    validator.save_expectation_suite(discard_failed_expectations=False)
    print(f"Suite '{SUITE_NAME}' saved.")


if __name__ == "__main__":
    import sys
    parquet_dir = sys.argv[1] if len(sys.argv) > 1 else "data/parquet/date=2024-01-01"
    build_suite(parquet_dir)

Run it to save the suite:

python quality/expectations.py data/parquet/date=2024-01-01

Expected output: Suite 'gharchive_events' saved. and a new file at gx/expectations/gharchive_events.json.

Step 3 — Wire a checkpoint

Create a checkpoint configuration file at gx/checkpoints/gharchive-events.yml:

name: gharchive-events
config_version: 1.0
class_name: SimpleCheckpoint
validations:
  - batch_request:
      datasource_name: gharchive
      data_asset_name: events
      options:
        path: data/parquet/date=2024-01-01
    expectation_suite_name: gharchive_events

Alternatively, run the interactive wizard:

great_expectations checkpoint new gharchive-events

Then configure it to point at the gharchive datasource, the events asset, and the gharchive_events suite.

Step 4 — Run the checkpoint

great_expectations checkpoint run gharchive-events

Expected output: a summary showing all expectations passing, followed by a line like:

Validation Succeeded!
  Evaluated 11 expectations
  11 passed, 0 failed

GE also writes a full HTML report to:

gx/uncommitted/data_docs/local_site/index.html

Step 5 — Break it on purpose

This is the most instructive step. Open Python and write a Parquet file with a deliberately bad row:

import pandas as pd, pathlib

# Read an existing hour
df = pd.read_parquet("data/parquet/date=2024-01-01/")

# Inject a row with an unknown event type
bad_row = df.iloc[0:1].copy()
bad_row["type"] = "ElonMuskEvent"
df_bad = pd.concat([df, bad_row], ignore_index=True)

# Save to a new test directory
out = pathlib.Path("data/parquet/date=2024-01-01-bad")
out.mkdir(parents=True, exist_ok=True)
df_bad.to_parquet(out / "events.parquet", index=False)
print(f"Written {len(df_bad)} rows with 1 bad type value")

Now update your checkpoint's path to point at data/parquet/date=2024-01-01-bad and re-run:

great_expectations checkpoint run gharchive-events

Expected: the expect_column_values_to_be_in_set expectation fails. The output shows the unexpected count (1) and the partial unexpected list (["ElonMuskEvent"]).

Step 6 — Open the data docs

open gx/uncommitted/data_docs/local_site/index.html

Browse the per-expectation results. Click on the failed expectation to see the sample of offending values. Click on "Validation History" to see the run that passed followed by the run that failed. This is what GE's data docs look like in production — a living record of every validation run.

What you should leave with

• A working GE suite + checkpoint that validates the L02 Parquet output.
• A visceral feel for what GE shows you when an expectation fails: the unexpected count, the partial list, the HTML report.
• One intentionally broken run in your data docs history — a useful reminder that failures are normal and expected (pun intended).

Troubleshooting