Adding columns without downtime; full lineage end-to-end.
Data Engineering Module 5 ProductionEvery production pipeline accumulates schema changes. Upstream teams add columns, rename fields, and split tables — often without warning. Two deployment patterns emerge:
Consumers break in production. On-call at 3 am.
No outage. No scramble. No 3 am pages.
| Mode | Definition | Safe change pattern |
|---|---|---|
| Backward | New code reads old data without error | ADD column with a default value |
| Forward | Old code reads new data without error | Never REMOVE or RENAME an existing column |
| Full | Both backward and forward hold simultaneously | ADD nullable column (safe in both directions) |
The gharchive pipeline targets full compatibility throughout its schema history. The data_contract.yml declares this as a guarantee to all consumers.
int32 → int64)These can deploy with zero coordination. Consumers see new data; old code ignores the new column.
int64 → int32)These require a dual-write window, a consumer audit, and a planned removal date.
The sample pipeline demonstrates three safe additive schema versions:
| Version | Columns | Change | Compatible? |
|---|---|---|---|
| v1 | id, actor, repo, created_at |
Baseline | — |
| v2 | v1 + payload_size |
ADD nullable int64 | Full |
| v3 | v2 + event_type |
ADD string with default "unknown" |
Full |
Both v2 and v3 deploy with zero downstream changes. A v1 consumer reading v3 data simply ignores payload_size and event_type.
A data contract is a YAML the producer publishes. It declares: here is my schema; here is my deprecation policy; here is who owns this. PRs that change the schema must also update the contract — and the contract test must pass before merge.
version: v3
namespace: gharchive-pipeline
dataset: events
fields:
- { name: actor, type: string, required: true, added_in: v1 }
- { name: event_type, type: string, required: true, added_in: v3, default: unknown }
compatibility:
backward: true
forward: true
deprecation_policy:
removal_notice_minimum_days: 30The full contract lives at sample-pipeline/schema_evolution/data_contract.yml.
def test_v3_reads_v1_consumer_safe():
"""A v1 consumer reading v3 must still find v1 columns (backward compat)."""
v3 = pq.read_table(FIXTURES / "v3.parquet").select(
["id", "actor", "repo", "created_at"])
assert v3.num_rows == 10
assert v3.column_names == ["id", "actor", "repo", "created_at"]
def test_v1_reads_v3_producer_safe():
"""v1 columns are a strict subset of v3 (forward compat)."""
assert _schema("v1.parquet").issubset(_schema("v3.parquet"))The full suite at schema_evolution/test_compatibility.py runs 7 tests in CI. It catches any schema change that would break a v1 consumer before the PR is merged. Add new tests alongside every new version.
Lesson 12 emitted job-level START and COMPLETE events to Marquez. Lesson 13 makes lineage queryable end-to-end: which dbt model produced this column? Which downstream table reads from it? What changed in the last deploy?
extract → transform → load. Already in Marquez from L12.SchemaDatasetFacet. Marquez shows the before/after diff without custom tooling.Screenshot placeholder: Marquez column-lineage view — actor traced from raw_events through dim_events to daily_counts.
version: 2
models:
- name: dim_events
columns:
- name: event_type
description: "Top-level GitHub event type (push, pull_request, ...)"
meta:
contract:
added_in: v3
default: unknowndbt emits column-level facets to OpenLineage when dbt-openlineage is in the project. Marquez stitches these across operators — you see which raw column flows into which mart column, without writing any custom lineage code. The meta.contract block ties the dbt model column back to the data_contract.yml version.
When a downstream dashboard goes red, the debugging flow becomes a graph traversal:
Marquez REST API query:
curl "http://localhost:5000/api/v1/lineage?nodeId=dataset:gharchive-pipeline:dim_events"The response lists every upstream job and run. The most recent run with a schema diff facet is usually the culprit. No log-grepping required — the lineage graph is the audit trail.
This course stops at batch. Streaming (Kafka, Flink, Spark Structured Streaming) lets you push freshness from minutes to seconds, but it adds exactly-once and watermark concerns we won’t cover here. The OpenLineage and data-contracts patterns from this lesson transfer directly to streaming pipelines — just the engines change. Schema registries (Confluent, Apicurio) play the same role as data_contract.yml in a batch pipeline: they are the deploy gate between producers and consumers.
country nullable column as v4 — a safe additive change.data_contract.yml to v4 and add two new compatibility tests.By the end of lab you have a living v4 contract, a green test suite, and a lineage event that records the evolution.
sample-pipeline/
+ schema_evolution/
+ __init__.py
+ generate_fixtures.py # run once; NOT committed as fixture output
+ test_compatibility.py # 7-test pytest suite; runs in CI
+ data_contract.yml # producer/consumer schema guarantee
+ fixtures/ # v1/v2/v3 Parquet; generated locally
observability/ # unchanged from L12
airflow/ # unchanged from L12
dbt/ # unchanged from L12The actor column and all upstream schema contracts from Modules 1–4 are preserved. No existing tests break. The only additions are the schema_evolution/ tree.