Schema Evolution & Lineage

Lesson 13 · Data Engineering with Python · ~15 min read

1. Why schema evolution is the second-hardest problem in data

The hardest problem in data engineering is data quality — ensuring that the values in your tables actually mean what you think they mean (covered in Lesson 3). Schema evolution is the second-hardest because it sits at the intersection of three systems that change independently: the code that writes data, the infrastructure that stores it, and the contracts that downstream consumers rely on.

Software engineers who have worked primarily in service-oriented architectures sometimes underestimate this difficulty. In a web service, you control both the API producer and the consumer, and you can deploy them together. In a data platform, you may have dozens of consumers — dashboards, dbt models, ad-hoc notebook analyses, ML feature pipelines — each of which references columns by name. A single rename that takes thirty seconds to type can silently break six downstream assets over the next twenty-four hours as they pick up the new data.

Martin Kleppmann’s Designing Data-Intensive Applications (Chapter 4, “Encoding and Evolution”) frames this precisely: data outlives the code that wrote it. A Parquet file written by your v1 transform will still be sitting in object storage when your v3 transform runs. Your v3 code must be able to read it correctly. That is the backward-compatibility constraint. The symmetric constraint — forward compatibility — arises when you deploy a new producer before all consumers have upgraded. Both matter, and both are testable with the techniques in this lesson.

2. The three compatibility modes

Kleppmann’s framework gives us precise language for three modes of schema compatibility. Understanding the mode your change falls into determines how much coordination you need before you can safely deploy.

Backward compatibility means new code can read old data without error. In the gharchive pipeline context: your v3 transform code reads a v1 Parquet file (one that predates the payload_size and event_type columns) and produces a correct result. PyArrow handles this transparently — columns that are declared in the schema but absent from the file get filled with nulls or defaults. The test test_v3_reads_v1_consumer_safe in the sample pipeline verifies this explicitly: it reads a v3 Parquet with a v1 column selection and asserts the result is correct.

Forward compatibility means old code can read new data without error. Your v1 dashboard queries the events table and selects only actor, repo, and created_at. The table now contains v3 data with two extra columns. The dashboard ignores payload_size and event_type — it simply never selects them. No error, no stale data. The test test_v1_reads_v3_producer_safe verifies this: the v1 column set is a strict subset of the v3 column set.

Full compatibility is both properties simultaneously. It is achievable whenever you add nullable columns or columns with safe defaults. It is not achievable when you rename, remove, or narrow a column — those operations break at least one direction. The gharchive pipeline targets full compatibility throughout its schema history, as declared in data_contract.yml.

3. Safe vs unsafe schema changes

Not every schema change is compatible. Knowing which operations are safe lets you deploy without coordination; knowing which are unsafe lets you plan migrations before they become incidents.

For unsafe operations the migration pattern is: dual-write → consumer upgrade → remove. During the dual-write window, the producer emits both the old and new representation (e.g., both actor and the new actor_username column). Consumers migrate at their own pace. After at least 30 days and after every known consumer has migrated, the old column is removed and the contract is bumped to a new major version.

4. Data contracts

A data contract is a machine-readable declaration that a producer makes to its consumers. It answers the questions: what is the schema? What compatibility guarantees do I offer? Who owns this dataset? What is the deprecation policy? Unlike a docstring or a Confluence page, a data contract lives in the repository, is reviewed in pull requests, and is enforced by a CI test suite.

The sample pipeline’s contract lives at schema_evolution/data_contract.yml. Every field carries a required flag, an added_in version tag, and (where applicable) a default value. The top-level compatibility block declares the guaranteed modes. The deprecation_policy block sets the minimum notice period for breaking changes. The ownership block names the responsible team and their alert channel.

The CI enforcement mechanism is the pytest suite in test_compatibility.py. A PR that adds a new column must also add the corresponding field entry in data_contract.yml and a new test that proves the addition is compatible. If the test is missing, CI fails. If the contract YAML is missing the field, the contract validation step fails. The contract and the tests are co-owned: neither is trusted without the other.

Two adjacent standards are worth knowing. dbt’s model contracts feature (introduced in dbt Core 1.5) lets you declare column types and constraints directly in models.yml; dbt enforces them at build time with enforced: true. The data-contract open standard at datacontract.com provides a more comprehensive YAML specification covering SLAs, quality rules, and multi-team ownership — the data_contract.yml in this course is compatible with its spirit, though simplified for teaching purposes.

5. Lineage end-to-end

Lesson 12 established OpenLineage as the event bus for job-level lineage. A START event fires when a DAG task begins; a COMPLETE event fires when it finishes. Marquez stores these events and builds a job graph: extract → transform → load. That graph is sufficient to answer “which DAG runs touched this dataset today?”

Lesson 13 upgrades the picture with column-level lineage facets. When dbt is instrumented with dbt-openlineage, each model emits a ColumnLineageDatasetFacet alongside the standard run event. Marquez stitches these facets across operators: you can now trace the actor column from the raw JSON extract through the PyArrow Parquet transform, into the dim_events dbt model, and onward to the daily_counts mart. No custom ETL lineage code is required; the instrumentation is purely declarative.

The practical payoff is fast debugging. When a dashboard goes red, the procedure is: open Marquez, find the broken dataset, query for the most recent upstream write, follow the run’s input datasets backward until you reach a human-authored change. In a real incident, this traversal takes seconds via the REST API or the Marquez UI rather than hours of log-grepping. The Marquez query is:

curl "http://localhost:5000/api/v1/lineage?nodeId=dataset:gharchive-pipeline:dim_events"

The response lists every upstream job and the schema facets emitted during each run. The most recent run that changed the schema is almost always the culprit.

6. Streaming as the next frontier

Batch pipelines stop at the freshness floor set by their schedule — for the gharchive pipeline, that is roughly 90 minutes. Streaming pipelines (Kafka, Flink, Spark Structured Streaming) push that floor to seconds. The schema-evolution and lineage patterns from this lesson transfer directly: a Kafka topic has a schema registry (Confluent Schema Registry, Apicurio) instead of a YAML file, but it plays exactly the same role as data_contract.yml — it is the deploy gate between producers and consumers. OpenLineage events flow from Flink jobs just as they do from Airflow tasks. The engines change; the principles do not. This course covers batch; a streaming companion course is on the roadmap.

7. Common pitfalls

References