Homework 13 — Schema Evolution & Lineage

Lesson 13 · Data Engineering with Python · 3 exercises

All three tasks build on the schema_evolution/ tree added in Lesson 13. Task 1 is a written migration-planning exercise. Task 2 involves adding a dbt model contract and running dbt build to prove enforcement works. Task 3 is an observational tracing exercise using Marquez or the JSON event files.

Task 1 — Propose an unsafe change and write the migration plan

Pick one of the following unsafe schema changes to the events dataset:

Rename actor to actor_username
Remove the payload_size column
Narrow id from int64 to int32

For the change you pick, write a 3-step migration plan:

Dual-write window — describe exactly what the producer emits during the transition period (both old and new). State how long this window must last according to the data_contract.yml deprecation policy.
Consumer audit — describe how you would enumerate every downstream consumer that references the field being changed. Name at least two consumer types from the sample pipeline (e.g., dbt models, notebooks, dashboards).
Removal — describe the conditions that must be met before the old field can be dropped, and what the data_contract.yml entry looks like after removal (include a removed_in field).

Self-check: your plan is at least 100 words, explicitly references the 30-day minimum from data_contract.yml’s deprecation_policy, and names the consumer count you need to audit before removal.

Task 2 — Add a forward-compatibility contract to the dbt project

In the dbt project under data-engineering/sample-pipeline/dbt/, add a model contract block for dim_events in models.yml (create the file if it does not exist yet):

version: 2
models:
  - name: dim_events
    config:
      contract:
        enforced: true
    columns:
      - { name: id,         data_type: bigint,     constraints: [{type: not_null}] }
      - { name: actor,      data_type: varchar }
      - { name: created_at, data_type: timestamp }

Run the dbt build to confirm the contract is enforced:

cd data-engineering/sample-pipeline/dbt
dbt build

Then deliberately introduce a type mismatch (for example, change id’s data_type to varchar in the contract while the model still casts it as bigint) and run dbt build again to confirm it fails with a contract violation error.

Self-check: the first dbt build succeeds; the second (with the deliberate mismatch) fails with a message that clearly names the column and the type conflict. Revert the mismatch before moving on.

Tip: if you do not have a working dbt project yet, you can still complete this task by reading the dbt model-contracts documentation (docs.getdbt.com) and writing the YAML as a design artifact. Describe what the expected dbt build failure message would look like and why enforced: false defeats the purpose of having a contract.

Task 3 — Trace lineage one hop upstream

Pick any downstream table from the sample pipeline — for example dim_events or events_v4 — and use Marquez (or the JSON event files under sample-pipeline/observability/events/) to identify the most recent upstream job that wrote to its input dataset.

Using Marquez: open http://localhost:3000, navigate to Datasets, select your chosen dataset, then click Lineage to see the upstream job. Expand the job to read its run-id and completion timestamp.

Using the file backend: inspect the JSON files in observability/events/. Each file is a single OpenLineage event object. Find the most recent COMPLETE event whose outputs array contains the dataset you are tracing. Read the run.runId and eventTime fields.

# List events sorted by modification time (most recent last)
ls -lt data-engineering/sample-pipeline/observability/events/ | head -10

# Inspect a specific event file
python -c "
import json, pathlib
p = pathlib.Path('data-engineering/sample-pipeline/observability/events/')
files = sorted(p.glob('*-COMPLETE.json'), key=lambda f: f.stat().st_mtime)
if files:
    print(json.dumps(json.loads(files[-1].read_text()), indent=2))
"

Self-check: you can state the upstream job’s name, its run-id, and the timestamp at which it completed. If you used Marquez, include the dataset-lineage graph description (even as a text description of what you saw on screen).

View solutions →