Lesson 12 Observability & SLAs
Theme:

Observability & SLAs

Lesson 12 · Data Engineering with Python · ~15 min read

1. Why observability comes before optimization

A pipeline that runs silently is not a healthy pipeline — it is an unobserved one. Before you optimise query costs, evolve schemas, or add access controls, you need the answer to a simpler question: is the data in the table the right data, and is it there when it is supposed to be? Without observability you cannot answer either half.

Barr Moses’s “five pillars of data observability” framework (freshness, volume, schema, distribution, lineage) and the LinkedIn DataHub paper both converge on the same insight: the majority of production data incidents are not caused by job failures, which are loud and self-announcing, but by silent failures — jobs that succeed but load wrong, stale, or incomplete data. An Airflow task that turns green after writing zero rows is a silent failure.

Industry pull-quote worth remembering: “A pipeline is in production the day it has a freshness SLA — not the day it has a DAG.” The DAG tells the scheduler what to run. The SLA tells the data team what “healthy” means. You cannot have the second conversation without the first.

Observability is placed first in Module 5 because every subsequent lesson — schema evolution, cost attribution, access controls — relies on it. You cannot detect a schema change you did not instrument. You cannot attribute cost to a pipeline run you did not trace. Instrument first; optimise second.

2. The four pillars (freshness / volume / schema / distribution)

The four pillars that cover the vast majority of production data quality failures are freshness, volume, schema, and distribution. Each has a canonical numeric signal that can be checked automatically without human intervention.

Freshness measures how recent the newest data in a table is. The canonical signal is now − max(created_at): the age of the most-recent event. For a table receiving hourly data, an age of 3 hours is a breach; an age of 45 minutes is normal. Freshness is the most actionable pillar because it has a natural threshold derived from the source’s publish cadence. This lesson builds the freshness check end-to-end.

Volume measures whether the expected number of rows is arriving. The canonical signal is the row count delta: how many rows landed in the latest hour compared to the trailing 24-hour average. A drop to 10% of the average means either the source is down or the extract is broken. This is covered as Task 1 in the homework.

Schema measures whether the shape of the table has changed unexpectedly. The canonical signal is a hash of the column set: frozenset(table.columns), serialised and compared to the previous run. A mismatch means a new column appeared or an existing one was renamed or dropped. This is the subject of Lesson 13.

Distribution measures whether values are in expected ranges. The canonical signals are null rate (what fraction of a non-nullable column is null?) and top-N value share (if the actor column’s top value accounts for 70% of all rows today when it normally accounts for 0.5%, a bot run or a data-load error has changed the data). Distribution checks belong to Great Expectations suites; they are set up in Lesson 03 and refined in Lesson 13.

3. OpenLineage as the standard event format

Before OpenLineage (openlineage.io, adopted by the Linux Foundation in 2021), lineage in a multi-tool pipeline was assembled by hand: scrape Airflow’s task logs, parse dbt’s manifest.json, query the data warehouse for table dependencies. The result was always incomplete and always out of date.

OpenLineage solves this with a single event envelope that every tool emits in the same JSON shape. The envelope fields are:

A rendered START event from the sample pipeline extract task looks like this: 

{
  "eventType": "START",
  "eventTime": "2026-05-26T10:00:00.000000+00:00",
  "producer": "https://github.com/dmitry-kharytonov/claude-lessons/data-engineering",
  "job": {
    "namespace": "gharchive-pipeline",
    "name": "gharchive_hourly.extract"
  },
  "run": {"runId": "4b3a2c1d-8e6f-4a1b-b2c3-d4e5f6a7b8c9"},
  "inputs":  [{"namespace": "gharchive-pipeline", "name": "raw_events"}],
  "outputs": [{"namespace": "gharchive-pipeline", "name": "dim_events"}]
}

When the task completes, a matching COMPLETE event arrives with the same runId. Marquez joins them by runId to build the run timeline. If a COMPLETE event never arrives (the process was killed), the run is recorded as failed — which is exactly the right signal.

4. Marquez as the OpenLineage backend

Marquez (marquezproject.github.io) is the reference implementation of an OpenLineage backend: a REST API that receives events, stores them in Postgres, and derives a dataset-and-job graph that the web UI renders. It is open source, actively maintained, and small enough to run in a single docker compose up on a laptop.

What Marquez stores: every event that arrives at its POST /api/v1/lineage endpoint, plus the derived graph of jobs, runs, and datasets. The web UI at port 3000 renders three panels: the Jobs panel (every emitting job), the Runs panel (the timeline for a selected job), and the Datasets panel (the lineage graph). The Datasets graph is the key operational artefact: a directed acyclic graph where a node is a table and an edge is a job that reads or writes it.

Why Marquez over a custom Postgres table? The free lineage UI; the OpenLineage-native storage schema (no ETL from events to graph); and the HTTP API that lets you query the graph programmatically (useful for impact analysis: “which downstream dashboards are affected if I rename this column?”). For a team that already runs DataHub, Atlas, or another metadata platform, OpenLineage events from the pipeline flow to whichever backend is already deployed.

5. Freshness SLAs in practice

An SLA is a commitment with a consequence. “The events table will not be older than 90 minutes” is an SLA; “near real-time” is not. The first can be tested automatically; the second cannot.

The formula for deriving a freshness SLA for a batch pipeline:

SLA = source publish cadence + pipeline run cadence + processing time + buffer

For the gharchive pipeline: source publishes hourly (60 min), DAG is scheduled hourly (60 min), extract + transform + load takes roughly 5 minutes, buffer for retries and late data is 25 minutes. Worst-case age = 150 minutes; SLA = 90 minutes (a breach at 90 minutes means you have at least 60 minutes to investigate before the worst case is reached).

One common pitfall: SLAs that apply only during business hours. A Sunday-morning gharchive outage that starts at 02:00 UTC and is not caught until Monday at 09:00 is a 31-hour data gap. If the SLA only fires alerts during business hours, on-call never sees it until after the data is seven hours old and stakeholders have already noticed. Freshness checks must run 24/7, unconditionally. Use a single SLA per critical table; do not sub-divide by partition or day-of-week. The uniformity makes the SLA predictable and its breach page-worthy rather than ignorable.

6. Alert design

Three rules for alerts that data teams actually act on:

Rule 1: every alert names a runbook URL. An alert without a runbook trains the on-call engineer to ignore it. A runbook does not need to be long — three bullet points (check the Airflow DAG, check the source, check the table row count) is enough. The discipline of writing the runbook before the alert fires is how you discover whether the alert is actionable.

Rule 2: alerts go to a channel a human reads. A #data-alerts Slack channel that receives 40 alerts per day is not a channel anyone reads. Route SLA breach alerts directly to the on-call rotation. Route volume-drop alerts to the data team’s primary channel. An alert that fires into the void is not an alert; it is a log line.

Rule 3: freshness alerts must hold-down. If the data source publishes a batch five minutes late, the freshness check will fire once and then clear on the next run. Without a hold-down (a minimum breach duration before the alert is sent), a brief source delay pages someone every hour. A five-minute hold-down eliminates most transient false positives while still catching real outages within ten minutes.

7. Common pitfalls

Common pitfalls

  1. Treating “DAG succeeded” as “pipeline succeeded.” The load step can return zero rows and Airflow marks the task green. A green DAG is a necessary condition for a healthy pipeline, not a sufficient one. Always add a row-count assertion or a freshness check as the final task.
  2. Setting freshness SLAs in adjectives instead of minutes. “Near real-time,” “daily,” and “timely” cannot be automated. The moment you cannot write if age > N: alert(), the SLA is not operational. Always express freshness SLAs as a number of minutes.
  3. Emitting OpenLineage events from a try/except that swallows the failure event. A pattern like try: do_work(); emit(COMPLETE) except: pass means that when do_work() raises, the COMPLETE event is never emitted and Marquez records the run as still in-flight indefinitely. The correct pattern: emit FAIL in the except block, then re-raise. Failed runs vanish from Marquez only when the FAIL event is swallowed.
  4. Building dashboards no one looks at and forgetting alerts. Dashboards are forensics: you review them after an alert fires to understand what happened. They do not substitute for alerts. A beautiful Grafana dashboard that no one checks is not observability; it is decoration. Instrument alerts first; add dashboards to help explain breaches.

References