Airflow Basics

Lesson 04 · Data Engineering with Python · ~12 min read

1. From cron to an orchestrator

Every data pipeline eventually outgrows a single cron line. At first, the growth is invisible: one script runs at midnight, writes a file, and that's enough. Then a second script appears that depends on the first. You manage it with a shell wrapper — if the first file exists, run the second. Then a third script, a fourth. Suddenly you have a chain of shell scripts that nobody fully understands, where one silent failure mid-chain means all downstream jobs run on stale data without anyone knowing.

The failure modes that cron cannot address fall into four categories. Dependencies: cron knows nothing about whether an upstream job succeeded; you must encode "wait for file X" manually, and getting that wrong causes races. Partial-failure recovery: if step 3 of 7 fails, cron has no notion of which steps to re-run and which to skip; you often re-run everything and waste compute. Observability: determining whether last night's pipeline ran successfully requires grepping log files or checking output timestamps — there is no dashboard. Backfill: re-running 30 days of historical data without disturbing today's scheduled run is genuinely difficult with raw cron.

An orchestrator solves exactly these four problems. It tracks task dependencies and only runs a task when its upstream tasks have succeeded. It retries failed tasks automatically with configurable backoff. It provides a UI where you can see the state of every task in every past run at a glance. And it supports backfill — explicitly asking it to replay a date range — as a first-class operation. Apache Airflow is the dominant open-source orchestrator for batch data pipelines, and it is what the rest of Module 2 uses.

2. Airflow's core abstractions

Airflow has three core abstractions. Understanding them cleanly — and not conflating them — will save you hours of debugging.

A DAG (Directed Acyclic Graph) is a Python file that declares a workflow: the set of tasks and the dependency edges between them. It is acyclic because a task can never depend on itself or on a task that (transitively) depends on it — otherwise the scheduler would loop forever. A DAG does not contain the task logic itself; it is a configuration object that tells the scheduler what to run and in what order.

A Task is one node in the DAG — one unit of work. Each time the DAG runs (each DagRun), one task instance is created for each Task. Task instances carry state: they start as none, move to scheduled, then queued, then running, and finally to success or failed. The history of every task instance is stored in Airflow's metadata database and visible in the Grid view.

An Operator is the type of task — it defines what code runs when the task executes. PythonOperator calls a Python callable. BashOperator runs a shell command. BranchPythonOperator decides at runtime which downstream branch to follow. There are hundreds of community operators (S3, Snowflake, dbt, Spark, etc.) available in apache-airflow-providers-* packages. In this course we use primarily PythonOperator.

The key relationship: a DAG is a container for Tasks; each Task is instantiated from an Operator; each DagRun spawns one task instance per Task. Operators define what to run; the DAG defines when (the schedule and dependencies); Airflow's scheduler coordinates execution.

3. The Airflow runtime stack

Airflow's runtime consists of four components that all share a single metadata database. Understanding which component does what makes it much easier to diagnose problems.

The Scheduler is the core loop. Every few seconds it reads all DAG files, evaluates which DagRuns should exist (based on the schedule and start date), queues task instances whose dependencies are satisfied, and sends them to the executor. The scheduler is stateless with respect to execution — it writes decisions into the metadata DB; the executor picks them up.

The Executor is responsible for actually running task instances. For a single-machine development setup, LocalExecutor runs tasks as subprocesses on the same machine. For production workloads, CeleryExecutor distributes tasks across a pool of worker machines, and KubernetesExecutor spins up a fresh pod per task. Changing executors is a config-file change — the DAG code does not need to change.

The Webserver is a Flask application that serves the Airflow UI on port 8080. It reads run history and task state from the metadata DB and streams task logs from wherever the executor wrote them (local filesystem for LocalExecutor, object storage for cloud executors).

The Metadata DB (Postgres or MySQL) is the source of truth for every DAG definition, DagRun, task instance state, variable, connection, and pool. All four other components read and write through it. In the course's Docker Compose setup, Postgres runs in a companion container and its data volume persists across docker compose down restarts.

Here is the complete DAG file state after this lesson:

from airflow import DAG
from airflow.operators.python import PythonOperator
from datetime import datetime

import sys, pathlib
sys.path.insert(0, str(pathlib.Path(__file__).parent.parent.parent))  # find sample-pipeline source

def _extract(**context):
    from extract.gharchive import fetch_hour
    fetch_hour(hour="2026-05-01-12")  # hardcoded for L04; parameterized in L05

def _transform(**context):
    from transform.events import count_by_type
    count_by_type(
        input_path="data/raw/2026-05-01-12.json.gz",
        output_path="data/out/2026-05-01-12.csv",
    )

with DAG(
    dag_id="gharchive_hourly",
    start_date=datetime(2026, 5, 1),
    schedule=None,
    catchup=False,
    tags=["data-engineering", "module-2"],
) as dag:
    extract = PythonOperator(task_id="extract", python_callable=_extract)
    transform = PythonOperator(task_id="transform", python_callable=_transform)
    extract >> transform

4. The DAG file lifecycle

Understanding how Airflow discovers and re-parses DAG files is essential for keeping the scheduler healthy. The scheduler polls the dags/ directory every dag_dir_list_interval seconds (default: 30s). For each Python file it finds, it imports the file and records any DAG objects it discovers. This process repeats on every heartbeat — which means your DAG file is imported roughly twice per minute around the clock.

The practical implication: imports must be fast. Airflow enforces a dagbag_import_timeout (default: 30s per file). If your file takes longer than that to import — because it does a database query, an HTTP request, or a heavy model load at module scope — Airflow marks it as an import error and the DAG disappears from the UI.

Side-effects at import time are bad. If you write code at module scope that fires an HTTP request or writes to disk, that code runs every 30 seconds on the scheduler — not just when the task runs. The correct pattern is to define functions and reference them in PythonOperator(python_callable=fn). The function body only runs when the task instance executes, not when the file is parsed.

Also notice the deferred import pattern in the DAG above: from extract.gharchive import fetch_hour appears inside _extract rather than at the top of the file. This means the gharchive module (and its transitive dependencies, including requests) is only loaded when the task actually runs — keeping parse time under a millisecond.

5. The task instance lifecycle

A task instance progresses through a well-defined set of states, all stored in the metadata DB. Understanding the state machine helps you interpret the Grid view and decide what action to take when something goes wrong.

The happy path: none (task exists in DB but hasn't been evaluated) → scheduled (scheduler decided this task should run and its upstream is satisfied) → queued (handed to the executor, waiting for a worker slot) → running (the callable is executing) → success (callable returned without raising).

On failure: the task moves to failed. If retries > 0, Airflow transitions it to up_for_retry, waits retry_delay (default: 5 minutes), then moves it back to scheduled. This cycle repeats until either the task succeeds or the retry budget is exhausted. Logs are stored per task instance per attempt — attempt 1, attempt 2, etc. — so you can compare what changed between runs.

A task can also reach skipped: this happens when a BranchPythonOperator chose a different branch, or when an upstream task failed and the downstream task's trigger rule (all_success by default) was not satisfied.

The Grid view maps these states to colours: grey = none/scheduled, light blue = queued, yellow = running, dark green = success, red = failed, orange = up_for_retry, light grey = skipped. One cell per task per run.

6. Web UI tour

The Airflow UI has five panels you will use constantly during development:

Grid view (default) shows a matrix: rows are tasks, columns are DagRuns ordered by execution date. Each cell is a coloured square representing one task instance. Hover for details; click to open the task instance panel with logs, duration, and the option to clear (re-run) the task.

Graph view shows the DAG topology as a directed graph. For a two-task DAG like ours it is trivial, but for production DAGs with 50+ tasks this view is indispensable for understanding the dependency structure.

Task Logs are accessible from the task instance panel. Airflow captures stdout and stderr from the callable, prefixing each line with a timestamp and the log level. You can page through multiple attempts (attempt 1, attempt 2, ...) and download the log as a text file.

Code shows the actual Python source that the scheduler parsed. This is the canonical answer to "what DAG does Airflow think is running?" — useful when your file on disk doesn't match what the scheduler loaded (e.g., because you edited the file while Airflow was mid-parse).

Trigger DAG opens a modal where you can start a manual run and optionally pass a JSON configuration payload. The JSON lands in context["params"] inside the callable — a simple way to parameterise a DAG run without changing the code.

7. Common pitfalls

References: Apache Airflow stable docs · Docker Compose quickstart · PythonOperator reference · Airflow REST API (trigger DAGs from CI)