Homework 04 — Extending the first DAG
Lesson 04 · Data Engineering with Python · 3 tasks
These tasks extend what you built in the lab. You will add new operators to the existing DAG, explore the Airflow logs, and deepen your understanding of the scheduler's behaviour. All tasks assume Airflow is still running from the lab session.
Task 1 — Add a third task: summarize
Edit data-engineering/sample-pipeline/airflow/dags/gharchive_hourly.py to add a
third PythonOperator called summarize, to be run after
transform.
The _summarize callable should use pandas to read the Parquet file
written by the transform task and print the top 5 event types with their counts
to the task log.
Trigger a new manual run from the UI and verify the summarize task succeeds.
Open its Task Logs and confirm the top-5 table is printed.
✓ Self-check: the DAG now shows three tasks in the Graph view
(extract → transform → summarize); the summarize Task Logs
show the same top-5 event types you computed in the Lesson 01 homework.
Task 2 — Bash operator practice: cleanup
Add a fourth task cleanup (to run after summarize) using
BashOperator. The bash command should print the disk usage of the sample-pipeline
data directory:
du -sh /opt/airflow/sample-pipeline/data/
Note: BashOperator runs inside the Airflow container, so the path must use the
container's mount point (/opt/airflow/sample-pipeline/), not the host path.
Trigger another run and open the cleanup Task Logs.
✓ Self-check: the Task Logs for cleanup show a size
reading (e.g., 18M /opt/airflow/sample-pipeline/data/).
Task 3 — Read the parse log
Every time the Airflow scheduler re-parses your DAG file, it logs the import duration. Fetch the scheduler logs and find that line:
docker compose logs airflow-scheduler 2>&1 | grep "gharchive_hourly" | tail -5
Look for a line containing the word Processing or took. It will look
something like:
[2026-05-26 09:10:01] {dag_processing.py:712} INFO - Processing dags/gharchive_hourly.py took 0.34 secondsNote how long the import took. Production tip: DAG imports that take more than 2 seconds slow the scheduler heartbeat and can delay task launches across all DAGs in the deployment.
✓ Self-check: you can paste the import-time milliseconds from the log
(e.g., 0.34 seconds). If your DAG takes more than 1 second, check whether you have
heavy imports at module scope.