Lesson 10 · Slide 1

Lesson 10 — Iceberg Tables

Add a table format to your data lake — and get snapshots, time travel, and ACID for free.

Data Engineering Module 4 Modern Lakehouse

Lesson 10 · Slide 2

The problem L09 left open

What L09 gave us

DuckDB queried our Parquet directly — fast, zero-config, no copy step. The same SQL worked in CLI scripts, in tests, and inside Airflow tasks.

What we’re still missing

What’s the schema of events? What if two writers append at once? What did the data look like yesterday? Plain Parquet on disk has no answers to any of these.

Lesson 10 · Slide 3

File format vs table format

	File format	Table format
Examples	Parquet, ORC, Avro	Iceberg, Delta Lake, Hudi
Stores	Bytes (one file)	A set of files + metadata + history
Knows the schema?	One file at a time	Across the whole table
Snapshots / time travel	No	Yes
ACID writes	No	Yes
Many engines read it?	Yes	Yes (decouples engine from format)

The Iceberg pitch: keep your Parquet files, gain a metadata layer on top.

Lesson 10 · Slide 4

What Iceberg is

An open-spec table format — created at Netflix, donated to Apache in 2018.
Metadata that describes which Parquet files belong to which version of the table, what their schemas were, and how partitioning evolved over time.
Implementations in Java, Python (pyiceberg), Rust, and Go — engines choose their native client.
Engines that read Iceberg today: DuckDB, Spark, Trino, Flink, Snowflake, BigQuery — all using the same metadata files.

Lesson 10 · Slide 5

The Iceberg directory layout

warehouse/default/events/
  data/
    event_date=2026-05-26/
      00000-0-<uuid>.parquet
      00000-1-<uuid>.parquet
    event_date=2026-05-27/
      ...
  metadata/
    v1.metadata.json    ← table-level metadata (schemas, snapshots)
    v2.metadata.json    ← new version after first append
    snap-<id>-1.avro    ← snapshot manifest (lists data files)
    <uuid>.manifest.avro ← per-file stats (min/max per column)

The data/ half is the Parquet you already know. The metadata/ half is what Iceberg adds.

Lesson 10 · Slide 6

Snapshots

Every write to an Iceberg table produces a new immutable snapshot — a pointer to the exact set of data files that constitute the table at that instant.

Past snapshots remain on disk until you expire_snapshots().
Time travel is a metadata lookup, not a data copy — just read an older metadata.json.
Each snapshot also records the table’s schema at that moment, so you can read old data with the schema that was current then.

Append → new snapshot. Overwrite → new snapshot. Delete → new snapshot. The table history is a linked list of snapshots.

Lesson 10 · Slide 7

Time travel in 3 lines

# Current state
arrow_curr = table.scan().to_arrow()

# State as of an earlier snapshot (pyiceberg API)
arrow_prev = table.scan(snapshot_id=12345).to_arrow()

# Or via DuckDB by pointing at an older metadata.json
# SELECT count(*) FROM iceberg_scan('warehouse/.../v3.metadata.json');

Every snapshot points to a metadata.json. Reading an older JSON reads the table as of that point — no backup tape, no restore step.

Lesson 10 · Slide 8

ACID on a data lake

Optimistic concurrency control. Each writer reads the current v<N>.metadata.json, composes a new v<N+1>.metadata.json referencing new data files, then asks the catalog to swap the pointer atomically.
If two writers race, one wins; the loser retries against the updated head. No coordinator service required — the catalog (SQLite, Postgres, Glue) is the only serialisation point.
Data and metadata files are immutable once written — this is what makes Iceberg run safely on cheap object storage (S3, GCS) without a distributed lock manager.

Lesson 10 · Slide 9

Schema evolution

Iceberg tracks columns by stable numeric field IDs, not names.
Rename a column — old snapshots still resolve because the field ID is unchanged.
Add a column — old snapshots return NULL for the new column.
Drop a column — old snapshots still expose its data.

table.update_schema().add_column("commit_count", LongType()).commit()
# Rename without rewriting any data files:
table.update_schema().rename_column("actor", "actor_username").commit()

Backward-compatible by design — no file rewrite needed for any of these operations.

Lesson 10 · Slide 10

Partition evolution

Change the partition spec without rewriting old files.
Yesterday’s data stays under event_date=2026-05-26/; tomorrow’s might sit under event_date=2026-05-27/event_hour=14/ after you add an hourly transform.
The metadata tracks which partition spec applied to which snapshot. Engines handle the mixed layout transparently — no manual SQL gymnastics.
Contrast with Hive-style Parquet: changing the partition layout means rewriting or manually reorganising every existing file.

Lesson 10 · Slide 11

The Iceberg + DuckDB local stack

Write path (pyiceberg)

Catalog: SqlCatalog backed by a SQLite file.
Warehouse: a local directory.
Writer: write_from_parquet.py reads L02 Parquet → appends to the Iceberg table. One append = one snapshot.

Read path (DuckDB)

INSTALL iceberg; LOAD iceberg;
SELECT * FROM iceberg_scan('…metadata.json');
No JVM, no Spark, no Hadoop. The whole stack fits on a laptop.

Lesson 10 · Slide 12

Maintenance

expire_snapshots() — deletes data files no longer referenced by any retained snapshot. Frees disk space. Wire into a weekly maintenance task.
rewrite_data_files() — compacts many small files into fewer large ones. One Airflow run per partition produces one small Parquet file; compact periodically to keep scan latency low.

Expiring snapshots is irreversible — once a snapshot is expired, its data files are subject to deletion and time travel to that point is gone. Test the retention policy on a dev table first. Default retention: at least as long as your longest backfill window.

Lesson 10 · Slide 13

What Iceberg is not

Not a query engine. DuckDB, Spark, Trino still do the actual reads — Iceberg is the metadata layer they all share.
Not a compute layer. No autoscaling, no job scheduling, no distributed execution. It’s a table format, not a cloud service.
Not opinionated about storage location. Local disk, S3, GCS, Azure ADLS — same metadata format everywhere. L11 swaps the SQLite catalog + local directory for a cloud catalog + object storage bucket without changing a single SQL query.

Lesson 10 · Slide 14

Recap & what's next

Table format adds schema, snapshots, and ACID over your existing Parquet — no data rewrite required.
DuckDB reads Iceberg via the iceberg extension — same SQL surface as L09, richer table contract underneath.
Maintenance = expire + compact — snapshot expiration frees space; file compaction keeps scans fast.

Coming up: Lesson 11 — lift the lakehouse into Snowflake (or BigQuery): external tables on Iceberg, cost models, materialise-vs-query. With a DuckDB-only fallback for readers without a paid cloud account.