Lesson 10 Iceberg Tables
Theme:

Iceberg Tables

Lesson 10 · Data Engineering with Python · ~15 min read

1. The gap Iceberg fills

Lesson 09 left the pipeline querying Parquet directly. That works for analysis: DuckDB reads the partition files in place, pushes predicates into row groups, and returns results in milliseconds. But it breaks down for operating a table over time.

Consider what plain Parquet on disk cannot answer: what is the canonical schema of the events directory today? Was it the same schema three weeks ago? Can two Airflow workers safely append to the same partition simultaneously without corrupting each other’s output? If a bad batch landed yesterday, how do you roll back without deleting and re-running everything downstream? None of these questions has a Parquet-native answer — Parquet is a file format, not a table format.

Iceberg is the metadata layer that closes those gaps. It sits on top of Parquet (and ORC, and Avro) without replacing it. The data bytes remain in standard Parquet files that any engine can read. Iceberg adds a metadata tree that tracks which files belong to which version of the table, what the schema was at each version, and how the partition layout has evolved. The result: your Parquet files gain a schema, a history, and ACID semantics — without moving a single byte of data.

2. File format vs table format

The distinction is worth internalising before going further. A file format like Parquet describes how bytes are laid out in a single file — column groups, row groups, compression codecs, dictionary encoding, and per-row-group statistics. Any engine that understands the spec can open one Parquet file and read it; that is the full extent of the contract.

A table format like Iceberg describes a set of files as a coherent table. It adds:

Engines such as DuckDB, Spark, Trino, Snowflake, and BigQuery all implement the Iceberg spec independently. They each read the same metadata files, then read the underlying Parquet files with their own execution engines. This decoupling is the key architectural advantage: you can swap the engine without migrating the data.

3. The directory layout

An Iceberg table stored locally (as our sample pipeline does) lives under a warehouse root directory. The layout for the default.events table is: 

warehouse/
  default/
    events/
      data/
        event_date=2026-05-26/
          00000-0-<uuid>.parquet
        event_date=2026-05-27/
          00000-0-<uuid>.parquet
      metadata/
        v1.metadata.json
        v2.metadata.json
        snap-<snapshot-id>-1.avro
        <manifest-uuid>.avro

The data/ subtree is exactly the Hive-partitioned Parquet layout you built in L02 (the partition column is event_date in the Iceberg-internal schema). The metadata/ subtree is new:

The catalog (a SQLite database in our local stack) holds a single pointer: the path to the current v<N>.metadata.json. Switching the table to a different version is an atomic pointer swap in the catalog — no data files move.

4. Snapshots and time travel

A snapshot is a single immutable view of the table — it lists exactly which Parquet files are part of the table at that instant. Each append, overwrite, or delete operation creates a new snapshot. Past snapshots remain on disk until explicitly expired.

Time travel is therefore a metadata lookup, not a data operation: 

table = catalog.load_table("default.events")

# Current state:
arrow_curr = table.scan().to_arrow()

# State as of snapshot id 12345:
arrow_prev = table.scan(snapshot_id=12345).to_arrow()

# Or from DuckDB by pointing at an older metadata.json:
# SELECT * FROM iceberg_scan('warehouse/default/events/metadata/v3.metadata.json');

Practical uses: backfill debugging (“what did daily_counts look like before yesterday’s broken DAG?”), regulatory audits (“show me the row state on 2026-01-31”), regression triage (“did the row count change between these two deployments?”). None of these require a separate backup. The snapshot list is the backup.

5. Schema evolution by field ID

Iceberg’s most important design choice for schema safety: columns are never identified by name in the metadata. Every column has a stable numeric field_id assigned at table creation time. The name is just a display alias; the field_id is the identity.

Consequences: 

# Add without rewriting any files:
table.update_schema().add_column("commit_count", LongType()).commit()

# Rename the actor column (pedagogical demo -- real column name is "actor"):
table.update_schema().rename_column("actor", "actor_username").commit()
# Reading old snapshots now returns actor_username;
# the field_id (3) is unchanged so every old data file resolves correctly.

This solves the most painful Parquet-on-disk migration pattern: needing to rewrite every file just because a stakeholder decided a column should have a different name.

6. ACID via optimistic concurrency

No daemon, no lock manager, no coordinator service. Each writer follows five steps:

  1. Read the current v<N>.metadata.json from the catalog to learn the current table state.
  2. Write new data files into data/ (immutable once written).
  3. Build a new manifest referencing those data files.
  4. Compose v<N+1>.metadata.json referencing the new manifest and pointing to the new snapshot.
  5. Ask the catalog to swap the pointer atomically and conditionally: “set pointer to v<N+1> if and only if it is still v<N>.”

If the swap fails (another writer raced and committed first), the loser discards its new metadata.json and retries from step 1 against the updated head. pyiceberg implements this retry loop internally up to a configurable number of attempts.

The catalog is the only component that needs to be transactional. Data and metadata files are immutable once written. This is why Iceberg runs safely on cheap object storage (S3, GCS) without requiring a distributed lock manager — the catalog can be a single SQLite file locally, or AWS Glue / Hive Metastore / Polaris in production.

7. The Iceberg + DuckDB local stack

The sample pipeline implements the smallest useful version of this architecture:

The whole stack fits on a laptop, runs with no JVM, and is the literal production architecture — just with the SQLite catalog and local directory replaced by cloud equivalents. Nothing changes about how you write the Python or SQL when you make that swap.

8. Common pitfalls

Common pitfalls

  1. Two SqlCatalog instances pointing at the same SQLite file from different processes. SQLite’s file lock conflicts, causing OperationalError: database is locked. Use one catalog per process; or move to a Postgres-backed catalog URI when you need real concurrent writers.
  2. Skipping expire_snapshots for months. The data/ directory grows without bound because old data files are never garbage-collected. Wire snapshot expiration into a weekly maintenance DAG.
  3. Too short a retention window on expire_snapshots. If your longest backfill window is 30 days but you expire after 7, a re-run of an old DAG reads a snapshot that no longer exists and fails. Set retention to at least your longest backfill window — when in doubt, use 30 days.
  4. Editing data files in data/ by hand. Iceberg owns the data layout; mutating files outside the table API corrupts the metadata invariants (file lists, per-file stats) and can produce silently wrong query results.
  5. Mixing engines that don’t share a catalog. Two writers pointed at different catalog URIs will never see each other’s snapshots. Every reader and writer must use the same catalog URI — confirm this when moving from local to cloud.

References