Why CSV was the wrong answer — and what to use instead.
Data Engineering Module 1[r1_c1, r1_c2, r1_c3]
[r2_c1, r2_c2, r2_c3]
[r3_c1, r3_c2, r3_c3]
...Each row is stored together. Reading one column means reading all of them.
[r1_c1, r2_c1, r3_c1]
[r1_c2, r2_c2, r3_c2]
[r1_c3, r2_c3, r3_c3]
...Each column is stored together. Reading one column skips all the others.
A file format developed at Twitter / Cloudera in 2013; an Apache project since 2015. It is the de facto storage layer of the modern lakehouse.
A Parquet file contains:
Arrow is the columnar in-memory format. Parquet is the columnar on-disk format. They are designed to interoperate — reading a Parquet file produces an Arrow table in memory with near-zero copy overhead.
pyarrow is the Python implementation of both. When you call
pq.read_table(), you get an Arrow table back. When you call
pq.write_table(), Arrow columns are serialised to Parquet on disk.
You rarely need to think about Arrow directly — but knowing it exists explains why DuckDB, Pandas, Polars, and Spark all read Parquet without an intermediate format conversion.
import pyarrow as pa
import pyarrow.parquet as pq
import gzip, json, pathlib
def to_parquet(json_gz: pathlib.Path, parquet_out: pathlib.Path) -> None:
rows = []
with gzip.open(json_gz, "rt") as f:
for line in f:
ev = json.loads(line)
rows.append({
"type": ev["type"],
"actor": ev["actor"]["login"],
"repo": ev["repo"]["name"],
"created_at": ev["created_at"],
})
table = pa.Table.from_pylist(rows)
pq.write_table(table, parquet_out, compression="snappy")4 columns kept this time, not just type — and the file will still be smaller than the CSV.
Partitioning splits one logical table across many physical directories. Layout:
data/parquet/
date=2024-01-01/part-0.parquet
date=2024-01-02/part-0.parquet
...
Queries that filter on date only open the directories that match —
this is called partition pruning. A query for a single day touches 1/365 of
the data, not all of it.
| Format | Size on disk | Count PushEvents | Count by type |
|---|---|---|---|
| Raw JSON.gz | ~50 MB | ~2.5 s | ~2.5 s |
| CSV (L01) | ~3 MB | ~0.4 s | ~0.4 s |
| Parquet (snappy) | ~1.5 MB | ~0.05 s | ~0.05 s |
| Parquet, partitioned by type | ~1.5 MB | ~0.002 s | ~0.05 s |
Numbers are illustrative — run the lab and measure yours.
import pandas as pd
df = pd.read_parquet("data/parquet/date=2024-01-01/")
df["type"].value_counts().head(10)
read_parquet understands partitioned layouts natively — point it at the directory
and it discovers all the part files automatically.
pandas delegates to pyarrow under the hood. The pyarrow engine is the
default and the recommended choice; it handles all compression codecs and nested schemas.
SELECT type, COUNT(*)
FROM 'data/parquet/date=2024-01-01/*.parquet'
GROUP BY type;No database. No schema setup. No COPY command. Parquet on disk is the table. DuckDB reads it with full predicate pushdown and column pruning, then streams the result.
Parquet shines at the scale where you cared about format choice. Below ~10 MB, almost any format is fine.
compression='snappy' — uncompressed Parquet still beats CSV, but you're leaving half the win on the table.