Apache Parquet Format

Parquet is the default storage format of the modern data lake: Spark, Trino, DuckDB, Snowflake external tables, Delta Lake and Iceberg all read it natively. This page explains the file layout that makes it fast, shows how readers skip data, and gives runnable PyArrow and DuckDB examples.

Apache Parquet is a columnar storage file format optimized for use with data processing systems like Apache Hadoop, Apache Spark, and cloud-based data lakes. It is highly efficient for large-scale data storage and retrieval, especially for analytic workloads.

Key Features of Parquet:

Columnar Storage: Parquet stores data in a columnar format, which makes it ideal for analytical queries that only need to access specific columns of data. This reduces I/O by minimizing the amount of data read.
Efficient Compression: Parquet applies compression techniques like dictionary encoding, run-length encoding (RLE), and bit-packing to compress data efficiently, leading to lower storage costs and faster query performance.
Schema Evolution: Parquet supports schema evolution, allowing changes to the schema of the data over time, such as adding new columns, without breaking compatibility with existing data.
Optimized for Analytics: The columnar format makes it well-suited for OLAP workloads and big data analytics, where aggregating, filtering, and selecting specific columns of data are common operations.
Interoperability: Parquet is supported by many big data processing engines such as Apache Hive, Apache Impala, Apache Spark, and cloud services like AWS Redshift, Azure Synapse, and Google BigQuery.
Splitting and Parallelism: Parquet files can be split into multiple parts, allowing large files to be processed in parallel across a distributed system for improved performance.

Benefits of Using Parquet:

Reduced Storage Costs: Parquet's efficient compression reduces the size of stored data, cutting down on storage expenses.
Faster Query Performance: Columnar storage and compression allow queries to run faster by reading only the necessary columns of data.
Scalability: Parquet files are designed to scale easily in distributed environments, making them a go-to format for large-scale data processing systems.
Compatibility: Parquet's wide support across multiple platforms and data processing frameworks ensures easy integration into modern data architectures.

Use Cases:

Big data analytics on massive datasets that require efficient storage and fast query performance.
Data warehousing environments that need optimized storage and access for analytical queries.
ETL pipelines where data is ingested, transformed, and processed across distributed systems.

1. File layout

A Parquet file is split horizontally into row groups, and each row group stores every column contiguously as a column chunk. Column chunks are divided into pages, the unit of encoding and compression. The footer at the end of the file holds the schema and, for each column chunk, statistics such as min, max and null count. Because the footer is at the end, writers can stream data and only finalize metadata when they close the file.

Within a page, values are encoded before compression. Dictionary encoding replaces repeated strings with small integer ids, run-length and bit-packing encodings shrink low-cardinality and boolean columns, and delta encodings help sorted integers and timestamps. Compression codecs (Snappy, ZSTD, GZIP) then apply on top.

2. How readers skip data

Columnar layout gives projection pushdown: a query that touches two columns of a 100-column table reads roughly 2 percent of the bytes. Footer statistics give predicate pushdown: if a row group's max order_date is earlier than the filter, the reader skips it without decompressing anything. Pruning works best when data is sorted or clustered on the filter column, so that each row group covers a narrow range of values.

3. Worked example

import pyarrow as pa
import pyarrow.parquet as pq
import pyarrow.dataset as ds

table = pa.table({
    "region": ["us", "eu", "us", "apac"] * 250_000,
    "amount": list(range(1_000_000)),
})
# Write partitioned by region, ZSTD compression, ~100k rows per row group
pq.write_to_dataset(table, root_path="sales", partition_cols=["region"],
                    compression="zstd", row_group_size=100_000)

# Read only one column, only the "us" partition, only rows matching a filter
dataset = ds.dataset("sales", format="parquet", partitioning="hive")
us = dataset.to_table(columns=["amount"],
                      filter=(ds.field("region") == "us") & (ds.field("amount") > 900_000))
print(us.num_rows)

# Inspect footer metadata
f = pq.ParquetFile(next(iter(dataset.files)))
print(f.metadata.num_row_groups, f.metadata.row_group(0).column(0).statistics)

-- DuckDB queries the same files directly
SELECT region, sum(amount)
FROM read_parquet('sales/*/*.parquet', hive_partitioning = true)
GROUP BY region;

4. Parquet versus other formats

Format	Layout	Best for
Parquet	Columnar	Analytics, wide tables, data lakes
ORC	Columnar	Hive-centric stacks
Avro	Row	Kafka messages, write-heavy ingestion, schema evolution
CSV / JSON	Row, text	Interchange, small files, human inspection

Parquet files are immutable. Updates, deletes and ACID transactions come from a table format layered on top, such as Delta Lake or Apache Iceberg.

5. Pitfalls

Small files. Thousands of kilobyte-sized files make footer reads dominate; compact to roughly 128 MB to 1 GB.
Over-partitioning. Partitioning on a high-cardinality column creates the small-file problem by design.
Unsorted data. Random order makes min/max statistics useless for pruning.
Type mismatches across files (int32 versus int64, timestamp units) break schema merging.