DuckLake

DuckLake is a 2025 lakehouse format from the DuckDB team. Its central design choice is unusual: instead of storing table metadata as JSON / Avro files in the data directory (the Iceberg / Delta / Hudi pattern), DuckLake puts the catalog metadata in a regular SQL database (any Postgres, MySQL, SQLite, or DuckDB). Data files remain Parquet on object storage. The result is a single small binary plus a Postgres — no HMS, no REST catalog server, no JVM — with full ACID semantics on the lakehouse.

A lakehouse table format has two jobs: store data files in open formats on cheap object storage, and track which files make up each version of each table so readers see consistent snapshots and writers can commit atomically. Iceberg, Delta Lake, and Hudi do the second job with metadata files written next to the data. DuckLake does it with ordinary tables in a transactional SQL database, which is the idea the rest of this page examines.

Architecture

DuckLake: SQL catalog plus Parquet on object storage Engines DuckDB laptop ducklake extension DuckDB in a job same catalog Other readers via open spec Catalog database DuckDB file, SQLite, PostgreSQL or MySQL: snapshots, schemas, file lists, statistics Object storage Parquet data files and delete files on S3, GCS, R2, Azure or local disk Only the catalog is transactional; data files are immutable and written before the catalog commit references them.

The catalog database holds tables describing snapshots, schemas, tables, columns, data files, delete files, and per-file column statistics. A query asks the catalog which Parquet files belong to the table at the requested snapshot, using the statistics to prune files, then reads only those files from storage. Several DuckDB processes can share a lake when the catalog is a client-server database such as PostgreSQL; with a local DuckDB file as the catalog, the lake is effectively single-writer.

Key Features:

Why It’s Notable:

The big three open table formats (Iceberg, Hudi, Delta) reinvented database internals on top of object storage — transaction logs, optimistic concurrency, snapshot isolation, all hand-built. DuckLake takes the opposite philosophical approach: object storage holds the data, a real database holds the metadata, and you compose two well-understood systems instead of building a new one. It’s the simplest lakehouse design currently shipping.

Trade-offs:

Use Cases:

DuckLake is brand new — treat it as a credible architectural option to track, not yet as a default production choice.

How a commit works

Writing to a DuckLake table Write Parquet new immutable data files BEGIN transaction in catalog DB Insert rows data_file entries, new snapshot id COMMIT atomic across all tables Readers see new snapshot; old ones remain If the commit fails, the new files are simply unreferenced and can be cleaned up later; readers never see partial writes.

In Iceberg, a commit writes a new metadata file and atomically swaps a pointer in a catalog, retrying when another writer won the race; multi-table atomicity needs extra protocol. In DuckLake the commit is one SQL transaction, so changes to several tables, plus schema changes, either all appear or none do, and conflict detection is handled by the catalog database. Older snapshots stay queryable until they are expired, which provides time travel.

Worked example with DuckDB

INSTALL ducklake;
LOAD ducklake;

-- Catalog in a local DuckDB file, data files in a local folder (could be s3://bucket/path/)
ATTACH 'ducklake:metadata.ducklake' AS lake (DATA_PATH 'lake_data/');
USE lake;

CREATE TABLE trips (trip_id INTEGER, city VARCHAR, fare DECIMAL(8,2));
INSERT INTO trips VALUES (1, 'Oslo', 21.50), (2, 'Lima', 9.75);
UPDATE trips SET fare = 10.25 WHERE trip_id = 2;
ALTER TABLE trips ADD COLUMN tip DECIMAL(8,2);

-- List snapshots: 0 = empty catalog, 1 = CREATE, 2 = INSERT, 3 = UPDATE, 4 = ALTER
SELECT * FROM ducklake_snapshots('lake');
-- Read the table as it was before the UPDATE (Lima fare still 9.75)
SELECT * FROM trips AT (VERSION => 2);

-- Shared catalog for several writers: point the catalog at PostgreSQL instead
-- ATTACH 'ducklake:postgres:dbname=lake_catalog host=db.internal' AS lake
--     (DATA_PATH 's3://my-bucket/lake/');

Each statement above that changes data or schema creates a snapshot (verified with DuckDB 1.5.6 and the ducklake extension: attaching creates snapshot 0, and the four statements create snapshots 1–4). Because the catalog is a database, it can also be inspected directly with SQL, for example by attaching the metadata file and selecting from its ducklake_data_file table to see which Parquet files belong to which table.

DuckLake vs Iceberg and Delta

AspectDuckLakeApache IcebergDelta Lake
Metadata storageTables in a SQL databaseManifest and metadata files + catalogJSON log + checkpoints in _delta_log
Multi-table transactionsYes, one SQL transactionCatalog dependentNot natively
Engine supportMainly DuckDB so farVery broad (Spark, Trino, Flink, Snowflake, ...)Broad, strongest on Spark and Databricks
Operational footprintA database you may already runCatalog service (REST, Glue, Polaris, ...)None beyond storage, or Unity Catalog
MaturityNew (2025)MatureMature

Pitfalls