The file-level table layers of the lakehouse: Iceberg, Delta Lake, Hudi, Paimon, DuckLake, Lance and XTable.
Apache Paimon (formerly Flink Table Store) is a streaming-first open table format that originated in the Apache Flink community and graduated to a top-level…
Apache XTable (formerly OneTable, originated at Onehouse) is an open-source interoperability layer that translates table metadata between Apache Hudi, Apache…
Apache Hudi (Hadoop Upserts Deletes and Incrementals) is an open-source data management framework that simplifies large-scale data ingestion and provides…
Apache Iceberg is a modern data management framework built to support large-scale data lakes with ACID transactions and flexible partitioning, making it…
Delta Lake is the third major open table format alongside Apache Hudi and Apache Iceberg. Originated at Databricks in 2019 and donated to the Linux…
DuckLake is a 2025 lakehouse format from the DuckDB team. Its central design choice is unusual: instead of storing table metadata as JSON / Avro files in the…
Lance is an open-source columnar data format optimized for machine-learning and AI workloads. Built by LanceDB, Lance is positioned as a successor to Parquet…
A data lake on its own is a directory of Parquet or ORC files. Engines that read it have to list directories, guess the schema and hope nobody is writing at the same time. An open table format adds a metadata layer that records exactly which files make up each version of the table, along with schema, partitioning and column statistics. Writers commit by atomically swapping a pointer to new metadata, which gives ACID transactions, snapshot isolation for readers, time travel to earlier versions, schema evolution and file-level pruning. The catalog stores that pointer and lets many engines share one copy of the data.
| Need | Good fit |
|---|---|
| Multi-engine analytics, vendor neutrality | Iceberg |
| Databricks or Spark-centred platform | Delta Lake |
| Frequent upserts and change-data-capture ingestion | Hudi or Paimon |
| Streaming-first pipelines on Flink | Paimon |
| Small-to-medium lakehouse with simple operations | DuckLake |
| Embeddings, images, ML training data | Lance |
| Existing tables in one format, readers needing another | XTable |
The major formats have converged on features, so the decision usually follows the engines and catalog you already run. Interoperability efforts such as XTable and Delta UniForm reduce the cost of choosing wrong.
Table formats depend on a catalog; see Catalogs for Unity Catalog, Project Nessie and Apache Polaris, and Query Engines for Trino and StarRocks.