Apache Iceberg is a modern data management framework built to support large-scale data lakes with ACID transactions and flexible partitioning, making it highly scalable and efficient for managing complex datasets.
An Iceberg table is a tree of immutable files on object storage plus one mutable pointer in a catalog. The catalog (Hive Metastore, AWS Glue, a REST catalog such as Apache Polaris, Nessie or Unity Catalog, or a JDBC catalog) stores only the location of the current metadata.json. That file records the schema, partition specs and the list of snapshots. Each snapshot points to a manifest list, which points to manifest files, which list the actual data files together with per-column statistics such as row counts, null counts and min/max values.
A commit writes new data files and new metadata files, then atomically swaps the catalog pointer from the old metadata.json to the new one. If two writers race, the swap fails for one of them and that writer retries against the new state. This optimistic concurrency is what gives Iceberg serializable ACID commits on storage systems like S3 that have no transactions of their own.
Query engines never list directories. They load the current snapshot, skip whole manifests whose partition ranges cannot match the filter, then skip individual files whose column statistics rule them out. Only the remaining files are opened. Because partitioning is hidden, a filter on event_ts automatically uses a days(event_ts) partition transform; users do not need to know about or filter on a separate partition column, and the partition scheme can change later without rewriting old data.
CREATE TABLE lake.db.events (
event_id BIGINT,
user_id BIGINT,
event_ts TIMESTAMP,
kind STRING
) USING iceberg
PARTITIONED BY (days(event_ts));
INSERT INTO lake.db.events VALUES (1, 42, TIMESTAMP '2026-09-30 10:00:00', 'click');
-- row-level upsert
MERGE INTO lake.db.events t
USING updates u ON t.event_id = u.event_id
WHEN MATCHED THEN UPDATE SET *
WHEN NOT MATCHED THEN INSERT *;
-- schema and partition evolution, no rewrite
ALTER TABLE lake.db.events ADD COLUMN country STRING;
ALTER TABLE lake.db.events ADD PARTITION FIELD bucket(16, user_id);
-- time travel and history
SELECT * FROM lake.db.events.snapshots;
SELECT * FROM lake.db.events VERSION AS OF 1234567890123;
SELECT * FROM lake.db.events TIMESTAMP AS OF '2026-09-30 00:00:00';
-- maintenance
CALL lake.system.expire_snapshots(table => 'db.events', older_than => TIMESTAMP '2026-09-01 00:00:00');
CALL lake.system.rewrite_data_files(table => 'db.events');The catalog name lake is configured in Spark with spark.sql.catalog.lake=org.apache.iceberg.spark.SparkCatalog plus a catalog type or URI. Columns are tracked by ID rather than name, so renaming or reordering columns is safe.
| Format | Strength | Choose when |
|---|---|---|
| Apache Iceberg | Engine-neutral spec, hidden partitioning, partition evolution, REST catalog | Many engines (Spark, Trino, Flink, Snowflake, Athena, DuckDB) share one table |
| Delta Lake | Deep Databricks integration, transaction log in _delta_log | Primarily Databricks or Spark workloads |
| Apache Hudi | Record-level indexes, merge-on-read for streaming upserts | High-rate CDC ingestion |
| Apache Paimon | LSM-based streaming lake format | Flink-centric real-time tables |
| Apache XTable | Translates metadata between formats | Exposing one copy of data as several formats |
The formats have converged on features: all offer ACID commits, time travel and schema evolution, and Delta Lake can expose Iceberg-compatible metadata through UniForm. The practical decision is therefore mostly about ecosystem and governance. If your organisation runs several engines against the same data, or wants to avoid binding table metadata to a single vendor, Iceberg with a REST catalog is the most neutral option. If almost everything runs on Databricks, Delta Lake is the path of least resistance. Streaming-heavy upsert pipelines are where Hudi and Paimon still differentiate.
rewrite_data_files) and snapshot expiry, or reads slow down and metadata grows.remove_orphan_files periodically, with a retention window longer than your longest write.