This page goes beneath the feature list: how the Delta transaction log turns a directory of Parquet files into a table with ACID guarantees, how concurrent writers are coordinated, and the SQL and Python you use day to day.
Delta Lake is the third major open table format alongside Apache Hudi and Apache Iceberg. Originated at Databricks in 2019 and donated to the Linux Foundation in 2022, it adds ACID transactions, schema enforcement, time travel, and incremental processing to Parquet-on-object-storage. Delta is the native table format on Databricks and is increasingly first-class on Snowflake, Microsoft Fabric, and AWS.
_delta_log/) — ordered JSON files (and periodic Parquet checkpoints) that record every operation against the table.mergeSchema or explicit DDL.VERSION AS OF) or timestamp (TIMESTAMP AS OF) for audit, debugging, and rollback.A Delta table is a directory of Parquet data files plus a _delta_log/ folder of transaction-log entries. Each commit appends a numbered JSON file describing the added and removed Parquet files; periodic Parquet checkpoints summarize the log so readers don’t have to replay every entry. Optimistic concurrency control on the log gives serializable writes without a coordinator.
MERGE.For deeper Databricks-specific patterns — managed vs. external tables, Delta Live Tables, and SCD Type 2 — see the Databricks Delta Lake reference.
A Delta table is a directory of Parquet files plus a _delta_log/ folder. Each commit writes a numbered JSON file (00000000000000000012.json) listing actions: files added, files removed, metadata and schema changes, and commit info. The current state of the table is the replay of every action up to the latest version. Every ten commits by default, a Parquet checkpoint summarises the state so readers do not replay the entire history. Data files are never modified in place; an UPDATE writes new files and logically removes the old ones.
Delta uses optimistic concurrency. A writer reads the latest version, writes its new data files, then attempts to create the next log entry. Creating that file must be atomic and fail if it already exists. On HDFS, Azure and GCS this is native; on S3 it historically required a coordination service, though S3 now supports conditional writes. If another writer won the race, Delta checks whether the two transactions touched overlapping files; appends usually succeed after a retry, while conflicting updates fail with a concurrent modification error.
from delta.tables import DeltaTable
df = spark.createDataFrame([(1, "a", 10), (2, "b", 20)], ["id", "name", "amt"])
df.write.format("delta").mode("overwrite").save("/tmp/delta/customers")
updates = spark.createDataFrame([(2, "b", 25), (3, "c", 30)], ["id", "name", "amt"])
tgt = DeltaTable.forPath(spark, "/tmp/delta/customers")
(tgt.alias("t")
.merge(updates.alias("s"), "t.id = s.id")
.whenMatchedUpdateAll()
.whenNotMatchedInsertAll()
.execute())
tgt.history().select("version", "operation").show()
spark.read.format("delta").option("versionAsOf", 0).load("/tmp/delta/customers").show()-- Time travel, compaction and cleanup
SELECT * FROM customers VERSION AS OF 0;
OPTIMIZE customers ZORDER BY (id);
VACUUM customers RETAIN 168 HOURS;
DESCRIBE HISTORY customers;MERGE is the workhorse for CDC and slowly changing dimensions. OPTIMIZE compacts small files and Z-ordering co-locates rows with similar values so file-level statistics prune more. VACUUM physically deletes files no longer referenced, which also limits how far back time travel can go.
| Aspect | Delta Lake | Apache Iceberg | Apache Hudi |
|---|---|---|---|
| Metadata | Ordered JSON log + checkpoints | Metadata tree: snapshots, manifest lists, manifests | Timeline of instants |
| Strength | Tight Spark/Databricks integration | Engine-neutral, hidden partitioning | Record-level upserts, incremental pulls |
| Partition evolution | Liquid clustering | Native | Limited |
Interoperability layers such as UniForm and Apache XTable can expose one table's metadata in another format.