Databricks Delta Lake Design for Big Data

1. What Delta Lake is

Delta Lake is an open-source table format and storage layer that adds database-like guarantees to files in cloud object storage. A Delta table is a directory of Parquet data files plus a _delta_log directory that records every change as an ordered series of commits. That log is what provides ACID transactions, schema enforcement, time travel and efficient metadata at petabyte scale, and it lets the same table serve batch jobs, streaming jobs and SQL queries without copies. Delta Lake was created by Databricks and is now a Linux Foundation project; Databricks uses it as the default table format, and engines such as Spark, Trino, Flink and DuckDB can read it.

2. How the transaction log works

Each commit is a JSON file named with a zero-padded version number. It lists actions: files added, files removed, metadata or schema changes, and commit information. To read version N, an engine loads the latest checkpoint at or before N and replays the JSON commits after it to determine which files are live. Writes use optimistic concurrency: a writer reads the current version, writes new Parquet files, and then tries to create the next log entry. Object stores or a coordination service guarantee only one writer can create a given version; a loser checks whether its changes conflict and either retries or fails. Because old files are only marked removed, previous versions remain readable until VACUUM deletes files past the retention period.

3. Medallion architecture

The medallion pattern described above organises tables by quality. Bronze keeps raw data as received so that any downstream table can be rebuilt. Silver applies parsing, de-duplication, type casting and joins. Gold holds business-level aggregates for dashboards and machine learning features. Delta's ACID writes and MERGE support make the incremental hops between layers reliable, and Change Data Feed can propagate only changed rows.

4. Worked example

CREATE TABLE sales.silver_orders (
  order_id BIGINT, customer_id BIGINT, amount DECIMAL(10,2), order_ts TIMESTAMP
) USING DELTA;

-- upsert a batch of changes
MERGE INTO sales.silver_orders t
USING sales.bronze_orders_updates s
ON t.order_id = s.order_id
WHEN MATCHED THEN UPDATE SET *
WHEN NOT MATCHED THEN INSERT *;

DESCRIBE HISTORY sales.silver_orders;                 -- one row per commit
SELECT * FROM sales.silver_orders VERSION AS OF 3;     -- time travel
RESTORE TABLE sales.silver_orders TO VERSION AS OF 3;  -- roll back

OPTIMIZE sales.silver_orders ZORDER BY (customer_id); -- compact small files
VACUUM sales.silver_orders RETAIN 168 HOURS;          -- delete unreferenced files

# streaming write from Bronze to Silver with PySpark
(spark.readStream.table("sales.bronze_orders")
      .filter("amount IS NOT NULL")
      .writeStream
      .option("checkpointLocation", "/chk/silver_orders")
      .toTable("sales.silver_orders_stream"))

Several features build on the log. Schema enforcement rejects writes whose columns or types do not match, and schema evolution (for example mergeSchema) allows controlled additions. Change Data Feed records row-level inserts, updates and deletes so downstream tables can process only changes. Deletion vectors mark deleted rows without rewriting whole files, which makes updates and deletes cheaper. Liquid clustering replaces fixed partitioning and Z-ordering with incremental clustering on chosen keys that can be changed later without rewriting the table.

5. Delta vs alternatives

6. Pitfalls

• Many small files from frequent streaming writes; schedule OPTIMIZE or enable auto-compaction.
• VACUUM with a short retention breaks time travel and can break long-running readers.
• Over-partitioning on high-cardinality columns; prefer clustering or Z-ordering.
• Writing to the same table from non-Delta-aware tools, which bypasses the log.

7. Related topics

• Managed, External and Live tables
• Delta Live Tables
• DLT SCD Type 2
• Delta Lake overview

Featured Videos