WHEN TO USE DELTA LIVE TABLES (DLT)

Delta Live Tables (DLT) is most useful when you want automated, reliable, and declarative ETL pipelines with built-in data quality and orchestration. Below are the primary scenarios where DLT is the best choice.

1. Continuous or Streaming Ingestion

• Data arrives frequently (minutes or seconds).
• You need automated incremental ingestion.
• DLT works naturally with Auto Loader and STREAMING LIVE TABLES.

2. Built-in Data Quality (Expectations)

• You want to validate data with declarative rules.
• DLT can warn, drop, or fail on bad records.
• No custom validation framework needed.

3. Automatic Bronze → Silver → Gold Orchestration

• DLT builds dependency DAGs automatically.
• Ensures correct ETL ordering.
• Retries, error handling, and job coordination handled by Databricks.

4. Data Lineage, Monitoring, and Observability

• DLT provides a full graphical lineage view.
• You can monitor row counts, errors, and latency.
• Ideal for production-grade auditing and compliance.

5. Automatic Table Optimization

• DLT performs background OPTIMIZE and VACUUM.
• Improves performance without manual maintenance jobs.

6. Managing SCD Type 2 with Minimal Code

• dlt.apply_changes can implement SCD Type 2 automatically.
• No need to write long MERGE statements.
• DLT handles versioning, current flags, and historical tracking.

7. Declarative Pipelines Instead of Manual Jobs

• You define what the tables should be, DLT figures out how to build them.
• Reduces boilerplate and makes pipelines easier to maintain.

8. Teams That Prefer Low-Code / No-Ops Data Engineering

• DLT reduces operational overhead.
• Good for teams without heavy Spark expertise.
• Ensures consistent engineering practices across the project.

Summary

1. Use DLT for streaming, continuous, or reliable incremental pipelines.
2. Use DLT when you want built-in data quality, monitoring, and orchestration.
3. Use DLT to simplify complex logic such as SCD Type 2.
4. Use DLT when you want automatic optimization and fewer maintenance tasks.

Summary

Delta Live Tables (DLT) is Databricks' declarative ETL framework, renamed Lakeflow Declarative Pipelines in 2025 with the same Python and SQL concepts. Instead of writing jobs that read, transform, write and schedule each step, you declare the tables you want and the query that defines each one. The runtime builds the dependency graph, provisions compute, processes data incrementally, enforces data-quality rules, and maintains the tables. The list above summarises when that trade is worth making; the sections below show how it works and what the code looks like.

How a pipeline runs

A pipeline contains two kinds of datasets. A streaming table processes each input record once and is appended to incrementally; it is the natural choice for ingestion with Auto Loader or Kafka. A materialized view stores the result of a query and is refreshed to reflect its sources, recomputing incrementally where possible. When an update runs, DLT resolves references between datasets into a DAG, starts the upstream tables first, and records each step in an event log that also powers the lineage graph in the UI. Pipelines run in triggered mode (process available data then stop) or continuous mode (keep running for low latency).

Expectations

Expectations are named boolean constraints attached to a dataset. The default action keeps violating rows and counts them; expect_or_drop removes them; expect_or_fail stops the update so bad data never lands. Because results go to the event log, teams can chart quality over time and alert on regressions without a separate validation framework.

Worked example: bronze to gold with SCD2

import dlt
from pyspark.sql.functions import col, count, to_date

@dlt.table(comment="Raw JSON events from cloud storage")
def bronze_events():
    return (spark.readStream.format("cloudFiles")
            .option("cloudFiles.format", "json")
            .load("/Volumes/main/raw/events/"))

@dlt.table
@dlt.expect_or_drop("has_user", "user_id IS NOT NULL")
@dlt.expect("recent", "event_ts > '2020-01-01'")
def silver_events():
    return dlt.read_stream("bronze_events").withColumn("event_date", to_date(col("event_ts")))

@dlt.table(comment="Daily counts for dashboards")
def gold_daily():
    return dlt.read("silver_events").groupBy("event_date").agg(count("*").alias("events"))

# SCD Type 2 customer dimension from a CDC feed
dlt.create_streaming_table("dim_customer")
dlt.apply_changes(
    target="dim_customer", source="customers_cdc",
    keys=["customer_id"], sequence_by=col("updated_at"),
    stored_as_scd_type=2)

-- Equivalent SQL for the silver step
CREATE OR REFRESH STREAMING TABLE silver_events (
  CONSTRAINT has_user EXPECT (user_id IS NOT NULL) ON VIOLATION DROP ROW
) AS SELECT *, to_date(event_ts) AS event_date FROM STREAM(bronze_events);

The apply_changes call replaces a hand-written MERGE: it orders changes by updated_at, closes the previous version of a changed customer with an end timestamp, and inserts the new version.

DLT vs jobs and notebooks

Pitfalls

• Side effects (writing files, calling APIs) inside dataset functions; they may run multiple times.
• Using a materialized view where a streaming table is needed, causing full recomputation.
• Expecting to write to pipeline tables from outside the pipeline.
• Full refresh of a streaming table whose source no longer retains old data, which loses history.

Related topics

• Delta Live Tables
• DLT SCD Type 2
• Managed, External and Live tables
• Delta Lake design

Featured Videos