Databricks
Delta Live (DLT), Managed, External Tables

Key Differences:

1. Delta Live Tables (DLT)

Delta Live Tables (DLT) is a framework designed for building and managing ETL pipelines. It automates data processing, handling dependencies, and optimizing workflows in both batch and streaming data pipelines.

Example of Delta Live Tables Pipeline:

import dlt
from pyspark.sql.functions import *

@dlt.table
def clean_data():
    return spark.read("path/to/raw_data").filter(col("age") > 18)
    

2. Managed Tables

Managed tables are fully controlled by Databricks. Databricks manages both the metadata and data storage. When you drop a managed table, both the metadata and the underlying data files are deleted.

Example of Creating a Managed Table:

CREATE TABLE my_managed_table (
    id INT,
    name STRING
) USING DELTA;
    

3. External Tables

External tables allow users to store data externally and only manage metadata within Databricks. The actual data remains in external storage, such as AWS S3, Azure Blob, or HDFS.

Example of Creating an External Table:

CREATE TABLE my_external_table (
    id INT,
    name STRING
) USING DELTA LOCATION '/mnt/external_data/';
    

Summary

Databricks offers three ways to own a Delta table, and the choice controls who manages the files, what happens when the table is dropped, and how the table is refreshed. A managed table stores data in a location chosen and governed by the metastore (in Unity Catalog, the managed storage of the catalog or schema). An external table points at a path you choose with LOCATION, so the metastore owns only the metadata. A Delta Live Tables table (renamed Lakeflow Declarative Pipelines in 2025) is defined in a pipeline as a streaming table or materialized view; the pipeline creates it, refreshes it and manages its lifecycle, and its data is stored as managed data.

Ownership and lifecycle

Dropping a managed table removes its metadata and marks the underlying files for deletion; in Unity Catalog the files are kept for a short recovery window during which UNDROP can restore the table, after which they are deleted. Dropping an external table removes only the metadata; the files remain and can be re-registered with another CREATE TABLE ... LOCATION. Pipeline tables follow the pipeline: removing a table definition from the pipeline, or deleting the pipeline, removes the table. Managed tables also gain platform features that rely on owning the files, such as predictive optimization that schedules OPTIMIZE and VACUUM automatically.

Choosing a table type

Start from the workload. If the table is the output of a repeatable transformation with data-quality rules and dependencies, define it in a pipeline. Otherwise create a plain table, and choose external only when there is a concrete reason: other engines write the files directly, the data must survive table drops, or storage must sit in a specific bucket for compliance or cost allocation. In Unity Catalog, external tables require an external location backed by a storage credential, which is itself a governed object.

Worked example

-- Managed: Unity Catalog chooses the storage path
CREATE TABLE main.sales.orders (id BIGINT, amount DECIMAL(10,2)) ;

-- External: data lives where you say
CREATE TABLE main.sales.orders_ext (id BIGINT, amount DECIMAL(10,2))
LOCATION 's3://acme-lake/sales/orders_ext';

-- Inspect type and location
DESCRIBE TABLE EXTENDED main.sales.orders_ext;   -- Type: EXTERNAL, Location: s3://...

DROP TABLE main.sales.orders;      -- data scheduled for deletion
UNDROP TABLE main.sales.orders;    -- within the retention window
DROP TABLE main.sales.orders_ext;  -- files remain in S3

import dlt
from pyspark.sql.functions import col

@dlt.table(comment="Adults only, validated")
@dlt.expect_or_drop("valid_age", "age IS NOT NULL AND age >= 0")
def clean_people():
    return spark.readStream.table("main.raw.people").filter(col("age") > 18)

The pipeline version adds an expectation that drops invalid rows and records how many were dropped, something a plain table would need custom code for.

Trade-offs

Pitfalls

• Two external tables pointing at the same path, which lets one overwrite the other.
• Writing into a pipeline-owned table from a notebook; the pipeline will overwrite or conflict.
• Assuming DBFS root storage is appropriate for production managed tables; use Unity Catalog managed storage.
• Dropping an external table and expecting storage costs to go away.

Related topics

• Managed vs External tables
• Delta Live Tables
• When to use DLT
• Delta Lake design

Featured Videos