Data pipelines on Databricks move data from source systems into Delta tables and refine it through bronze, silver and gold layers. This section focuses on change data capture (CDC), the pattern for applying inserts, updates and deletes from a source instead of reloading it, and places it alongside the other ingestion options on the platform.
Declarative pipeline construction on Databricks.
New files are picked up incrementally with Auto Loader, and change events from databases arrive through a CDC tool (Debezium, a vendor connector, or Lakeflow Connect) as a stream of row-level operations. The bronze layer stores these as received. The silver layer applies them to keep a current-state table, using MERGE INTO in a notebook or APPLY CHANGES INTO in Delta Live Tables, which also handles out-of-order events and SCD Type 2 history. Gold tables aggregate for reporting.
| Pattern | How | Use when |
|---|---|---|
| Full reload | Overwrite the table each run | Small tables, no change tracking at source |
| Incremental append | Auto Loader, Structured Streaming | Immutable events and files |
| CDC with MERGE | Jobs + Delta MERGE | Custom logic, existing notebooks |
| CDC with DLT | APPLY CHANGES INTO | Declarative pipelines, SCD Type 1/2, built-in quality checks |
Delta Live Tables (now part of Lakeflow Declarative Pipelines) trades some flexibility for managed orchestration, retries, lineage and data-quality expectations; plain Jobs with MERGE give full control at the cost of writing that plumbing yourself.