Data lineage is the record of where data came from and where it goes: which sources, jobs and transformations produced a table, column or dashboard number. This page covers table-level and column-level lineage, how lineage is collected automatically with the OpenLineage standard, and the everyday uses that justify the effort: impact analysis, root-cause debugging, and compliance.
Formally, lineage is a directed acyclic graph. Nodes are datasets (files, tables, topics, dashboards, models) and usually the jobs that read and write them; edges mean "was derived from". Walking edges to the left answers "where did this come from?" (upstream). Walking to the right answers "what depends on this?" (downstream).
Lineage is related to, but different from, provenance. Provenance usually refers to the history of a specific record or value, including which run produced it and when. Most data platforms capture lineage at dataset or column granularity plus run metadata, which covers the majority of practical questions.
Modern data stacks contain thousands of tables built by many teams. Without lineage, three common questions become slow detective work. Impact analysis: "If I drop this column or change this table, what breaks?" Root-cause analysis: "The revenue dashboard is wrong; which upstream job or source caused it?" Compliance: "Where does this customer's personal data flow, so we can honour a deletion request or prove that PII never reaches the analytics zone?"
Lineage also makes data contracts practical, because before approving a breaking change you can list every consumer, and it makes data quality alerts actionable by showing which downstream assets are affected by a failed check.
Table-level lineage records that mart.revenue_daily depends on stg.orders_clean and ref.fx_rates. It is cheap to collect from job inputs and outputs and is enough for scheduling and coarse impact analysis.
Column-level lineage records that revenue_usd is computed from amount and fx_rate, joined on currency. It requires parsing the SQL or the query plan, so it is harder to get right, but it removes the false alarms of table-level analysis: dropping the unused notes column affects nothing. It is also what privacy teams need to follow a PII field through renames and derivations.
| Aspect | Table-level | Column-level |
|---|---|---|
| Captured from | Job inputs/outputs, query logs | SQL parsing or engine query plans |
| Cost to collect | Low | Higher; needs parser per dialect |
| Impact analysis | Over-reports (whole table) | Precise to the field |
| PII tracing | Coarse | Follows fields through transforms |
| Typical tools | Airflow/OpenLineage, warehouse query history | Unity Catalog, dbt + SQL parsers, DataHub, sqllineage |
There are three collection strategies. Static analysis parses SQL or dbt models before they run; it is fast but cannot see dynamic SQL. Runtime instrumentation has the execution engine report what it actually read and wrote; this is the approach of OpenLineage, an open standard (a Linux Foundation AI and Data project) with integrations for Airflow, Spark, dbt and Flink. Log mining reconstructs lineage from warehouse query history, which is how many catalogs build lineage for Snowflake or BigQuery.
The OpenLineage model has three core entities: a Job (a recurring process), a Run (one execution of it), and Datasets (inputs and outputs). Extra detail, such as schema, column lineage or data quality metrics, is attached as facets. Events go to a backend such as Marquez (the reference implementation) or a catalog like DataHub, which merges them into the graph.
Most teams never write events by hand because the integrations do it, but writing one shows exactly what is captured. This emits START and COMPLETE events for a job that builds the revenue mart:
# pip install openlineage-python
from datetime import datetime, timezone
from uuid import uuid4
from openlineage.client import OpenLineageClient
from openlineage.client.event_v2 import Dataset, Job, Run, RunEvent, RunState
client = OpenLineageClient(url="http://localhost:5000") # e.g. a Marquez server
NS = "warehouse://analytics"
job = Job(namespace="etl", name="build_revenue_daily")
run = Run(runId=str(uuid4()))
inputs = [Dataset(NS, "stg.orders_clean"), Dataset(NS, "ref.fx_rates")]
outputs = [Dataset(NS, "mart.revenue_daily")]
def emit(state):
client.emit(RunEvent(eventType=state,
eventTime=datetime.now(timezone.utc).isoformat(),
run=run, job=job, producer="https://example.com/etl",
inputs=inputs, outputs=outputs))
emit(RunState.START)
# ... run the actual SQL / Spark work here ...
emit(RunState.COMPLETE)Once lineage exists, impact and root-cause analysis are just graph traversals. The snippet below uses networkx on a small graph shaped like the one a backend would return through its API:
# Impact analysis on a lineage graph: who is affected if a table changes?
import networkx as nx
g = nx.DiGraph()
g.add_edges_from([
("raw.orders", "stg.orders_clean"),
("raw.customers", "stg.customers"),
("stg.orders_clean", "mart.revenue_daily"),
("ref.fx_rates", "mart.revenue_daily"),
("stg.customers", "mart.customer_ltv"),
("stg.orders_clean", "mart.customer_ltv"),
("mart.revenue_daily", "dash.exec_kpis"),
("mart.customer_ltv", "ml.churn_features"),
])
changed = "raw.orders"
print("downstream of", changed, "->", sorted(nx.descendants(g, changed)))
print("root causes for dash.exec_kpis ->", sorted(nx.ancestors(g, "dash.exec_kpis")))The output shows that a change to raw.orders reaches the executive dashboard and the churn feature set, so both owners must be consulted, and that a wrong number on dash.exec_kpis can only come from five upstream assets. On a real platform the same queries run against thousands of nodes in milliseconds.
Lineage programmes fail when they try to map everything at once. A rollout that works in most organisations looks like this. First, turn on the lineage that your tools already emit: the OpenLineage integration for Airflow or Spark, dbt's manifest (which already contains the model dependency graph), and your warehouse's query history. That alone usually covers most of the table-level graph for free.
Second, pick one high-value use case and make it excellent. Common choices are the executive revenue dashboard, a regulatory report, or the feature tables behind an important model. Fill the gaps for that path, including the BI layer and any hand-run scripts, and add column-level lineage where it matters.
Third, wire lineage into daily workflows rather than leaving it in a catalog UI that nobody opens. Post the downstream list on schema-change pull requests, attach affected assets to data quality alerts, and show upstream freshness next to dashboard tiles. Measure coverage (share of production jobs that emit lineage) and push it upward over time. Lineage that people see in the tools they already use stays accurate, because gaps get noticed and reported.
prod.sales.orders by one tool and ORDERS by another produces two nodes. Agree on namespaces.