Pipelines can succeed while producing wrong data. Data observability is how teams notice: automatic monitoring of every table's freshness, volume, schema and value distributions, tied together with lineage.
Data observability is the practice of continuously monitoring the health of data as it moves through pipelines, so that problems are detected, explained and fixed before people downstream act on bad numbers. The idea borrows from software observability (metrics, logs, traces) and applies it to tables: instead of CPU and latency, you watch how fresh a table is, how many rows arrived, whether its schema changed, whether its values look normal, and what depends on it.
These five pillars, popularised by vendors such as Monte Carlo, give a practical checklist. Their key property is that they can be collected automatically from warehouse metadata and query logs for every table, without anyone writing a test first.
Data incidents are usually silent. A broken API key returns zero rows, a vendor renames a column, an upstream team changes a status code from "C" to "cancelled", and every pipeline still reports success. The first person to notice is an executive looking at a dashboard that is suddenly 40 percent lower, or a model whose predictions drift for weeks. The cost lies in the time to detection and the decisions made in between.
Traditional data quality testing catches the failures you anticipated. Observability catches the ones you did not, by learning what "normal" looks like for each table and alerting on deviations. The two are complementary: tests encode explicit business rules; observability provides broad, low-effort coverage.
Freshness measures the time since a table was last updated, from information_schema or the max of a timestamp column. An alert fires when the gap exceeds the table's usual cadence by a margin. Volume tracks rows added per load or per day. Schema monitoring diffs column names and types between runs and flags additions, removals and type changes. Distribution tracks per-column statistics: null rate, distinct count, mean, min, max, percentiles, and category frequencies. Lineage parses SQL and orchestration metadata to build a graph of which tables feed which.
Most tools detect anomalies by comparing each new value against its own history. A simple, robust approach is a rolling median with a band based on the median absolute deviation (MAD):
score = |x − median| / (1.4826 × MAD)
The constant 1.4826 makes the score comparable to a z-score for normally distributed data; scores above about 3 to 4 are suspicious. Unlike mean and standard deviation, the median and MAD are not distorted by the very outliers you are trying to find. Production systems add weekly seasonality (Monday volumes differ from Sunday) and let users mark false positives so thresholds adapt.
An alert on its own says "something is wrong with orders_raw". Lineage adds two answers. Upstream, it shows where the problem started, so the team fixes the source rather than patching three marts. Downstream, it lists the blast radius: the revenue fact table, the executive dashboard and the churn model are affected; the CRM sync is not. That list determines who must be notified and which reports should carry a warning. Column-level lineage narrows it further, since a broken discount column may not affect consumers that never read it.
The script below computes freshness, volume and null-rate metrics for a table in DuckDB and flags anomalies using the MAD rule. The same SQL runs, with small dialect changes, on Snowflake, BigQuery or Postgres.
import duckdb, numpy as np, pandas as pd
con = duckdb.connect()
rng = np.random.default_rng(7)
days = pd.date_range("2025-06-01", periods=30, freq="D")
rows = []
for d in days:
n = int(rng.normal(1000, 40)) if d != days[24] else 480 # day 25: partial load
null_rate = 0.01 if d < days[27] else 0.30 # day 28: upstream bug
for i in range(n):
rows.append((d, i, None if rng.random() < null_rate else rng.uniform(5, 300)))
con.register("rows_df", pd.DataFrame(rows, columns=["load_date", "order_id", "amount"]))
con.execute("CREATE TABLE orders AS SELECT * FROM rows_df")
metrics = con.sql("""
SELECT load_date,
COUNT(*) AS row_count,
AVG(CASE WHEN amount IS NULL THEN 1.0 ELSE 0 END) AS null_rate
FROM orders GROUP BY load_date ORDER BY load_date
""").df()
def mad_flags(series, window=14, threshold=4.0):
flags = []
for i, x in enumerate(series):
hist = series[max(0, i - window):i]
if len(hist) < 7:
flags.append(False); continue
med = np.median(hist)
mad = np.median(np.abs(hist - med)) or 1e-9
flags.append(abs(x - med) / (1.4826 * mad) > threshold)
return flags
metrics["volume_alert"] = mad_flags(metrics["row_count"].to_numpy())
metrics["null_alert"] = mad_flags(metrics["null_rate"].to_numpy())
freshness_hours = con.sql(
"SELECT date_diff('hour', MAX(load_date), TIMESTAMP '2025-07-01 06:00') FROM orders").fetchone()[0]
print(metrics[metrics.volume_alert | metrics.null_alert])
print("hours since last load:", freshness_hours)The output flags day 25 for volume and days 28 onward for null rate. If the null-rate problem were left unfixed, after about a week the bad values would make up most of the 14-day window, the median would shift, and the alert would stop firing: an anomaly detector learns a new normal if nobody acts, which is why alerts must be routed to an owner and acknowledged.
Once the basics are stable, encode the lessons from each incident as an explicit test or data contract, so that the same failure is caught deterministically next time rather than statistically.
| Practice | Question answered | Set-up effort | Catches unknown unknowns | Examples |
|---|---|---|---|---|
| Data tests | Does data meet the rules I wrote? | Per rule | No | dbt tests, Great Expectations, Soda |
| Data observability | Is anything behaving unusually? | Low, automatic | Yes | Monte Carlo, Bigeye, Elementary, Anomalo |
| Data contracts | Did the producer change the agreed interface? | Per interface | Partly | Schema registries, contract specs in CI |
| Data catalog / lineage | What exists and what depends on it? | Medium | No (context only) | DataHub, OpenLineage, Unity Catalog |