Reliable analytics and machine learning depend on data that is correct and on clear rules for who owns and may use it. This section covers both: testing data quality in pipelines, and the governance that surrounds it.
Data quality asks whether data is fit for its purpose: complete, accurate, valid, consistent, unique and on time. Data governance asks who is responsible for data and under what rules it may be used: ownership, definitions, access control, privacy, lineage and retention. The two are linked. Quality rules need an owner who decides what "correct" means, and governance policies are only credible if the data they govern can be trusted.
For people learning data engineering, the practical entry point is quality testing: writing checks that stop bad data before it reaches dashboards and models. Governance grows from there as more teams share the same data.
Quality checks sit inside the pipeline as gates between stages: structural checks right after ingestion (schema, nulls, row counts) and business-rule checks after transformation. Governance sits across the pipeline: a catalog records what each table means and who owns it, access policies control who can read sensitive columns, and lineage shows which sources feed which reports so the impact of a failed check is visible.
Tools fall into three groups. Rule-based testing frameworks such as Great Expectations, dbt tests, Soda and Deequ let engineers write explicit checks in code. Data observability and quality platforms, including Collibra Data Quality, profile data and learn expected behaviour to flag anomalies without hand-written rules. Catalogs and governance platforms such as Unity Catalog, Apache Polaris and Collibra's catalog manage metadata, ownership and policies.
| Need | Good fit | Why |
|---|---|---|
| Test pipeline outputs in Python, Spark or SQL | Great Expectations | Large expectation library, checkpoints, HTML Data Docs |
| Test warehouse models you already build with dbt | dbt tests | No extra infrastructure; tests live beside models |
| Spot unknown problems across many tables | Observability / rule-inference tools (e.g. Collibra DQ) | Learns baselines, detects drift and anomalies |
| Control who sees which columns across engines | Catalog-based governance (Unity Catalog, Polaris) | One policy enforced by every engine |
| Prove where a report's numbers came from | Lineage (OpenLineage, catalog lineage) | Traces columns from source to dashboard |
Most organisations combine several: explicit tests for known rules, anomaly detection for unknown ones, and a catalog for ownership and access. Start with a handful of tests on your most important tables and an owner for each; add tooling when the number of tables and teams makes manual tracking impossible.
Each dimension maps to concrete checks: null-rate thresholds, regex and range rules, reconciliation against source totals, primary-key uniqueness, and freshness checks on the latest timestamp.