A data catalog answers the first questions anyone asks about data: what exists, what it means, who owns it, where it came from and whether it can be trusted. This page explains the kinds of metadata, how catalogs collect and serve them, and how the main tools compare.
Metadata is data about data: what a table contains, who owns it, where it came from, how fresh it is and what it means. A data catalog is the system that collects that metadata from across a platform and makes it searchable, so analysts and engineers can find the right dataset, understand it and decide whether to trust it.
Technical metadata comes from systems (schemas, types, file locations). Business metadata comes from people (a definition of "active customer", the owning team, a sensitivity classification). Operational metadata comes from pipeline runs (last refresh, row counts, test results, lineage). Usage metadata comes from query logs (who queries it, how often), which is a surprisingly strong signal of which of five similar tables is the real one.
As a platform grows past a few hundred tables, the main cost is no longer storage or compute but people's time spent asking "which table should I use?" and "why did this number change?". Catalogs reduce duplicated work, speed up onboarding, and make governance possible: you cannot protect personal data you have not found and tagged, and regulations such as GDPR require knowing where it lives. Lineage also makes change safer, because you can see every downstream dashboard before altering a column.
Metadata is also increasingly consumed by machines rather than people. Text-to-SQL assistants and AI agents that answer questions over a warehouse depend on good table and column descriptions, glossary definitions and certified-dataset flags to pick the right sources; without them they confidently query the wrong table. Data-quality tools read ownership to route alerts, and access-control systems read classification tags to decide what to mask. A catalog therefore works best as shared infrastructure with APIs, not just a website people browse.
A catalog has three parts. Ingestion gathers metadata, either by pull (scheduled crawlers reading information_schema, dbt's manifest.json, BI APIs) or by push (systems emitting events when something changes, for example OpenLineage events from Airflow, Spark or dbt). Storage keeps entities (datasets, columns, dashboards, jobs, people, glossary terms) and the relationships between them, usually in a graph-like model backed by a database plus a search index. Serving exposes search, entity pages, lineage graphs and APIs so other tools (access-control systems, data-quality checks, IDEs) can use the metadata.
Some catalogs are also enforcement points. Unity Catalog in Databricks, AWS Glue Data Catalog plus Lake Formation, and Apache Polaris for Iceberg are technical catalogs that query engines consult to resolve table names and permissions. Discovery-oriented catalogs such as DataHub, Amundsen and OpenMetadata sit across many systems and focus on search, context and lineage. Many organisations run one of each.
Lineage records how data flows from sources through transformations to consumers. Table-level lineage is easy to obtain from orchestrators or SQL parsing; column-level lineage requires parsing each transformation and is far more useful for impact analysis and for propagating sensitivity tags (if email is PII, every column derived from it should be too).
The script below pushes business metadata (a description, an owner and a tag) for a table into DataHub using its Python emitter. It assumes a DataHub instance at localhost:8080 (pip install acryl-datahub).
from datahub.emitter.mce_builder import make_dataset_urn, make_user_urn, make_tag_urn
from datahub.emitter.mcp import MetadataChangeProposalWrapper
from datahub.emitter.rest_emitter import DatahubRestEmitter
from datahub.metadata.schema_classes import (
DatasetPropertiesClass, OwnershipClass, OwnerClass, OwnershipTypeClass,
GlobalTagsClass, TagAssociationClass,
)
urn = make_dataset_urn(platform="snowflake", name="analytics.sales.fct_orders", env="PROD")
emitter = DatahubRestEmitter("http://localhost:8080")
aspects = [
DatasetPropertiesClass(
description="One row per completed order. Revenue in USD, net of refunds.",
customProperties={"grain": "order_id", "sla": "daily 06:00 UTC"}),
OwnershipClass(owners=[OwnerClass(owner=make_user_urn("jdoe"),
type=OwnershipTypeClass.DATAOWNER)]),
GlobalTagsClass(tags=[TagAssociationClass(tag=make_tag_urn("tier-1"))]),
]
for aspect in aspects:
emitter.emit(MetadataChangeProposalWrapper(entityUrn=urn, aspect=aspect))
print("emitted", urn)Technical metadata for the same table would normally come from a DataHub ingestion recipe pointed at Snowflake, so people only write the parts machines cannot know. The same documentation can live in dbt YAML (description, meta: {owner: ...}) and be ingested, keeping it next to the code.
# models/marts/schema.yml (dbt)
version: 2
models:
- name: fct_orders
description: "One row per completed order. Revenue in USD, net of refunds."
meta: {owner: "jdoe", tier: "tier-1"}
columns:
- name: order_id
tests: [unique, not_null]
- name: customer_email
meta: {pii: true}| Catalog | Type | Strengths | Notes |
|---|---|---|---|
| DataHub | Open-source discovery + lineage | Push-based events, broad connectors, column lineage | Originated at LinkedIn |
| Amundsen | Open-source discovery | Simple search-first UX, usage-based ranking | Originated at Lyft; smaller scope |
| OpenMetadata | Open-source discovery + quality | Unified schema, built-in data quality tests | Single metadata standard |
| Unity Catalog | Technical + governance | Enforces access, lineage within Databricks; open-sourced | Strongest inside Databricks |
| AWS Glue Data Catalog | Technical (Hive-compatible) | Used by Athena, EMR, Redshift Spectrum | Pair with Lake Formation for permissions |
| Collibra, Alation, Atlan | Commercial governance/discovery | Glossaries, stewardship workflows | Licensing cost |
A business glossary defines shared terms once ("active customer: placed an order in the last 90 days") and links each term to the columns and metrics that implement it. This is where catalogs pay off for non-engineers: a finance analyst searches for "net revenue", lands on the glossary entry, and sees the certified table and the dashboards that use it. Glossaries work best when each term has a steward who approves changes.
Classification tags sensitive data: PII, financial, health. Automated classifiers scan column names and sampled values for patterns such as email addresses or card numbers, then a human confirms. Confirmed tags should flow into access policies (masking, row filters) in the engine, and through lineage onto derived columns.
A practical rollout order: (1) ingest technical metadata automatically from the warehouse and transformation tool; (2) assign owners to the most-used datasets, which query logs reveal; (3) document and certify those tier-1 datasets; (4) add lineage from the orchestrator; (5) add classification and policies. Measuring adoption (searches per week, share of queries hitting certified tables) tells you whether it is working.