Unity Catalog

Unity Catalog is Databricks’ unified governance layer for the lakehouse. Open-sourced in mid-2024 (Unity Catalog OSS, Apache 2.0), it has become the default multi-engine catalog on Databricks and Microsoft Fabric, with growing support for non-Databricks engines through the open APIs. Unity is broader than a table catalog — it governs tables, volumes (files), ML models, AI tools, and SQL functions in one namespace hierarchy.

The sections below explain the object hierarchy, how credential vending enforces access, worked SQL for grants, column masks and row filters, and how Unity Catalog compares with Polaris and Nessie.

1. Key Features

Three-Level Namespace. catalog.schema.object — a clean break from the flat Hive Metastore. Catalogs map naturally to environments, business units, or regions.
Multi-Asset Governance. Tables (Delta + Iceberg), file volumes, registered ML models, AI agents, and functions all live under the same RBAC and lineage system.
Iceberg + Delta Coexistence. Unity Catalog OSS supports both formats; tables can be read as either via UniForm.
Fine-Grained Access Control. Row filters, column masks, dynamic views, attribute-based policies.
Lineage & Audit. Automatic table-level and column-level lineage across notebooks, jobs, and dashboards.
Open APIs. Iceberg REST and Delta Sharing protocols mean Trino, Spark, Snowflake, and other engines can authenticate and read governed data without going through Databricks runtimes.

2. Architecture

Unity Catalog runs as a metadata service backed by a relational store. Compute engines (Databricks clusters, Trino, Spark) authenticate to UC, request access, and receive scoped cloud credentials and metadata pointers. Object storage paths are abstracted by external locations and storage credentials, so end users see logical names, not S3 ARNs.

3. Unity Catalog vs. Apache Polaris

Unity Catalog is broader: tables + files + models + functions, with deep Databricks integration.
Polaris is narrower: Iceberg-only, REST-API-first, vendor-neutral.
They overlap on Iceberg governance; in practice many large orgs run both, federating through Iceberg REST.

4. Use Cases

Multi-workspace governance across an enterprise Databricks deployment.
Unified RBAC over tables, ML models, and unstructured files.
Cross-engine sharing of Delta and Iceberg tables to Trino, Snowflake, and partners.
Compliance regimes that require column-level audit and lineage.

5. Data model: the three-level namespace

Unity Catalog organises every governed object as catalog.schema.object under a metastore. On Databricks the metastore is regional and shared by the workspaces attached to it, so the same table name and permissions apply in every workspace. Objects include managed and external tables, views, volumes (governed directories of files such as PDFs or images), registered ML models and SQL or Python functions. Privileges such as USE CATALOG, USE SCHEMA, SELECT and MODIFY are inherited downward, so granting on a schema covers its tables.

6. How access is enforced

Engines never hold long-lived storage keys. An engine asks Unity Catalog for a table; Unity Catalog checks the caller's privileges, records the access for audit and lineage, and returns the table's location with short-lived, path-scoped cloud credentials. The engine reads the files directly from object storage. Row filters and column masks are enforced by trusted Databricks compute; external engines reading through the open APIs receive credentials only for tables they may read in full, so fine-grained policies require compute that understands them.

7. Worked example: grants, masks and lineage

CREATE CATALOG IF NOT EXISTS prod;
CREATE SCHEMA IF NOT EXISTS prod.sales;

CREATE TABLE prod.sales.orders (
  order_id BIGINT, customer_email STRING, region STRING, amount DECIMAL(12,2)
);

-- analysts can read the schema
GRANT USE CATALOG ON CATALOG prod TO `analysts`;
GRANT USE SCHEMA, SELECT ON SCHEMA prod.sales TO `analysts`;

-- mask email for everyone outside the support group
CREATE FUNCTION prod.sales.mask_email(e STRING) RETURN
  CASE WHEN is_account_group_member('support') THEN e ELSE '***' END;
ALTER TABLE prod.sales.orders ALTER COLUMN customer_email SET MASK prod.sales.mask_email;

-- restrict rows by region
CREATE FUNCTION prod.sales.emea_only(r STRING) RETURN
  is_account_group_member('emea') OR r <> 'EMEA';
ALTER TABLE prod.sales.orders SET ROW FILTER prod.sales.emea_only ON (region);

-- audit who can do what
SHOW GRANTS ON TABLE prod.sales.orders;

The row-filter function here hides EMEA rows from anyone outside the emea group. Lineage needs no code: queries run on Unity Catalog-enabled compute are recorded automatically and appear as table- and column-level graphs in the catalog explorer and system tables.

8. When to use it

Requirement	Unity Catalog	Alternative
Databricks-centred platform	Default choice	—
Iceberg-only, engine-neutral catalog	Possible via Iceberg REST	Apache Polaris
Git-style branching of table state	Not its model	Project Nessie
Governance of files, models and functions alongside tables	Strong	Separate tools per asset type

9. Pitfalls

Migrating from the Hive metastore means rewriting three-part names and re-creating grants; plan the upgrade per schema.
Features differ between Databricks-hosted Unity Catalog and the open-source server; check which you run before depending on a feature.
Granting ALL PRIVILEGES on a catalog is easy and hard to audit; prefer group-based, schema-level grants.
Masks and filters evaluate a function per row; keep them simple to avoid query slowdowns.

See the Catalogs overview, Delta Lake and Apache Iceberg for the table formats it governs, and Trino as an external engine.