Apache Polaris

Apache Polaris is an open-source REST catalog for Apache Iceberg, originally built and open-sourced by Snowflake in 2024 and subsequently donated to the Apache Software Foundation. It implements the Iceberg REST Catalog specification and is rapidly becoming the standard answer to “what catalog do I run for Iceberg?” alongside AWS Glue and Nessie.

Key Features:

Iceberg REST Catalog. Implements the open Iceberg REST spec, so any compliant engine (Spark, Trino, Flink, Snowflake, Dremio, StarRocks) can connect with no custom integration.
Multi-Tenancy. Catalogs, namespaces, and tables with role-based access control; one Polaris instance can serve many isolated workspaces.
Credential Vending. Polaris hands out short-lived, scoped cloud credentials to engines so they can read S3 / GCS / ADLS directly — no long-lived keys in client configs.
Identity Provider Integration. OAuth, OIDC, SAML against existing corporate IdPs.
External Catalog Federation. Can sit in front of AWS Glue, Hive Metastore, or another Iceberg REST catalog.
Vendor-Neutral. Apache governance — no single-vendor lock-in despite Snowflake’s origin.

Architecture:

Polaris exposes the Iceberg REST API over HTTP, backed by a relational metastore (Postgres / managed Snowflake) for catalog state. Engines authenticate, list namespaces, and request table metadata pointers; Polaris then vends temporary credentials scoped to the requested table’s storage location. The actual data reads/writes happen directly between the engine and object storage.

Use Cases:

Single open catalog across heterogeneous engines (Snowflake + Spark + Trino) on the same Iceberg tables.
Replacing Hive Metastore or AWS Glue with a vendor-neutral, RBAC-aware catalog.
Multi-tenant data platforms that need workspace isolation and auditable access.
Organizations standardizing on Iceberg for their open lakehouse foundation.

4. Where Polaris Fits

An Iceberg table is a set of data files plus a tree of metadata files on object storage. What the files alone cannot provide is a single, atomically updated answer to "which metadata file is the current version of this table?" That is the catalog's job. Polaris implements the Iceberg REST Catalog specification, so any engine with a REST catalog client can list namespaces, load tables, and commit changes through it. Because every engine goes through the same catalog, a table written by Spark is immediately visible to Trino or Snowflake with consistent snapshots and a single place to enforce permissions.

5. How a Commit Works

When an engine loads a table, Polaris returns the location of the current metadata file and, if credential vending is configured, temporary storage credentials scoped to that table's location. The engine reads or writes data and manifest files directly on object storage; Polaris is not in the data path. To commit, the engine sends the new metadata along with requirements such as "the current snapshot is still X". Polaris applies the change only if the requirements hold, giving optimistic concurrency: one of two conflicting writers succeeds and the other retries against the new state.

6. Access Control Model

Polaris separates principals, principal roles, and catalog roles. Privileges such as TABLE_READ_DATA or NAMESPACE_CREATE are granted to catalog roles on a catalog, namespace, or table; catalog roles are granted to principal roles; principal roles are assigned to principals, which are the service identities that engines authenticate as. This indirection lets one engine identity have different rights in different catalogs. Combined with credential vending, engines never need broad standing access to the storage bucket.

7. Worked Example: Spark Configuration

from pyspark.sql import SparkSession

spark = (SparkSession.builder
  .config("spark.sql.extensions",
          "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions")
  .config("spark.sql.catalog.polaris", "org.apache.iceberg.spark.SparkCatalog")
  .config("spark.sql.catalog.polaris.type", "rest")
  .config("spark.sql.catalog.polaris.uri", "https://polaris.example.com/api/catalog")
  .config("spark.sql.catalog.polaris.warehouse", "analytics")      # catalog name
  .config("spark.sql.catalog.polaris.credential", "CLIENT_ID:CLIENT_SECRET")
  .config("spark.sql.catalog.polaris.scope", "PRINCIPAL_ROLE:ALL")
  .config("spark.sql.catalog.polaris.header.X-Iceberg-Access-Delegation",
          "vended-credentials")
  .getOrCreate())

spark.sql("CREATE NAMESPACE IF NOT EXISTS polaris.sales")
spark.sql("""CREATE TABLE IF NOT EXISTS polaris.sales.orders
             (id BIGINT, amount DECIMAL(12,2), ts TIMESTAMP)
             USING iceberg PARTITIONED BY (days(ts))""")
spark.sql("SELECT count(*) FROM polaris.sales.orders").show()

The Iceberg Spark runtime jar matching your Spark version must be on the classpath. Keep the client secret in a secret store rather than in code.

8. Polaris vs Other Catalogs

Catalog	Interface	Notes
Apache Polaris	Iceberg REST	Vendor-neutral, RBAC, credential vending
Unity Catalog	Own API plus Iceberg REST	Delta-first, Databricks ecosystem
AWS Glue Data Catalog	Glue API, Iceberg REST endpoint	AWS-native, IAM permissions
Hive Metastore	Thrift	Legacy, no fine-grained Iceberg features
Nessie	Own API plus Iceberg REST	Git-like branches across tables

9. Pitfalls

The default in-memory metastore loses state on restart; configure a persistent backend for anything beyond testing.
Misconfigured storage roles break credential vending with confusing access-denied errors from the storage service, not from Polaris.
Engines that bypass the catalog and write files directly can corrupt a table's view of its own history.