Data mesh is a socio-technical approach to analytical data, introduced by Zhamak Dehghani in 2019, that moves ownership of data from a central team to the business domains that produce it. It is as much about organisation as technology. This page explains the four principles, what a data product actually is, how governance stays coherent, and when a mesh is the wrong choice.
In a classic setup, operational teams produce data and a central data engineering team ingests everything into a warehouse or lake, cleans it, and serves it to analysts. As the company grows, that central team becomes a bottleneck: it lacks domain knowledge, owns pipelines for data it does not understand, and has a queue of requests from every department.
Data mesh inverts the flow of responsibility. Each domain (sales, supply chain, marketing) publishes its analytical data as data products, maintained by people who understand it. A central platform team provides the tooling, and a federated group sets the rules that make products interoperable. The architecture is decentralised in ownership but not in infrastructure; most meshes run on one shared lakehouse or warehouse platform.
The main argument for a mesh is scaling: the number of data sources and use cases grows faster than any central team can. Putting ownership with domains means the people who change a source system are also responsible for the analytical data it feeds, which removes the most common failure in data pipelines, an upstream change nobody downstream was told about. It also produces better semantics, because the sales team knows what a "booked order" means far better than a central engineer does.
The cost is real: every domain needs some data engineering capability, and the platform and governance functions must be strong. For small organisations the overhead outweighs the benefit.
Domain-oriented ownership. Analytical data is owned by the domain closest to its source or its main use. Ownership includes quality, documentation and on-call.
Data as a product. Each published dataset is treated like a product with users: it must be discoverable in a catalog, addressable at a stable location, trustworthy (tested, with SLOs), self-describing (schema and semantics), interoperable (shared identifiers and formats), and secure. In practice this means each product publishes a data contract.
Self-serve data platform. A platform team builds paved roads so that a domain can create a new data product without becoming infrastructure experts: storage, compute, pipeline templates, a catalog, access control and monitoring, all provisioned by configuration.
Federated computational governance. A group with representatives from domains and the platform defines global standards (identifiers, PII handling, naming, versioning). "Computational" means the rules are enforced by code in the platform, not by review meetings.
A data product is the unit of architecture in a mesh. It is more than a table: it bundles the data itself, the code that produces and tests it, metadata (owner, contract, lineage, SLOs) and policies (who can read what). It consumes from input ports (source systems or other products) and exposes output ports: a SQL table for analysts, an event stream for real-time consumers, or files for ML. Other products can build on it, which creates a graph of products whose lineage crosses domain boundaries.
A product descriptor registers the product with the platform. This one declares owner, ports, contract, SLOs and classification:
# data-product.yaml -- registered in the mesh catalog by the Sales domain
name: sales.orders
domain: sales
owner: team-checkout
description: Placed customer orders, one row per order, deduplicated.
output_ports:
- type: sql
location: lakehouse.sales.orders_v2
contract: contracts/orders.contract.yaml # schema + semantics + SLA
- type: kafka
topic: sales.orders.v2
slo:
freshness: 1h
availability: 99.5%
classification:
pii_columns: [customer_email]
default_access: masked
consumers_allowed: [finance, marketing, data-science]Federated governance becomes concrete when the platform validates every descriptor in CI. The script below encodes five global rules; a product that breaks any of them cannot be published:
# Federated computational governance: global rules checked automatically
# against every data product descriptor before it is published.
import yaml, glob, sys
RULES = {
"has_owner": lambda p: bool(p.get("owner")),
"has_contract": lambda p: all("contract" in o for o in p["output_ports"] if o["type"] == "sql"),
"has_freshness": lambda p: "freshness" in p.get("slo", {}),
"pii_masked": lambda p: not p.get("classification", {}).get("pii_columns")
or p["classification"].get("default_access") == "masked",
"versioned_port": lambda p: all(o.get("location", o.get("topic", "")).endswith(("_v1", "_v2", ".v1", ".v2"))
for o in p["output_ports"]),
}
failed = False
for path in glob.glob("domains/*/data-product.yaml"):
product = yaml.safe_load(open(path))
for name, rule in RULES.items():
if not rule(product):
print(f"{product['name']}: FAILS {name}")
failed = True
sys.exit(1 if failed else 0)Real platforms implement the same idea with policy engines (Open Policy Agent), catalog APIs, and infrastructure-as-code modules, but the principle is identical: rules agreed by the federation, executed by machines, applied to every domain equally.
| Approach | Ownership | Strength | Weakness |
|---|---|---|---|
| Central warehouse | Central data team | Consistency, simple governance | Bottleneck at scale, weak domain knowledge |
| Data lake / lakehouse | Usually central | Cheap storage, all data types | Can become a swamp without ownership |
| Data fabric | Central, metadata-driven | Automated integration via active metadata | Tool-centric; ownership question remains |
| Data mesh | Domains, federated rules | Scales with organisation, better semantics | Needs platform maturity and domain skills |
These are not exclusive. A mesh is commonly built on a lakehouse; it changes who owns which tables, not where the bytes live. For a comparison with the data fabric idea, see Data Fabric vs Data Mesh.
A mesh tends to make sense when there are many domains with distinct data, a central data team is visibly the bottleneck, and domains have (or can hire) engineering capacity. It tends not to make sense for a company with one product and a five-person data team, where a well-run central warehouse is simpler and cheaper. A common path is incremental: start with two or three domains that are motivated, build the platform capabilities they need, write the first global rules, then expand.
A mesh changes job descriptions as much as architecture. Each domain appoints a data product owner, who decides what the product offers, talks to consumers, and prioritises fixes, plus engineers who build and run it. The platform team treats domain teams as its customers and measures itself by how quickly a domain can ship a new, compliant product. The governance group is small and cross-functional: domain representatives, platform, security and legal. It decides global standards and turns them into automated checks. A useful health metric for the whole mesh is lead time: how many days from "we need this dataset" to a published, documented, access-controlled product.