Apache Hudi

Apache Hudi (Hadoop Upserts Deletes and Incrementals) is an open-source data management framework that simplifies large-scale data ingestion and provides ACID transaction support on data lakes. It’s designed for scenarios that require efficient data upserts (updates and inserts) and deletes in big data environments, while also enabling near real-time ingestion and querying of data.

Key Features of Apache Hudi:

ACID Transactions: Hudi brings ACID transactions to data lakes, allowing users to perform updates, inserts, and deletes with transactional guarantees on large datasets.
Efficient Storage: Hudi optimizes storage by managing file sizes and using efficient compression techniques, reducing the overall storage footprint while maintaining performance.
Near Real-Time Data Processing: With Hudi, you can perform incremental data ingestion, which ensures that only changed data is processed, reducing the latency in making fresh data available for analytics.
Indexing and Compaction: Hudi maintains indexes on data to accelerate query performance and also supports background compaction to clean up older versions of data, improving performance over time.
Time Travel Queries: Hudi allows users to perform queries on historical data by maintaining versions of data, enabling "time travel" queries to access previous versions of datasets.

Use Cases:

Data lakes that require efficient upserts and deletes at scale.
Building near real-time data pipelines with low latency.
Historical data querying and analytics with time-travel capabilities.
Optimizing large-scale data lakes for cost-effective storage and performance.

Conclusion:

Apache Hudi is ideal for environments where frequent updates, incremental processing, and ACID guarantees are necessary on top of a scalable data lake. It bridges the gap between traditional batch processing systems and real-time analytics by enabling near real-time ingestion and querying, making it a powerful tool for modern data architectures.

Architecture

A Hudi table is a directory of data files plus a .hoodie metadata folder holding the timeline: an ordered log of actions (commits, delta commits, compactions, cleans, rollbacks) each with a state of requested, inflight or completed. Readers only see data from completed instants, which provides snapshot isolation. Data is organised into file groups, each identified by a file id and holding all versions of a set of record keys. Hudi maintains an index (Bloom filters in file footers, a simple key join, HBase, or the metadata-table record index) that maps each incoming record key to its file group, so an upsert touches only the files that actually contain those keys.

Hudi also bundles table services that other formats leave to external jobs: clustering (re-sorting and resizing files), cleaning (removing file versions beyond a retention window), compaction and an internal metadata table that stores file listings and column statistics to avoid slow object-store listings.

Table Types

In a Copy-on-Write table, an update rewrites the affected Parquet base file with the changed records merged in. Reads are plain Parquet scans. In a Merge-on-Read table, updates are appended to row-based log files beside the base file and merged during reads (snapshot queries) or periodically by compaction into a new base file. MoR tables also offer a read-optimised query that reads only compacted base files, trading freshness for speed. Hudi also exposes incremental queries that return only records changed after a given instant, which is the foundation for chaining low-latency pipelines.

Example: Upserts with PySpark

# spark-submit with the hudi-spark bundle matching your Spark version
from pyspark.sql import SparkSession
spark = (SparkSession.builder
         .config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
         .getOrCreate())

opts = {
    "hoodie.table.name": "orders",
    "hoodie.datasource.write.recordkey.field": "order_id",
    "hoodie.datasource.write.precombine.field": "updated_at",
    "hoodie.datasource.write.partitionpath.field": "order_date",
    "hoodie.datasource.write.table.type": "MERGE_ON_READ",
    "hoodie.datasource.write.operation": "upsert",
}
path = "s3://lake/hudi/orders"

updates = spark.read.json("s3://landing/orders/2026-09-30/")
updates.write.format("hudi").options(**opts).mode("append").save(path)

# Incremental read: only rows committed after a given instant
changes = (spark.read.format("hudi")
           .option("hoodie.datasource.query.type", "incremental")
           .option("hoodie.datasource.read.begin.instanttime", "20260930000000")
           .load(path))
changes.show()

The record key identifies a row, the precombine field decides which version wins when the same key appears more than once (largest value wins), and the partition path sets the directory layout. Spark SQL also supports MERGE INTO, UPDATE and DELETE on Hudi tables, and time travel via TIMESTAMP AS OF.

Hudi vs Other Table Formats

Format	Strength	Typical fit
Apache Hudi	record-level upserts, built-in indexing, incremental pulls, table services	CDC and streaming ingestion into a lake
Delta Lake	transaction log, tight Spark and Databricks integration	Databricks-centric lakehouses
Apache Iceberg	engine-neutral spec, hidden partitioning, broad catalog support	multi-engine analytics platforms
Apache Paimon	LSM-tree primary-key tables, Flink-native streaming	high-frequency CDC with Flink

Pitfalls

Table services must run. Without compaction MoR log files grow and reads slow down; without cleaning, storage grows. Run them inline or as separate jobs.
Index choice matters. Bloom indexes slow down when keys are random across partitions; consider the record-level index for large tables.
Bundle versions. The Hudi Spark bundle must match the Spark and Scala versions in use.
Many config knobs. Start from documented defaults and change only what you can measure.

Apache Iceberg, Delta Lake, Apache Paimon
Apache XTable for cross-format metadata
Table formats index