Apache Paimon (formerly Flink Table Store) is a streaming-first open table format that originated in the Apache Flink community and graduated to a top-level Apache project in 2024. Where Hudi, Iceberg, and Delta evolved from batch-first roots, Paimon was designed from day one for high-frequency CDC and real-time ingest, using an LSM-tree storage layer rather than the snapshot-of-Parquet-files model.
UPSERT and DELETE semantics by primary key, like a database.A Paimon table lives in a filesystem or object store as data files (Parquet or ORC by default) plus snapshot, manifest and schema metadata. For primary-key tables, each bucket is organised as a log-structured merge tree. Incoming records are buffered and sorted in memory, flushed as sorted runs at level 0, and merged by background compaction into larger files at higher levels. A read merges the runs by key and applies a merge engine: deduplicate (last row wins, the default), partial-update (fill columns from several streams), aggregation (sum, max and similar per column) or first-row. Snapshots are committed at Flink checkpoints, which provides exactly-once semantics.
The LSM design is what separates Paimon from snapshot-of-files formats. Updating a few rows in a Parquet-based table means either rewriting whole files or writing delete files that every reader must reconcile. In Paimon, an update is just another sorted record in a new small file, and the merge cost is paid incrementally by compaction, which can run inside the writer job or as a dedicated compaction job.
Paimon has two main table types. Primary-key tables support upserts and deletes and can produce a changelog for downstream consumers; the changelog-producer option (none, input, lookup or full-compaction) controls how complete before and after images are generated. Append tables have no primary key and behave like an ordinary log or fact table, optimised for high-throughput inserts and batch scans. Partitioning works like Hive partitions; inside each partition, buckets bound the parallelism and file count.
CREATE CATALOG paimon WITH (
'type' = 'paimon',
'warehouse' = 's3://lake/paimon'
);
USE CATALOG paimon;
CREATE TABLE orders (
order_id BIGINT,
customer_id BIGINT,
status STRING,
amount DECIMAL(12, 2),
dt STRING,
PRIMARY KEY (order_id, dt) NOT ENFORCED
) PARTITIONED BY (dt) WITH (
'bucket' = '4',
'merge-engine' = 'deduplicate',
'changelog-producer' = 'lookup'
);
-- Continuous CDC ingest from a MySQL CDC source table defined elsewhere
INSERT INTO orders SELECT order_id, customer_id, status, amount, dt FROM mysql_orders;
-- Streaming read of changes, or a batch query in another session
SELECT status, COUNT(*) FROM orders GROUP BY status;Because the primary key must include partition columns, dt appears in the key. In streaming mode the final query maintains a continuously updated result from the changelog; in batch mode it reads the latest snapshot. Spark, Trino and other engines can query the same table through their Paimon connectors.
| Need | Paimon | Alternative |
|---|---|---|
| High-rate CDC upserts with Flink | strong fit: LSM merges are cheap | Hudi MoR |
| Streaming changelog for downstream jobs | built in | Hudi incremental, Delta change data feed |
| Multi-engine batch analytics, broad catalog support | supported but younger ecosystem | Iceberg |
| Databricks-centric platform | limited | Delta Lake |
| Append-only logs | append tables work | any format |
Paimon is most compelling when Flink is already the processing engine and tables must stay fresh within minutes or seconds while being updated by key. Teams centred on Spark batch, or needing the widest engine and catalog support, usually choose Iceberg or Delta.