Apache Flink is the most widely used open-source engine for stateful stream processing. This page explains its architecture, its two APIs, how it manages time and state, and how checkpoints give exactly-once results.
Apache Flink is an open-source engine for stateful computation over unbounded and bounded data streams. "Unbounded" means the input never ends — a Kafka topic of clicks, payments or sensor readings. Flink processes each event as it arrives, keeps whatever memory the computation needs (counts, windows, joins in progress) as managed state, and periodically snapshots that state so it can recover after a crash without losing or double-counting results.
A Flink cluster has a JobManager that accepts jobs, turns them into a graph of parallel tasks, schedules those tasks and coordinates checkpoints, and one or more TaskManagers, worker JVMs that each offer a number of task slots. Each slot runs one parallel slice of the pipeline. Records move between slots over the network with credit-based flow control, so a slow operator naturally slows its upstream (backpressure) rather than running out of memory.
Batch pipelines answer questions hours later. Many problems need answers within seconds: fraud scoring on a card swipe, alerting on a failing sensor, updating a live dashboard, or keeping a search index in sync with a database. Flink is built for these cases. Its distinguishing features are true event-at-a-time processing (rather than micro-batches), first-class event time with watermarks so late and out-of-order events are handled correctly, large keyed state that can spill to local disk via RocksDB, and exactly-once state consistency through distributed snapshots. With transactional sinks, that guarantee can extend end-to-end.
Flink also treats batch as a special case of streaming: a bounded input is simply a stream that ends. The same SQL query or DataStream program can backfill a year of history from files and then continue on live Kafka data, which removes the need to maintain separate batch and streaming code paths for the same metric. Common production uses include real-time ETL into lakehouse tables, continuous aggregation for dashboards, complex event processing (detecting patterns such as three failed logins followed by a password reset), and enrichment joins between a fast event stream and a slowly changing reference table.
The DataStream API (Java, with a Python version in PyFlink) gives full control: sources, transformations such as map and filter, keyBy to partition by key, windows, timers and process functions with explicit state. The diagram shows a typical shape: parse events, partition by user so that all of a user's events reach the same parallel instance, aggregate in windows, then write results.
The Table API and Flink SQL sit above it. You declare tables over Kafka topics or files and write standard SQL; the planner turns it into the same kind of operator graph. A GROUP BY on a stream produces a continuously updating result (a changelog), and windowing is expressed with table-valued functions like TUMBLE and HOP. Most new pipelines start in SQL and drop to DataStream only for custom logic.
Time and watermarks. Event time is when something happened; processing time is when Flink sees it. A watermark with timestamp t asserts "no more events older than t are expected." A window [12:00, 12:05) fires when the watermark passes 12:05. Watermarks are usually generated as max_event_time − allowed_delay; a larger delay tolerates more disorder at the cost of latency.
Flink's fault tolerance is based on the Chandy-Lamport distributed snapshot algorithm, adapted as asynchronous barrier snapshotting. The JobManager periodically injects a numbered barrier into every source. Barriers travel with the data. When an operator has received barrier n on all its inputs, its state reflects exactly the events before the barrier, so it snapshots that state to durable storage and passes the barrier on. When every operator has acknowledged, checkpoint n is complete, and the source offsets recorded with it mark where to resume.
After a failure, Flink restarts the job, restores every operator's state from the last completed checkpoint, and rewinds sources such as Kafka to the stored offsets. Events after the checkpoint are replayed, and because state was rolled back to match, each event affects state exactly once. Savepoints are the same mechanism triggered manually, used for upgrades, rescaling and migrations. State backends are either heap memory (fast, limited by RAM) or embedded RocksDB (larger than memory, supports incremental checkpoints).
This job reads JSON orders from Kafka, computes revenue per user per one-minute tumbling window in event time, and writes results to another topic. Run it in the Flink SQL client with the Kafka connector JAR on the classpath.
CREATE TABLE orders (
order_id STRING,
user_id STRING,
amount DECIMAL(10, 2),
order_time TIMESTAMP(3),
WATERMARK FOR order_time AS order_time - INTERVAL '10' SECOND
) WITH (
'connector' = 'kafka',
'topic' = 'orders',
'properties.bootstrap.servers' = 'localhost:9092',
'properties.group.id' = 'flink-revenue',
'scan.startup.mode' = 'earliest-offset',
'format' = 'json'
);
CREATE TABLE revenue_per_minute (
user_id STRING,
window_start TIMESTAMP(3),
window_end TIMESTAMP(3),
revenue DECIMAL(12, 2),
orders BIGINT
) WITH (
'connector' = 'kafka',
'topic' = 'revenue_per_minute',
'properties.bootstrap.servers' = 'localhost:9092',
'format' = 'json'
);
SET 'execution.checkpointing.interval' = '30s';
INSERT INTO revenue_per_minute
SELECT user_id, window_start, window_end, SUM(amount), COUNT(*)
FROM TABLE(TUMBLE(TABLE orders, DESCRIPTOR(order_time), INTERVAL '1' MINUTE))
GROUP BY user_id, window_start, window_end;The WATERMARK clause makes order_time the event-time attribute and tolerates ten seconds of disorder. Each window emits once, when the watermark passes its end, so the output is append-only. The equivalent in PyFlink's DataStream API would use key_by(lambda o: o.user_id), window(TumblingEventTimeWindows.of(Time.minutes(1))) and a reduce function.
| Engine | Model | State | Typical latency | Best fit |
|---|---|---|---|---|
| Apache Flink | Event-at-a-time, event time native | Large keyed state, RocksDB, exactly-once | Milliseconds | Complex stateful streaming, CEP, streaming SQL |
| Spark Structured Streaming | Micro-batch (default) | State store, exactly-once | Sub-second to seconds | Teams already on Spark, unified batch and stream |
| Kafka Streams | Library inside your app | Local RocksDB backed by Kafka changelogs | Milliseconds | Kafka-to-Kafka microservices, no cluster to run |
| Managed (Kinesis Data Analytics / Amazon Managed Service for Apache Flink) | Hosted Flink | Same as Flink | Milliseconds | Flink without cluster operations |
GROUP BY or join keeps state forever. Set state TTL (table.exec.state.ttl) or use windowed operations.