"Exactly-once" is the most misunderstood guarantee in data streaming. Networks drop acknowledgements and processes crash, so messages are inevitably delivered more than once. What systems actually provide is exactly-once effect: each message changes the result once, even if it was processed several times. This page explains the three delivery guarantees, the two techniques that make exactly-once possible (idempotence and transactions), how Kafka and Flink implement them, and where the guarantee stops.
A consumer does two things for each message: it applies an effect (writes to a database, produces an output, updates state) and it records progress (commits its offset). The order of those two steps decides the guarantee. Commit first and a crash loses the message: at-most-once. Process first and a crash means the message is replayed: at-least-once, with possible duplicates. Exactly-once requires the effect and the progress record to succeed or fail together, or the effect to be harmless when repeated.
Duplicates are harmless for some workloads (refreshing a cache with the latest value) and disastrous for others: double-charging a card, double-counting revenue, sending two emails, or inflating an aggregate that drives a business decision. Losses are worse because they are invisible. Many teams discover the problem only during an incident, because duplicates and losses appear exactly when brokers fail over or consumers restart, which is rare in testing and common in production.
Two failure points create duplicates. On the producer side, a broker writes a message but the acknowledgement is lost; the producer retries and the message is stored twice. On the consumer side, the consumer applies the effect and crashes before committing the offset; after restart it resumes from the last committed offset and applies the effect again. Neither can be prevented by "being careful"; they follow from the fact that a process cannot know whether a remote action completed when the reply never arrived (the Two Generals problem).
There are two building blocks, and real systems combine them.
Idempotence means applying an operation twice has the same result as once. "Set balance to 110" is idempotent; "add 10 to balance" is not. A sink becomes idempotent if every message carries a unique ID and the write is an upsert or an insert that ignores duplicate keys. Kafka's idempotent producer uses the same idea internally: each producer has an ID and each message a sequence number, so the broker discards retried duplicates.
Transactions make several writes atomic. If the output records and the consumer's offset commit are part of one transaction, a crash either leaves both or neither, and a replay starts exactly where the last committed transaction ended. This works when the outputs and the offsets live in the same transactional system, for example Kafka topics, or a database table that stores both results and offsets.
Kafka introduced exactly-once semantics (EOS) in version 0.11 (2017). It has three parts: the idempotent producer (no duplicates from retries within a partition), transactions (a producer with a transactional.id can write to multiple partitions and commit consumer offsets atomically), and read_committed consumers (they skip records from aborted transactions). The transactional.id also enables fencing: if a zombie instance of the same application wakes up after being replaced, its transactions are rejected. Kafka Streams wraps all this behind one setting, processing.guarantee=exactly_once_v2.
A transactional read-process-write loop with the confluent-kafka Python client:
# pip install confluent-kafka -- transactional read-process-write loop
from confluent_kafka import Consumer, Producer, KafkaException
consumer = Consumer({
"bootstrap.servers": "localhost:9092",
"group.id": "order-enricher",
"enable.auto.commit": False, # offsets go through the transaction
"isolation.level": "read_committed", # never read aborted data
})
producer = Producer({
"bootstrap.servers": "localhost:9092",
"transactional.id": "order-enricher-1", # stable per instance; enables fencing
"enable.idempotence": True,
})
consumer.subscribe(["orders"])
producer.init_transactions()
while True:
batch = consumer.consume(num_messages=100, timeout=1.0)
if not batch:
continue
producer.begin_transaction()
try:
for msg in batch:
if msg.error():
raise KafkaException(msg.error())
enriched = msg.value().upper() # stand-in for real logic
producer.produce("orders-enriched", key=msg.key(), value=enriched)
producer.send_offsets_to_transaction(
consumer.position(consumer.assignment()),
consumer.consumer_group_metadata())
producer.commit_transaction() # outputs + offsets, atomically
except Exception:
producer.abort_transaction() # nothing becomes visible
# rewind to last committed offsets and retry the batch
for tp in consumer.committed(consumer.assignment()):
consumer.seek(tp)When the sink is a database instead of Kafka, Kafka transactions cannot cover it. Make the sink idempotent instead, using an event ID as a key:
-- Idempotent sink: the same event applied twice leaves one row.
CREATE TABLE payments (
event_id TEXT PRIMARY KEY, -- unique id carried by every message
account_id TEXT NOT NULL,
amount NUMERIC(12,2) NOT NULL,
applied_at TIMESTAMP DEFAULT now()
);
INSERT INTO payments (event_id, account_id, amount)
VALUES ('evt-42', 'acct-7', 10.00)
ON CONFLICT (event_id) DO NOTHING; -- replay after a crash is a no-op
-- Balances are derived, never incremented in place:
SELECT account_id, SUM(amount) AS balance FROM payments GROUP BY account_id;The second pattern is the most portable way to get exactly-once effect into any external system: at-least-once delivery plus an idempotent write. It also makes manual replays and backfills safe.
| Approach | Guarantee | Cost | Works with |
|---|---|---|---|
| Auto-commit consumer | At-most-once or at-least-once (timing dependent) | Lowest | Anything; unsafe for money or counts |
| Manual commit after processing | At-least-once | Low | Any sink; duplicates on failure |
| At-least-once + idempotent sink | Exactly-once effect | Low to moderate | Databases with keys, object stores with deterministic names |
| Kafka transactions (EOS) | Exactly-once within Kafka | Some latency and throughput overhead | Kafka to Kafka, Kafka Streams |
| Flink checkpoints + two-phase commit sinks | End-to-end exactly-once | Checkpoint interval adds latency | Kafka, transactional file and table sinks |
Apache Flink takes consistent snapshots of all operator state and source offsets using the Chandy-Lamport style barrier algorithm. On failure it restores the last checkpoint and replays from the saved offsets, so internal state is exactly-once. For the sink, Flink uses a two-phase commit: writes are pre-committed during a checkpoint and committed only when the checkpoint completes. Spark Structured Streaming achieves similar end-to-end guarantees with replayable sources, checkpointed offsets and idempotent or transactional sinks such as Delta Lake. In both, output becomes visible only at checkpoint boundaries, which is the latency you pay for the guarantee.
A practical checklist when building a pipeline that must not lose or duplicate data. First, give every event a globally unique, deterministic ID at the source, ideally derived from the business transaction rather than generated at send time. Second, enable the idempotent producer; in recent Kafka clients it is on by default. Third, decide where the end-to-end boundary is: if everything stays in Kafka, use transactions or Kafka Streams with exactly_once_v2; if the result lands in a database or object store, use at-least-once delivery with an idempotent, keyed write. Fourth, store derived values (balances, counters) as recomputable aggregates over deduplicated events rather than in-place increments. Finally, add a uniqueness check on the event ID in the sink as a safety net and alert when it fires; it tells you the guarantee broke somewhere.
read_committed downstream.