Event Time, Windows and Watermarks

Every streaming aggregation has to answer two questions: which events belong together, and when is a result final? This page explains event time, the main window types and watermarks, with Spark and Flink SQL examples.

1. What These Terms Mean

Stream processors compute results over data that never ends: clicks, payments, sensor readings. To produce an answer such as "orders per minute", the unbounded stream must be cut into finite chunks called windows. Doing that correctly requires distinguishing two clocks:

• Event time — when the thing actually happened, stamped by the device or source system.
• Processing time — when the stream processor sees the record.

A watermark is the processor's running estimate of event-time progress: a declaration that "no more events older than time W are expected". When the watermark passes the end of a window, the window can be finalised and emitted. Events that arrive after that point are late data.

2. Why It Matters

Networks delay, mobile phones go offline, upstream batches are replayed, and partitions are consumed at different speeds. If you aggregate by processing time, a ten-minute network outage moves all its events into the wrong minute, and replaying yesterday's data gives different answers from running it live. Aggregating by event time gives correct, reproducible results — but then the system must decide how long to wait for stragglers. Watermarks are that decision made explicit: a tunable trade-off between latency (emit early) and completeness (wait for late events).

3. How It Works

Each event is drawn twice: once at the time it happened and once at the time it arrived. Most arrive shortly after they occur, but e3 was created on a phone with no signal and arrives after e4 and e5. The gap between the two lines is called skew or lag, and it is neither constant nor bounded.

• Tumbling windows are fixed-size, non-overlapping intervals: every event belongs to exactly one. Use them for "per minute" or "per hour" reports.
• Sliding (hopping) windows have a size and a slide; with size 5 minutes and slide 2.5 minutes each event falls in two windows. Use them for moving averages and "last 5 minutes, updated every 30 seconds" alerts.
• Session windows group bursts of activity per key, closing after a gap of inactivity. Their boundaries depend on the data, and two sessions can merge when a late event fills the gap between them.
• Global windows with custom triggers cover everything else, such as "every 100 events".

A common watermark strategy is bounded out-of-orderness: watermark = maximum event time seen so far minus a fixed delay, for example 30 seconds. When the watermark crosses 12:05, the window [12:00, 12:05) fires. The event stamped 12:03 that shows up at 12:09 is late. Engines offer three policies: drop it; route it to a side output for separate handling; or keep window state for an allowed lateness period and emit an updated result. Downstream consumers must then handle corrections, which is why sinks for updating streams are usually upserts keyed by window.

In parallel jobs each source partition has its own watermark and an operator uses the minimum across its inputs, so one idle or slow partition holds everything back. Flink therefore lets you mark sources idle (withIdleness).

4. Worked Example

Spark Structured Streaming expresses event-time windows with window() and the watermark with withWatermark(). This runs locally with the built-in rate source.

from pyspark.sql import SparkSession, functions as F

spark = SparkSession.builder.appName("windows").getOrCreate()

events = (spark.readStream.format("rate").option("rowsPerSecond", 20).load()
          # simulate device timestamps up to 20 s in the past (out of order)
          .withColumn("event_time",
                      F.col("timestamp") - F.expr("make_interval(0,0,0,0,0,0, rand()*20)"))
          .withColumn("user", (F.col("value") % 5).cast("string")))

per_window = (events
    .withWatermark("event_time", "30 seconds")             # tolerate 30 s lateness
    .groupBy(F.window("event_time", "1 minute", "30 seconds"), "user")  # sliding
    .count())

sessions = (events
    .withWatermark("event_time", "30 seconds")
    .groupBy(F.session_window("event_time", "10 seconds"), "user")
    .count())

q = (per_window.writeStream.outputMode("append")   # emit once watermark passes window end
     .format("console").option("truncate", False).start())
q.awaitTermination(120)

With outputMode("append") each window is printed once, after the watermark passes its end; events older than the watermark are dropped silently. Switching to "update" prints running counts every micro-batch. The equivalent in Flink SQL:

CREATE TABLE clicks (
  user_id STRING,
  ts TIMESTAMP(3),
  WATERMARK FOR ts AS ts - INTERVAL '30' SECOND
) WITH ('connector' = 'kafka', 'topic' = 'clicks', 'format' = 'json',
        'properties.bootstrap.servers' = 'localhost:9092');

SELECT window_start, window_end, user_id, COUNT(*) AS clicks
FROM TABLE(TUMBLE(TABLE clicks, DESCRIPTOR(ts), INTERVAL '1' MINUTE))
GROUP BY window_start, window_end, user_id;

5. Choosing the Watermark Delay

There is no universal right delay; it comes from your data. Start by measuring lag: for a day of events, compute processing time minus event time and look at the distribution. If the 99th percentile is 8 seconds and the 99.9th is 2 minutes, a 10-second delay drops about one event in a hundred, while a 2-minute delay drops one in a thousand at the cost of emitting every window two minutes later. Decide which matters more for the consumer. A fraud alert wants low latency and can tolerate a few missed events; a billing report wants completeness and can wait.

A common design uses both: a fast streaming result with a tight watermark for dashboards and alerts, and a periodic batch job over the same events, run hours later, that produces the authoritative numbers. This is sometimes called a lambda-style correction. With allowed lateness and upsert sinks, a single streaming job can often do both jobs.

Finally, publish the metrics: watermark lag behind wall-clock time, number of late events dropped, and state size. These three numbers tell you whether the delay is still right as traffic and devices change.

6. Comparison

7. Pitfalls

• Watermark too tight. Many valid events are dropped as late; monitor the late-event count.
• Watermark too loose. Results are delayed and state grows; hours of lateness means hours of buffered windows.
• Idle partitions. One quiet Kafka partition stalls the watermark for the whole job.
• Bad device clocks. Timestamps in the future push the watermark forward and make everything else late; clamp or reject them.
• Processing-time windows by accident. Using arrival time because it is convenient makes results non-reproducible.
• Append sinks for updating results. Re-emitted windows must overwrite, not duplicate.

8. Related Topics

• Kafka Producer and Consumer — the usual event source
• Partitioning and Sharding — partitions and per-partition watermarks
• Real-Time Embedding Pipeline — a streaming system design
• Data Pipelines — batch and streaming pipelines

9. Further Reading

• Akidau et al. (2015), The Dataflow Model, VLDB — the paper that defined windows, watermarks and triggers.
• Akidau, Chernyak and Lax (2018), Streaming Systems, O'Reilly.
• Apache Flink documentation, "Timely Stream Processing" and "Generating Watermarks".
• Spark Structured Streaming Programming Guide, "Window Operations on Event Time".
• Kleppmann (2017), Designing Data-Intensive Applications, chapter 11.