Every data pipeline decides how long to wait before computing an answer. Batch waits for a complete chunk of data; streaming answers continuously as events arrive. This page explains the trade-off between latency, completeness and cost, how streaming engines reason about time, and when micro-batch is the sensible middle.
The diagram shows the same twelve events handled two ways. Batch processing collects data for a bounded period (an hour, a day), then runs a job over the whole set at once. The input is finite: the job knows when it has seen everything, so it can sort, join and aggregate with full information, then stop. Stream processing treats data as an unbounded sequence of events and updates results continuously as each event (or small group of events) arrives. The job never "finishes"; it runs as a long-lived service that holds state such as running counts or open windows.
A useful mental model from the Dataflow paper is that batch is just streaming over a bounded input. Both compute the same function of the data; they differ in when results are produced and how sure the system is that a result is final.
The choice drives four properties that pull against each other:
Most business reporting tolerates hours of delay, so batch remains the default. Streaming earns its cost when a decision loses value quickly: fraud scoring, alerting, inventory, personalisation, operational dashboards.
The hard part of streaming is time. Event time is when something happened (the timestamp in the record); processing time is when the system sees it. Networks, mobile devices going offline, and retries make these diverge, so events arrive out of order.
Windowed aggregations (tumbling, sliding, session windows) are defined on event time. To know when a window can be closed, engines track a watermark: an estimate that "no events older than time W are still expected". A common heuristic is W = max event time seen − allowed lateness. When the watermark passes the end of a window, the engine emits the final result and discards that window's state. Events arriving after that are late: they can be dropped, sent to a side output, or used to emit a correction.
This is the completeness trade-off in concrete form: a larger allowed lateness gives more complete results but higher latency and more state; a smaller one gives fast, possibly incomplete answers. Batch avoids the question by waiting long enough that lateness rarely matters.
Delivery guarantees are the other mechanism. Streaming engines checkpoint their input offsets and state together; on failure they restart from the last checkpoint. Combined with an idempotent or transactional sink, this gives effectively exactly-once results even though some records are physically re-read.
Latency is a spectrum rather than a binary. Micro-batch engines, notably Spark Structured Streaming in its default mode, run a small batch job every few seconds or minutes over whatever arrived since the last trigger. This reuses the batch engine (and its optimiser) while giving latency in the seconds range. Record-at-a-time engines such as Apache Flink and Kafka Streams process each event as it arrives and can reach lower latency, at the cost of a different programming and operational model.
A practical pattern is the incremental batch: a scheduled job that processes only new data since its last run (tracked by a high-water mark or a stream checkpoint) and then stops. Spark's trigger(availableNow=True) does exactly this, letting the same streaming code run on a schedule when sub-minute freshness is not needed.
Suppose an online shop wants revenue per 5-minute window. The batch version runs once a day on a closed partition; the streaming version runs continuously with a 10-minute lateness tolerance.
# Same aggregation written two ways with PySpark: revenue per 5-minute window.
from pyspark.sql import SparkSession, functions as F
spark = SparkSession.builder.appName("batch-vs-stream").getOrCreate()
schema = "order_id STRING, amount DOUBLE, event_ts TIMESTAMP"
# --- Batch: read a closed day of files, compute, overwrite the partition ---
batch = spark.read.schema(schema).json("s3://shop/orders/dt=2026-09-30/")
(batch.groupBy(F.window("event_ts", "5 minutes"))
.agg(F.sum("amount").alias("revenue"))
.write.mode("overwrite").parquet("s3://shop/revenue_5m/dt=2026-09-30/"))
# --- Streaming: same logic, but incremental and stateful ---
stream = (spark.readStream.schema(schema).json("s3://shop/orders/incoming/")
.withWatermark("event_ts", "10 minutes")) # tolerate 10 min lateness
query = (stream.groupBy(F.window("event_ts", "5 minutes"))
.agg(F.sum("amount").alias("revenue"))
.writeStream
.outputMode("append") # emit a window once it is final
.option("checkpointLocation", "s3://shop/_chk/revenue_5m/")
.trigger(processingTime="30 seconds") # micro-batch every 30 s
.format("parquet").option("path", "s3://shop/revenue_5m_stream/")
.start())Notes on the code: the aggregation logic is identical. The streaming version adds three things: a watermark (how long to wait for stragglers), a checkpoint location (where offsets and window state are saved for recovery), and a trigger (how often to run a micro-batch). With outputMode("append"), a window is written only once the watermark passes it, so downstream readers never see partial values; "update" would instead emit changing values as events arrive.
Many teams run both: the stream powers a live dashboard, and a nightly batch recomputes the day from raw data to produce the authoritative figure, correcting anything the stream dropped as too late. This is the essence of the Lambda architecture; the Kappa architecture instead replays the log through the same streaming job when recomputation is needed.
| Aspect | Batch | Micro-batch | Continuous streaming |
|---|---|---|---|
| Typical latency | Minutes to a day | Seconds to minutes | Sub-second to seconds |
| Input | Bounded (files, partitions) | Unbounded, read in chunks | Unbounded, per event |
| Completeness | High: waits for late data | Watermark-dependent | Watermark-dependent |
| Cost profile | Run, then shut down | Always-on, modest | Always-on, sized for peak |
| Failure recovery | Re-run the job | Checkpoint replay | Checkpoint/snapshot replay |
| Typical tools | SQL warehouse, Spark, dbt, Airflow | Spark Structured Streaming | Flink, Kafka Streams |
| Good for | Reporting, ML training sets, backfills | Near-real-time tables, CDC | Fraud, alerting, event-driven apps |
A short decision rule: start with batch; move to incremental batch when runs get too long; move to streaming only when a named consumer needs fresher data and the value of that freshness is greater than the extra always-on cost.