Stream processing turns data pipelines from nightly jobs into continuously updating systems. This section covers the event log, the processing engines and the concepts — time, windows, state and guarantees — that tie them together.
Stream processing means computing over data continuously as it arrives rather than in scheduled batches. Instead of "load yesterday's orders at 2 a.m. and aggregate them", a streaming pipeline updates totals within seconds of each order. The input is treated as unbounded: there is no last row, so results are produced incrementally and the system must decide when a window of time is complete enough to emit an answer.
Most streaming platforms share the shape above. Producers — application events, change data capture (CDC) from OLTP databases, devices and logs — append to a durable, partitioned event log such as Apache Kafka or Amazon Kinesis. The log decouples producers from consumers and keeps data for replay. A stream processor reads the log, filters, joins, aggregates and enriches events while maintaining state, then writes results to lakehouse tables, low-latency serving stores, or alerting systems.
| Need | Good fit | Why |
|---|---|---|
| Complex stateful logic, low latency, event time | Apache Flink | Native event-at-a-time engine with large keyed state and exactly-once checkpoints |
| Team already on Spark / Databricks | Spark Structured Streaming | Same DataFrame API for batch and stream; micro-batch latency is often acceptable |
| Kafka-to-Kafka microservices | Kafka Streams | A library, not a cluster; state backed by Kafka topics |
| Managed AWS ingestion and delivery | Kinesis Data Streams and Firehose | Serverless, minimal operations; limited transformation |
| Hourly freshness is fine | Scheduled batch | Simpler to build, test and backfill |
The last row matters: streaming adds operational complexity, so choose it when freshness changes a decision, not by default. Many teams use incremental batch every few minutes as a middle ground.
Start with Batch vs Streaming to decide whether you need streaming at all, then the event log (Kafka producers and consumers) to understand partitions, offsets and consumer groups. Then learn CDC, because most business streams start as database changes. Read Event Time, Windows and Watermarks and Exactly-Once Semantics next, since every engine builds on those ideas. Then move to a processor: Spark Structured Streaming if you already know Spark, Flink if you need true low latency or complex state. Finish with where results land: lakehouse table formats for history and OLAP stores for live dashboards. Throughout, keep data quality in mind; a streaming bug corrupts data continuously rather than once a night.