How do you serve answers that are both complete and fresh? Lambda runs a batch pipeline and a streaming pipeline side by side; Kappa runs only a stream over a replayable log. This page explains both, how each corrects the past, and how to choose.
Lambda and Kappa are two reference architectures for systems that must answer queries over both historical and very recent data. They were named in the early 2010s, when stream processors were fast but considered unreliable, and batch systems such as Hadoop MapReduce were reliable but slow.
Nathan Marz described the Lambda architecture (in his work on Apache Storm and the book Big Data) as running two pipelines side by side: a batch layer that periodically recomputes everything from an immutable master dataset, and a speed layer that processes new events immediately. Jay Kreps answered in 2014 with the Kappa architecture: keep an immutable log as the source of truth, run only a streaming pipeline, and when you need to recompute, replay the log.
In the Lambda diagram, every event goes both ways. The batch layer produces complete but stale views; the speed layer covers only the gap since the last batch run. A query merges both, for example summing yesterday's batch count with today's real-time count.
The choice decides how many copies of business logic you maintain, how you fix bugs in historical results, and how fresh your dashboards and features can be. Many modern stacks are neither pure Lambda nor pure Kappa, but the vocabulary is still the clearest way to reason about trade-offs, and the questions it raises (how do I correct the past? how many code paths compute this metric?) come up in every streaming design review.
Kappa depends on two properties of the log. First, it must be durable and replayable: Kafka topics with long or infinite retention, tiered storage, or a lakehouse table that can be read as a stream. Second, the stream processor must be able to rebuild state from scratch at a useful speed, which modern engines (Apache Flink, Spark Structured Streaming, Kafka Streams) do by running the same job with higher parallelism over historical offsets.
Reprocessing follows the steps in the diagram: deploy version 2 of the job reading from the beginning of the log into a new output table; wait until it reaches the head; point queries at the new table; stop version 1. No separate batch code exists, so there is nothing to keep in sync.
The idea that makes this correct is event-time processing. Results are grouped by when events happened, not when they arrived, and watermarks tell the engine how long to wait for stragglers. Exactly-once sinks (transactional writes or idempotent upserts) ensure that a replay does not double-count.
Late or corrected data is where the two designs feel most different. In Lambda the speed layer can afford to be approximate, because the next batch run will overwrite it with the exact answer. In Kappa the streaming job itself must handle lateness: windows stay open for an allowed lateness, emit updates when late events arrive, and sinks must accept upserts. For a logic bug, Lambda reruns the batch job; Kappa replays the log.
Suppose we need page views per page per hour. Under Kappa, one Spark Structured Streaming job does the whole thing. The same code computes history (start from the earliest offset) and live data.
# pip install pyspark (plus the spark-sql-kafka package for your Spark version)
from pyspark.sql import SparkSession, functions as F
spark = SparkSession.builder.appName("pageviews-v2").getOrCreate()
events = (spark.readStream.format("kafka")
.option("kafka.bootstrap.servers", "localhost:9092")
.option("subscribe", "pageviews")
.option("startingOffsets", "earliest") # replay for a new version
.load()
.select(F.from_json(F.col("value").cast("string"),
"page STRING, ts TIMESTAMP").alias("e"))
.select("e.*"))
hourly = (events
.withWatermark("ts", "2 hours") # accept events up to 2h late
.groupBy(F.window("ts", "1 hour"), "page")
.count())
(hourly.writeStream
.outputMode("update")
.format("console") # use a table sink in practice
.option("checkpointLocation", "/tmp/chk/pageviews-v2")
.start()
.awaitTermination())Under Lambda you would write this aggregation twice: once as a nightly SQL job over the raw event archive, and once as the streaming job. The query layer would read the batch table for hours before the last batch cutoff and the streaming table after it. Every change to the definition of a "view" must be made, tested and deployed in both places.
| Aspect | Lambda | Kappa | Lakehouse streaming (common today) |
|---|---|---|---|
| Code paths | Two (batch + stream) | One (stream) | One, run as batch or stream |
| Source of truth | Master dataset in a lake | Replayable log | Table format (Delta, Iceberg, Hudi) |
| Reprocessing | Rerun batch job | Replay log into new table | Rerun over the table, or time travel |
| Late data | Fixed by next batch | Watermarks + upserts | MERGE / upserts |
| Operational load | Two systems to run | Log retention and replay capacity | One engine, table maintenance |
| Good fit | Legacy Hadoop + stream stacks, heavy ML recompute | Event-driven products, CDC, per-event logic | Analytics teams wanting both freshness and SQL |
The third column reflects a common modern pattern: engines like Spark and Flink can treat a table as both a batch input and a streaming source, so one code path can serve both roles without keeping a whole history in Kafka.
Start by asking how often historical results must be recomputed and how expensive that is. If recomputation is rare and your streaming engine can replay the history at an acceptable speed, Kappa removes an entire code path and is usually the better default for new systems. If recomputation is heavy (for example retraining features over years of data with complex joins that are cheaper in a warehouse), or the team already runs a mature batch platform, a Lambda-style split can be pragmatic, as long as both paths share definitions.
Also consider freshness requirements honestly. Many "real-time" dashboards are fine with a five-minute micro-batch over a lakehouse table, which avoids most of the complexity of either architecture. Reserve true per-event streaming for cases where the latency changes a decision: fraud checks, alerting, personalisation, operational control loops.