The list below describes the main tools used to move high volumes of data from sources into analytical systems. They are not interchangeable: some transport events, some transform them, and some route and manage flows. The sections after the list place each tool in a reference architecture, compare them, and show a minimal pipeline.
Most large ingestion systems separate three concerns. A durable event log, such as Kafka, Kinesis, Event Hubs, or Pulsar, decouples producers from consumers: producers append at their own rate, and consumers read at theirs, replaying from an offset when they fail or when a new consumer is added. Collectors bring data into the log, for example Debezium reading database change logs, NiFi pulling from files and APIs, or agents shipping logs. Processing engines, such as Flink or Spark Structured Streaming, read the log, clean and enrich records, and write to table formats in a lakehouse or to serving systems. Keeping the log in the middle is what allows multiple downstream systems to evolve independently.
| Tool | Role | Deployment | Typical use |
|---|---|---|---|
| Apache Kafka | Event log | Self-managed or managed | Backbone for event-driven systems |
| Amazon Kinesis | Event log | AWS managed | AWS-native streaming |
| Azure Event Hubs | Event log, Kafka-compatible API | Azure managed | Azure-native streaming |
| Apache Pulsar | Event log plus queues | Self-managed or managed | Multi-tenant, geo-replication |
| Apache Flink | Stateful stream processing | Self-managed or managed | Low-latency, event-time windows |
| Spark Structured Streaming | Micro-batch processing | Spark clusters, Databricks | Streaming into Delta tables |
| Google Cloud Dataflow | Managed Apache Beam runner | GCP managed | Unified batch and streaming |
| Apache NiFi | Flow routing and mediation | Self-managed | Many heterogeneous sources |
| Apache Storm | Record-at-a-time processing | Self-managed | Legacy real-time topologies |
from pyspark.sql import functions as F, types as T
schema = T.StructType([
T.StructField("order_id", T.StringType()),
T.StructField("amount", T.DoubleType()),
T.StructField("ts", T.TimestampType()),
])
raw = (spark.readStream.format("kafka")
.option("kafka.bootstrap.servers", "broker1:9092,broker2:9092")
.option("subscribe", "orders")
.option("startingOffsets", "latest")
.option("maxOffsetsPerTrigger", 500_000) # bound each micro-batch
.load())
orders = (raw.select(F.from_json(F.col("value").cast("string"), schema).alias("o"))
.select("o.*")
.withWatermark("ts", "10 minutes")
.dropDuplicates(["order_id", "ts"]))
(orders.writeStream.format("delta")
.option("checkpointLocation", "s3://lake/_chk/orders")
.trigger(processingTime="1 minute")
.toTable("bronze.orders"))The checkpoint stores consumed Kafka offsets and streaming state, so a restarted job continues where it stopped. Combined with Delta's transactional writes, each micro-batch is committed once, which gives effectively exactly-once results in the table.
Start from constraints rather than features. If the organization is committed to one cloud, the native managed log (Kinesis, Event Hubs, or Pub/Sub with Dataflow) removes cluster operations. If many teams and systems will share the stream, Kafka's ecosystem of connectors, schema registries, and client libraries is the safest default. Choose Flink when you need millisecond latency, large keyed state, or precise event-time semantics; choose Spark Structured Streaming when the destination is a lakehouse and the team already runs Spark for batch. Use NiFi when the hard part is the variety of sources and protocols rather than volume.