A streaming topology diagram shows how events move through a real-time system: which services produce them, how the Kafka topic is partitioned, which stream operators process them, and where results land. This sample counts orders per customer per minute with Apache Flink and writes the results to Kafka and an Iceberg table.
This is a topology diagram, not UML. UML covers software structure and behaviour; topology and dataflow graphs are among the other standard notations used alongside it. The shapes follow the logical job graph that Flink's web UI draws.
1. The Diagram
Producers publish order events with key=customer_id and acks=all, so a write is confirmed only after all in-sync replicas have it.
Kafka hashes the key to a partition, so any producer can write to any partition (the arrows end at the topic, not at one partition). All events for one customer land in one partition, which preserves their order. Each partition has a leader broker and two followers (replication factor 3).
KafkaSource runs three parallel instances, one per partition. Parsing and filtering are chained into the same task, so no network hop is needed.
keyBy(customer_id) (yellow) reshuffles events across the network so that each window instance owns a set of customers. The window operator can scale to six instances independently of the partition count.
Events arriving after the watermark has passed their window go to a side output instead of being silently dropped (red path).
Sinks commit through Flink's two-phase commit, so results appear exactly once. The checkpoint store holds offsets and window state together, which is what makes recovery consistent.
2. How to Read It
Always write the parallelism on each operator and the partition count on each topic: those two numbers decide throughput. Mark shuffles clearly, since they are where network cost and data skew appear.
3. Variant: Scaling a Consumer Group
Within one consumer group, each partition is read by exactly one consumer. With six partitions, three consumers get two each; eight consumers leave two idle. This diagram is the quickest way to explain why adding pods did not raise throughput: you must add partitions first (and adding them later reshuffles keys to new partitions). Flink sources follow the same rule; operators after a keyBy do not.
4. Pros, Cons, and When to Use
Pros
Exposes the numbers that matter: partitions, parallelism, replication.
Shows ordering guarantees (per key, per partition) visually.
Makes shuffles, side outputs and dead-letter paths explicit.
Matches the job graph in the Flink or Kafka Streams UI, so it is easy to verify.
Useful in capacity planning and incident reviews.
Cons
Static: lag, throughput and backpressure need dashboards.
Engine-specific detail (chaining, slots) can overwhelm business readers.
Hard to show time semantics (watermarks, windows) in boxes; add a timeline.
Gets busy quickly with many topics and joins.
Says little about schemas or data contracts on each topic.
When to use
Designing or reviewing a streaming job.
Explaining scaling limits and partitioning choices.
Documenting exactly-once guarantees end to end.
When not to use
Comparing overall architectures (use a Lambda/Kappa view).
Describing batch schedules (use a DAG).
5. As Diagrams-as-Code
Mermaid flowchart LR with one subgraph per system:
flowchart LR
subgraph Producers
checkout[checkout-svc]; mobile[mobile-api]; pos[pos-gateway]
end
subgraph Kafka["Kafka topic orders (3 partitions, RF=3)"]
p0[partition 0]; p1[partition 1]; p2[partition 2]
end
subgraph Flink["Flink job order-metrics"]
src["KafkaSource + parse (p=3)"]
win["Window agg 1 min (p=6)"]
late[late events side output]
end
Producers -- "murmur2(customer_id) % 3" --> Kafka
p0 & p1 & p2 --> src
src -- "keyBy(customer_id)" --> win
win -.-> late
win --> m[(Kafka order-metrics)]
win --> ice[(Iceberg on S3)]
late --> dlq[(Kafka orders-late)]