Data Engineering

Data Flow Diagram: Order Systemdata engineeringGane-Sarson level 1 and level 0 DFDs of an online shop: customer, payment gateway, warehouse. Pipeline DAG Diagramdata engineeringAirflow-style DAG with extract, validate, dbt transform, load and notify, plus a failed run. Data Lineage Graphdata engineeringTable lineage from sources through dbt staging and marts to dashboards, plus column lineage. Lambda vs Kappa Diagramdata engineeringLambda's batch and speed layers next to Kappa's single replayable stream, plus reprocessing. Streaming Topology Diagramdata engineeringKafka partitions, Flink operators, a keyed window, late events and exactly-once sinks. Medallion Lakehouse Diagramdata engineeringBronze, silver and gold layers from ingestion to serving, plus a quarantine quality gate.

Data engineers draw pictures of data rather than of objects: where it comes from, how it moves, in what order jobs run, and how quality improves layer by layer. This section collects six sample diagrams for a typical data platform, each drawn in full with a walkthrough, a notation legend, a variant, pros and cons, and the diagram as code.

Most of these diagrams are not UML. UML covers software structure and behaviour; data flow diagrams, DAGs, lineage graphs and architecture views are the other standard notations used alongside it.

1. Why These Diagrams

A data platform has three kinds of question. Logical: which processes and stores exist and what data passes between them (data flow diagrams). Operational: what runs when, in parallel or in sequence, and how real-time paths are partitioned (pipeline DAGs and streaming topologies). Architectural and governance: the overall shape of the platform and the provenance of every number (Lambda/Kappa, medallion and lineage diagrams).

No single picture answers all of them. A DAG says nothing about columns; lineage says nothing about schedules; a medallion view hides the stream partitions. Teams that draw the right view for the question avoid most "the diagram is wrong" arguments.

2. How They Fit Together

Which diagram describes which part of the platformLambda / Kappa: the overall shape of batch and streaming pathsStreaming topologyMedallion layersSourcesOLTP, eventsIngestCDC, KafkaLakehousebronze, silver, goldMartsfacts, dimsConsumersBI, ML, appsPipeline DAG: order of the batch jobsData lineage: where every table and column came fromData flow diagram: business processes, stores and external parties (logical)

The top brackets describe the platform's shape; the bottom brackets cut across it. Lineage and the DFD span the whole width because they follow data, not infrastructure.

3. Which Diagram When

QuestionDiagramSample page
What data moves between processes, stores and outside parties?Data flow diagram (Gane-Sarson)Data Flow Diagram: Order System
In what order do batch tasks run, and what is blocked by a failure?Pipeline DAGPipeline DAG Diagram
Where did this number come from; what breaks if I change it?Lineage graphData Lineage Graph
Do we need separate batch and streaming paths?Lambda vs Kappa architectureLambda vs Kappa Diagram
How many partitions and parallel operators; where are shuffles?Streaming topologyStreaming Topology Diagram
Which layer holds raw, clean and business-ready data?Medallion layer diagramMedallion Lakehouse Diagram

4. Further Reading

DiagrammingSectionall notations on this site Structured AnalysisSectionDFDs, context diagrams and data dictionaries Diagrams as CodediagramsMermaid, PlantUML, Graphviz and D2 Data ArchitectureSectionthe concepts behind these diagrams Data PipelinesSectionETL, orchestration and ingestion