Pipeline DAG Diagram

A pipeline DAG (directed acyclic graph) shows the tasks of a batch job and the order they must run in. Each box is a task, each arrow says "run after this one succeeds", and there are no cycles, so the job always finishes. This sample is an Airflow-style daily job that extracts orders, customers and payments, validates them, builds two dbt models, loads Redshift and posts to Slack.

A DAG is not a UML diagram, though it looks like a simplified activity diagram. It is the native picture of every orchestrator: Airflow, Dagster, Prefect, Argo and dbt all render one.

1. The Diagram

1. Extract (column 1). Three independent tasks copy yesterday's rows to S3. No arrows connect them, so they run in parallel.
2. Validate. validate_raw waits for all three (fan-in), then runs column checks (Airflow SQLColumnCheckOperator): no null keys, amounts non-negative, row counts within range.
3. Transform. Two dbt models. fct_orders joins to dim_customer, so it has an extra edge and runs second.
4. Load. load_redshift copies both models into the warehouse once both exist (fan-in again).
5. Notify. A Slack message confirms the run. Because it is last, a green Slack message means the whole chain succeeded.

The footer lists the DAG-level settings. They matter as much as the boxes: catchup=False stops Airflow from back-running every missed day, and max_active_runs=1 prevents two days overwriting the same tables at once.

2. How to Read It

Edges carry ordering only, not data; the data itself sits in S3 or the warehouse between tasks. Orchestrator UIs colour each task's border by its state in a given run, which turns the static graph into a run report. Arrange the graph left to right so time flows the same way readers expect.

3. Variant: A Failed Run

Here the validation found 37 orders with a null order_id and failed after its two retries. Every task downstream is marked upstream_failed and never starts, which is exactly what you want: no bad data reaches the warehouse. A separate alert_on_call task uses trigger_rule=one_failed, so it runs only when its parent fails. Drawing the failure path explicitly (dashed red) makes reviewers ask the right question: who gets paged, and how do we re-run safely?

4. Pros, Cons, and When to Use

Pros

• Matches what the orchestrator executes, often generated straight from the code.
• Parallelism and bottlenecks are visible at a glance (wide columns vs long chains).
• Run-state colouring makes it a live debugging tool.
• Fan-in and fan-out points show where a single failure blocks everything.
• Easy for analysts and engineers to share.

Cons

• Shows ordering, not what data moves or how it changes; pair it with lineage.
• Large DAGs (hundreds of tasks) become unreadable; group with task groups.
• Cross-DAG dependencies (sensors, datasets) are often hidden.
• No notion of time windows or streaming; it describes discrete runs.
• Retries, SLAs and trigger rules live in text, not in the shapes.

When to use

• Designing or reviewing a scheduled batch job.
• Explaining why a run is late or failed.
• Planning backfills and re-runs.

When not to use

• Continuous streaming jobs (use a streaming topology).
• Showing column-level provenance (use a lineage graph).

5. As Diagrams-as-Code

Mermaid flowchart LR draws the main graph:

flowchart LR
    extract_orders --> validate_raw
    extract_customers --> validate_raw
    extract_payments --> validate_raw
    validate_raw --> dim_customer
    validate_raw --> fct_orders
    dim_customer --> fct_orders
    dim_customer --> load_redshift
    fct_orders --> load_redshift
    load_redshift --> notify_success

In Airflow the Python dependency operators are the source of truth, and the UI's Graph view renders them:

# Airflow 2.x: the same graph in Python (the DAG file IS the diagram source)
[extract_orders, extract_customers, extract_payments] >> validate_raw
validate_raw >> [dim_customer, fct_orders]
dim_customer >> fct_orders
[dim_customer, fct_orders] >> load_redshift >> notify_success
validate_raw >> alert_on_call          # alert_on_call: trigger_rule="one_failed"

6. Related

• Pipeline Diagram — notation reference for pipeline drawings
• UML Activity Diagrams — the UML cousin with forks and joins
• Apache Airflow — the orchestrator behind this sample
• Idempotency and Backfills — re-running failed DAGs safely
• Data Flow Diagram: Order System — sibling sample
• Data Lineage Graph — sibling sample
• Lambda vs Kappa Diagram — sibling sample
• Streaming Topology Diagram — sibling sample
• Medallion Lakehouse Diagram — sibling sample
• Data Engineering diagrams — section overview