Pipeline Orchestration

Orchestration is the control plane of a data platform: it runs pipeline steps in the right order, at the right time, and recovers when they fail. This section covers the concepts and the main tools.

1. Overview

A data pipeline is rarely one job. It is a set of steps — extract, load, transform, test, publish — with dependencies between them, run on a schedule or in response to events. An orchestrator is the system that knows those dependencies, decides when each step may run, launches it on some compute, records what happened, retries failures and alerts people when something stays broken. It usually does not move data itself; it coordinates the tools that do (Spark, dbt, SQL warehouses, Python scripts, containers).

The core abstraction is the DAG, a directed acyclic graph. Nodes are tasks; edges say "this must finish before that starts". Acyclic means there is no loop, so there is always a valid order to run everything.

2. Core Concepts

• Scheduling: cron-style schedules or data-interval schedules (each run is responsible for a time slice, such as one day), plus event triggers such as a file arriving or an upstream dataset updating.
• Retries and timeouts: transient failures (a network blip, a throttled API) are retried with backoff; timeouts stop hung tasks from blocking the pipeline.
• Sensors: tasks that wait for an external condition, like a partition landing in S3. Prefer deferrable or event-driven sensors so waiting does not occupy a worker slot.
• Backfills: rerunning past intervals after a bug fix or a new pipeline; only safe when tasks are idempotent.
• Observability: run history, logs, durations, SLA alerts and lineage between datasets.

Because a retry reruns a task, every task should be idempotent: running it twice for the same interval leaves the same result (overwrite a partition, or MERGE, rather than blind append).

3. How the Pages Fit Together

Start with Apache Airflow, the most widely deployed open-source orchestrator, to learn task-centric DAGs, operators and the scheduler. Next, Dagster and Prefect show two newer designs: asset-centric pipelines and lightweight Python flows. Then read the general orchestration tools comparison, and the pipeline pages on idempotency and backfills, which explain the property that makes retries safe. Cloud-native alternatives such as AWS Step Functions and Glue Workflows suit teams that want a managed service tightly bound to one provider.

4. Choosing an Orchestrator