Dagster and Prefect

Two modern Python orchestrators that rethink Airflow in different directions: Prefect makes workflows feel like plain Python functions, while Dagster organises pipelines around the data assets they produce. This page explains both models and how to choose.

1. What They Are

Dagster and Prefect are open-source Python workflow orchestrators. Like Apache Airflow, they schedule data pipelines, run steps in dependency order, retry failures, and show you what happened. They were created later than Airflow (both emerged around 2018 to 2019) and each rethinks a different part of it. Prefect keeps the task-graph idea but makes it feel like ordinary Python: decorate functions, call them, and the orchestrator tracks the run. Dagster changes the unit of thinking from tasks to data assets: tables, files and models that should exist, each defined by the code that produces it.

The distinction matters. A task graph answers "what should run, and in what order?" An asset graph answers "what data do we have, how was it made, and is it up to date?" Both can schedule the same work, but they surface different information when something breaks.

2. Why It Matters

Orchestration is the backbone of a data platform: if the orchestrator is hard to test, every pipeline is hard to test. Airflow's early design (DAG files parsed by a scheduler, tasks communicating through XCom, no native notion of the datasets they produce) led to common pain points: slow local development, difficulty passing data between tasks, and weak lineage. Airflow has since added the TaskFlow API and data-aware scheduling with Datasets (renamed Assets in Airflow 3), narrowing the gap. Knowing the three tools' philosophies helps you choose and also helps you use any of them well.

3. How Each Works

Prefect

In Prefect 2 and later, a flow is a Python function decorated with @flow and a task is a function decorated with @task. Calling a flow runs it locally with full tracking; there is no DAG file to parse ahead of time, so branching and dynamic fan-out (task.map(items)) are decided at run time. Tasks get retries, timeouts, caching by input hash, and concurrency limits. To schedule runs you create a deployment, and work pools and workers pick up scheduled runs on your infrastructure (processes, Docker, Kubernetes, serverless). State and logs go to a Prefect server you host or to Prefect Cloud.

Dagster

In Dagster you declare software-defined assets with @asset; a function's parameter names declare its upstream assets, so lineage comes from the code itself. Dagster knows when each asset was last materialised, can mark assets stale when upstream code or data changes, and supports partitions (for example one per day) so you can backfill exactly the missing ones. Resources inject connections (warehouse, S3 client) so the same asset can run against test and production. IO managers handle storing and loading outputs. Asset checks attach data-quality tests to assets. Schedules and sensors trigger runs; declarative automation can materialise assets when their dependencies update. Tasks still exist (ops and jobs) for work that does not produce a data asset.

4. Worked Example: One Pipeline, Two Tools

The same tiny pipeline (load orders, compute revenue per customer) in each tool. Install with pip install prefect or pip install dagster dagster-webserver.

Prefect

from prefect import flow, task
import pandas as pd

@task(retries=3, retry_delay_seconds=10)
def extract() -> pd.DataFrame:
    return pd.DataFrame({"customer": ["a", "b", "a"], "amount": [10, 25, 5]})

@task
def transform(df: pd.DataFrame) -> pd.DataFrame:
    return df.groupby("customer", as_index=False)["amount"].sum()

@task
def load(df: pd.DataFrame) -> None:
    df.to_csv("revenue.csv", index=False)

@flow(log_prints=True)
def revenue_etl():
    load(transform(extract()))
    print("done")

if __name__ == "__main__":
    revenue_etl()            # runs and is tracked; use .serve(cron="0 2 * * *") to schedule

Dagster

import pandas as pd
from dagster import asset, Definitions, ScheduleDefinition, define_asset_job, materialize

@asset
def orders() -> pd.DataFrame:
    return pd.DataFrame({"customer": ["a", "b", "a"], "amount": [10, 25, 5]})

@asset
def revenue_by_customer(orders: pd.DataFrame) -> pd.DataFrame:   # depends on `orders`
    return orders.groupby("customer", as_index=False)["amount"].sum()

daily = ScheduleDefinition(job=define_asset_job("all_assets"), cron_schedule="0 2 * * *")
defs = Definitions(assets=[orders, revenue_by_customer], schedules=[daily])

if __name__ == "__main__":
    result = materialize([orders, revenue_by_customer])            # quick local run
    print(result.output_for_node("revenue_by_customer"))

Notice the difference. In Prefect, revenue_etl describes steps and the CSV is a side effect. In Dagster, revenue_by_customer is the dataset; the UI (dagster dev) shows its lineage, last materialisation and any checks, and you can rematerialise just that asset.

5. Comparison With Airflow

6. Choosing in Practice

Three questions settle most decisions.

What do people ask when something breaks? If the question is usually "which job failed and why?", a task-centred tool (Airflow or Prefect) maps directly onto it. If the question is usually "is the revenue table correct and current, and what feeds it?", Dagster's asset view answers that without extra tooling, because lineage and freshness are first-class.

How dynamic are the workflows? Pipelines whose shape depends on run-time data (one branch per file that arrived, a variable number of model-training jobs, event-driven triggers from webhooks) fit Prefect's "just Python" execution well. Mostly fixed nightly graphs fit any of the three.

What already exists? A large Airflow estate with many provider operators and a managed service (Amazon MWAA, Google Cloud Composer, Astronomer) is expensive to migrate; modern Airflow with TaskFlow and Assets may be good enough. Teams already centred on dbt often find Dagster's dbt integration, which turns each dbt model into an asset in the same graph as Python assets, a strong reason to choose it.

Whatever you choose, the same engineering habits apply: keep steps idempotent, parameterise by date or partition, keep secrets in a secret store rather than in code, write unit tests for the transformation functions themselves, and alert on missed schedules as well as on failures.

7. Pitfalls

• Doing heavy compute inside the orchestrator. Orchestrators coordinate; push big transforms to Spark, a warehouse or dbt.
• Non-idempotent steps. Retries and backfills rerun work; each step should be safe to repeat (overwrite partitions, upsert by key).
• Passing large data in memory between tasks. Use storage (object store, warehouse) and pass references, or configure IO managers.
• Version churn. Prefect 1 to 2 and Airflow 2 to 3 had breaking changes; pin versions and read migration guides.
• Modelling everything as an asset. Notifications and API calls that produce no data fit Dagster ops or sensors better.

8. Related Topics

• Apache Airflow — the incumbent task-based orchestrator
• Pipeline Orchestration — section overview
• ETL Pipeline — what is being orchestrated
• Data Pipelines — parent section
• Batch vs Streaming — when scheduled batches are the wrong tool

9. Further Reading

• Dagster documentation, "Software-defined assets" and "Partitions" concepts (docs.dagster.io).
• Prefect documentation, "Flows", "Tasks" and "Deployments" (docs.prefect.io).
• Apache Airflow documentation, "Data-aware scheduling".
• Joe Reis and Matt Housley, Fundamentals of Data Engineering (O'Reilly, 2022), chapter on orchestration as an undercurrent.