AWS Glue Workflow

AWS Glue Workflow is the native orchestration layer inside AWS Glue. It lets you compose multiple Glue crawlers, ETL jobs, and triggers into a single, dependency-aware DAG with shared run state, run-level monitoring, and conditional branching. It is purpose-built for Glue-only pipelines; for cross-service orchestration, Step Functions or MWAA are better fits.

Key Features:

Orchestration of ETL Jobs: Sequence Glue jobs and crawlers with explicit dependencies; downstream nodes only fire when upstream nodes succeed (or match defined conditions).
Conditional Triggers: Branch on job success, failure, or stopped state; mix AND/OR logic across multiple predecessors.
Visual Workflow Editor: Drag-and-drop console for designing the DAG; shows live status of each node during execution.
Glue Data Catalog Integration: Crawlers and jobs read and write the catalog as a shared source of truth for schema.
Multiple Trigger Types: On-demand, scheduled (cron), and event-based (EventBridge) starts.
Run Properties: Pass typed parameters between nodes within a single workflow run for templated pipelines.
Monitoring & Logging: Per-run history, CloudWatch Logs/Metrics, and structured run state for alerting.
Blueprints: Reusable workflow templates (e.g., partitioned data lake ingest) that materialize into customized workflows.

Common Use Cases:

Complex ETL Pipelines: Coordinate raw → bronze → silver → gold layer jobs with shared partition state.
Data Lake Management: Crawl new S3 prefixes, then transform and recatalog the curated output.
Data Warehousing: Sequence extract jobs, transform jobs, and load jobs into Redshift.
Event-Driven Processing: Trigger a workflow when an EventBridge rule fires (e.g., S3 object arrival, scheduled time).
Catalog Maintenance: Periodic crawler sweeps followed by partition cleanup or compaction jobs.

Service Limits & Quotas:

Workflows per account/region: default soft limit 250.
Triggers per workflow: default soft limit 100.
Jobs/crawlers per workflow: default soft limit 100 each.
Concurrent workflow runs: bounded by underlying job and crawler concurrency limits.
Run history retention: 90 days of run records by default.
Step Functions comparison: if you need branching across non-Glue services, choose Step Functions; Glue Workflow nodes are limited to Glue jobs and crawlers.

Pricing Model:

Workflow itself: no charge for orchestration — you only pay for the underlying jobs and crawlers it runs.
Glue ETL jobs: billed per DPU-second (1 DPU = 4 vCPU + 16 GB RAM) with a 1-minute minimum.
Glue crawlers: billed per DPU-second with a 10-minute minimum per run.
Common cost surprises: over-frequent triggers re-running expensive crawlers; jobs sized at default 10 DPU when 2 DPU would suffice; long retries after transient failures.

Code Example:


import boto3

glue = boto3.client("glue", region_name="us-west-2")

# Create the workflow shell
glue.create_workflow(
    Name="daily-events-pipeline",
    Description="Crawl raw S3, transform, then crawl curated.",
    DefaultRunProperties={"target_date": ""},
)

# Schedule trigger fires the crawler at 02:00 UTC every day
glue.create_trigger(
    Name="start-raw-crawl",
    Type="SCHEDULED",
    Schedule="cron(0 2 * * ? *)",
    WorkflowName="daily-events-pipeline",
    Actions=[{"CrawlerName": "raw-events-crawler"}],
    StartOnCreation=True,
)

# Conditional trigger fires the ETL job only when the crawler succeeds
glue.create_trigger(
    Name="run-transform-on-crawl-success",
    Type="CONDITIONAL",
    WorkflowName="daily-events-pipeline",
    Predicate={
        "Logical": "AND",
        "Conditions": [{
            "LogicalOperator": "EQUALS",
            "CrawlerName": "raw-events-crawler",
            "CrawlState": "SUCCEEDED",
        }],
    },
    Actions=[{"JobName": "events-transform"}],
    StartOnCreation=True,
)

Common Interview Questions:

When should you use Glue Workflow versus Step Functions or MWAA?

Glue Workflow is the right choice when every node is a Glue job or crawler — it shares run state and integrates with the catalog without extra plumbing. Step Functions is better when you need to mix Lambda, ECS, EMR, SageMaker, or external services. MWAA (managed Airflow) wins when you need a rich operator ecosystem, complex dependency logic, or shared scheduling across many teams.

How do you pass values between nodes in a workflow?

Use workflow run properties (typed key-value pairs) seeded at start time and updated by jobs via get_workflow_run_properties / put_workflow_run_properties. Each node sees the latest values.

What types of triggers can start a workflow?

On-demand (manual or API), scheduled (cron expression), and event-based via EventBridge — useful for S3 object-created events or upstream pipeline completions.

How does conditional branching work?

A conditional trigger evaluates a predicate (AND/OR over job or crawler states like SUCCEEDED, FAILED, STOPPED). If the predicate matches, its actions fire — letting you route success and failure paths to different downstream jobs.

How do you monitor and alert on workflow failures?

CloudWatch metrics expose per-job success/failure counts and durations. EventBridge emits state-change events that can fan out to SNS, Lambda, or PagerDuty. The Glue console shows run-level status and per-node history.

What is a Glue Blueprint?

A reusable workflow template parameterized for common patterns (e.g., partitioned ingest from S3). Operators instantiate a blueprint with specific parameters and Glue generates the underlying jobs, crawlers, and triggers.

AWS Glue Workflow is the simplest way to orchestrate Glue-native pipelines end to end. For Glue-only workloads it removes the need for an external orchestrator; for hybrid pipelines, reach for Step Functions or MWAA instead.