AWS Step Functions

AWS Step Functions is a serverless orchestration service that coordinates AWS services into durable workflows. You define a state machine in Amazon States Language (ASL) — JSON describing tasks, choices, parallel branches, retries, and waits — and Step Functions executes it with built-in error handling, observability, and exactly-once state transitions. Standard workflows support runs up to one year; Express workflows handle high-throughput, short-lived runs at lower cost. Distributed Map adds high-fan-out parallel processing across thousands of items.

Key Features:

Two Workflow Types: Standard (up to 1 year, exactly-once, durable) and Express (high-throughput, at-most-once or at-least-once, ≤5 min).
Visual Workflow Studio: Drag-and-drop state machine designer with side-by-side ASL JSON.
Direct AWS SDK Integration: Optimized integrations for Lambda, ECS/Fargate, EMR, SageMaker, Glue, Bedrock, DynamoDB, SNS, SQS, EventBridge, and 200+ AWS services via the SDK integration pattern.
Distributed Map: Parallel processing of up to 10,000 concurrent child executions across millions of items in S3 — the modern replacement for ad-hoc fan-out.
Built-in Error Handling: Retry with exponential backoff, Catch handlers, custom error names — declarative, no glue code.
Wait, Choice, Parallel, Map States: First-class control flow constructs in JSON.
Callback Patterns: Pause workflows for human approval or async callbacks via task tokens (.waitForTaskToken).
Observability: CloudWatch Logs, X-Ray traces, and per-execution event history for replay and debugging.
HTTPS Tasks: Call any HTTPS endpoint (with optional Connection-managed auth) without writing a Lambda.

Common Use Cases:

ETL Pipelines: Chain Glue, Lambda, EMR, and Athena steps with retries and conditional branches.
Microservices Orchestration: Saga-style coordination across multiple service boundaries with compensation steps.
Batch Processing: Distributed Map fans out S3 object processing across thousands of parallel workers.
Business Process Automation: Long-running approval flows with human-in-the-loop callbacks.
ML Pipelines: SageMaker training, evaluation, and conditional deployment.
GenAI Agents: Bedrock + tool calls coordinated via Step Functions for traceable, retryable LLM workflows.

Service Limits & Quotas:

Standard execution duration: up to 1 year.
Express execution duration: up to 5 minutes.
Standard state transitions: default soft limit 4,000/s per account.
Distributed Map child concurrency: up to 10,000 in parallel.
State machine size: 1 MB definition.
Input/output size: 256 KB per state transition.
Execution history events: 25,000 per Standard execution.
StartExecution rate: default soft limit varies by region (e.g., 1,300/s in major regions).

Pricing Model:

Standard workflows: billed per state transition (free tier of 4,000/month).
Express workflows: billed per request and per GB-second of execution duration — drastically cheaper for high-volume short tasks.
Underlying services: Lambda, Glue, etc. billed independently.
Distributed Map: child executions billed at the same rate as their workflow type.
Common cost surprises: Standard workflows used for high-volume short tasks (Express is 90%+ cheaper); chatty Choice/Pass loops; large state inputs that get logged repeatedly.

Code Example:

State machine in Amazon States Language with retries, a parallel branch, and a wait-for-callback step:


{
  "Comment": "Order processing pipeline",
  "StartAt": "ValidateOrder",
  "States": {
    "ValidateOrder": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": {
        "FunctionName": "validate-order",
        "Payload.$": "$"
      },
      "Retry": [{
        "ErrorEquals": ["States.TaskFailed"],
        "IntervalSeconds": 2,
        "MaxAttempts": 3,
        "BackoffRate": 2.0
      }],
      "Next": "ChargeAndShip"
    },
    "ChargeAndShip": {
      "Type": "Parallel",
      "Branches": [
        {
          "StartAt": "Charge",
          "States": {
            "Charge": {
              "Type": "Task",
              "Resource": "arn:aws:states:::lambda:invoke.waitForTaskToken",
              "Parameters": {
                "FunctionName": "charge-card",
                "Payload": {"taskToken.$": "$$.Task.Token", "input.$": "$"}
              },
              "End": true
            }
          }
        },
        {
          "StartAt": "Ship",
          "States": {
            "Ship": {
              "Type": "Task",
              "Resource": "arn:aws:states:::lambda:invoke",
              "Parameters": {"FunctionName": "ship-package", "Payload.$": "$"},
              "End": true
            }
          }
        }
      ],
      "End": true
    }
  }
}

Start an execution from boto3:


import boto3, json

sfn = boto3.client("stepfunctions", region_name="us-west-2")

resp = sfn.start_execution(
    stateMachineArn="arn:aws:states:us-west-2:123456789012:stateMachine:OrderPipeline",
    input=json.dumps({"order_id": "A-9001", "user_id": 42}),
)
print(resp["executionArn"])

Common Interview Questions:

When should you use Standard vs Express workflows?

Standard for long-running, durable, exactly-once workflows (orchestration, approvals, ETL). Express for high-volume, short-lived event processing (per-request orchestration in APIs, IoT events). Express is dramatically cheaper at scale and supports both at-most-once (sync) and at-least-once (async) modes.

What is Distributed Map and what problem does it solve?

Distributed Map fans out parallel child executions over a dataset (S3 inventory, JSON array, CSV) with up to 10,000 concurrent children and millions of items. It replaces ad-hoc Lambda fan-out patterns and removes the 25,000-event-history limit by isolating each item in its own execution.

How do you implement human approval steps?

Use the .waitForTaskToken integration pattern. The task hands a token to a Lambda or notification step; the workflow pauses until SendTaskSuccess or SendTaskFailure is called with that token (e.g., from an approval link in email).

How do you handle errors and retries declaratively?

Each Task state can declare a Retry array (with backoff, max attempts, error filters) and a Catch array (error filters and a Next state). This eliminates Lambda-level retry plumbing and centralizes error policy in the state machine.

What's the difference between optimized integrations and SDK integrations?

Optimized integrations (e.g., arn:aws:states:::lambda:invoke) provide special behavior like .sync (wait for completion) and .waitForTaskToken. SDK integrations (e.g., arn:aws:states:::aws-sdk:dynamodb:getItem) call any AWS API directly — fewer features, broader coverage.

What are the input/output size limits and how do you work around them?

State transitions cap input/output at 256 KB. For larger payloads, store data in S3 and pass references (the "claim check" pattern) between states.

Step Functions is the go-to AWS orchestrator for any workflow that needs more structure than chained Lambdas. With Distributed Map, Express workflows, and SDK integrations, it covers everything from per-request choreography to long-running, fan-out batch processing.