# Workflow Orchestration

> Pipeline steps run in the right order and on time. Failed steps get retried.

**Field:** Data Engineering  
**Category:** Orchestration  
**Also known as:** Data Orchestration, Pipeline Orchestration, Data Orchestrator

## Think of it like

It is like a coach with a whistle. Each player waits for the whistle, then runs their part.

## Definition

Workflow orchestration runs the steps of a data pipeline in the right order. Each step is a task, and the tasks form a map with arrows between them. The orchestrator starts each task when the tasks before it are done. It runs the whole map on a schedule. When a task fails, it retries it or raises an alert. It also keeps a record of every run.

## Example

Each night at 2 am, an orchestrator loads sales, then builds the reports, then emails the team. If the load fails, the reports wait.

## Why it matters

Real pipelines have dozens of steps that depend on each other. Without a tool to run them, someone runs them by hand and skips one.

## Common mistake

Doing the heavy data work inside the orchestrator. It should start jobs in a warehouse or Spark, not crunch the data itself.

## Three principles

- **Order.** A task starts when the ones before it finish.
- **Time.** Runs start on a schedule or an event.
- **Recovery.** Failed tasks retry, and people get told.

## Related terms

- [Data Pipeline](https://learn.benibix.com/term/data/data-pipeline)
- [Task Dependency](https://learn.benibix.com/term/data/task-dependency)
- [Apache Airflow](https://learn.benibix.com/term/data/apache-airflow)
- [Dagster](https://learn.benibix.com/term/data/dagster)
- [Prefect](https://learn.benibix.com/term/data/prefect)
- [Workflow as Code](https://learn.benibix.com/term/data/workflow-as-code)
- [Lakeflow Jobs](https://learn.benibix.com/term/data/lakeflow-jobs)

![Five tasks joined by arrows: extract orders and extract payments both feed a join task, which feeds build report and then email team. The join task is highlighted as failed, with a loop arrow labelled retry 2 of 3.](https://learn.benibix.com/diagrams/orchestration-dag.svg)

---

Source: https://learn.benibix.com/term/data/workflow-orchestration
Author: Behnia Zarei (a developer in Vienna, Austria, who builds and teaches data, AI and business intelligence at BENIBIX)
Publisher: BENIBIX Learn (https://learn.benibix.com/)
Last updated: 2026-11-17
Licence: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Quote freely with attribution.
