Skip to content
BENIBIX Learn

Workflow Scheduler

Data🗓️ OrchestrationWorkflow Orchestration

Also known as Pipeline Scheduler, Job Scheduler, Pipeline Schedule

The part that starts each pipeline at the right time. It often uses a cron rule.

Think of it like

It is like an alarm clock for jobs. When it rings, the job gets up and starts.

Example

Lucy schedules her sales pipeline for 6 am UTC. The night's orders are in by then, and the report is ready before the team logs on.

Why it matters

A run that starts too early reads data that is not there yet. One that starts too late leaves people waiting for numbers.

Definition

A workflow scheduler decides when each pipeline runs. Most schedules are written as cron rules, such as every day at 6 am. Some are presets, like hourly or daily. Each run usually covers one slice of time, such as yesterday. The scheduler starts the run, and then the tasks run in order. A pipeline with no schedule only runs when someone starts it.

Example code

python
@dag(schedule="0 6 * * *")   # 06:00 every day
@dag(schedule="@hourly")     # top of every hour
@dag(schedule=None)          # only when started by hand
Three Airflow schedules. The first is a cron rule: minute 0, hour 6, any day.
A schedule is the clock on the left. It starts the run at a set time, whether the data is ready or not.

Common mistake

Forgetting the time zone. A schedule set in UTC can run an hour off for half the year where clocks change.

Key takeaways

  1. 1
    Cron rules. Minute, hour, day, month and weekday.
  2. 2
    One slice per run. Each run covers a set stretch of time.
  3. 3
    Time zones. Know which clock the schedule follows.