Skip to content
BENIBIX Learn

Sensor Task

Data🗓️ OrchestrationWorkflow Orchestration

Also known as Airflow Sensor, Dagster Sensor

A task that waits for a file or a table. Then the rest go on.

Think of it like

It is like a dog by the door. It sits and waits until the mail drops in.

Example

A partner drops a file on a server some time between 1 and 4 am. A sensor waits for it, and only then does the load start.

Why it matters

Other teams rarely deliver on a fixed clock. A sensor starts the work when the data is really there, not when you guessed it would be.

Definition

A sensor is a task that waits for something outside the pipeline to happen. It might wait for a file to land, a table to fill or another job to end. It checks again and again, at a set gap. When the thing happens, the sensor succeeds and the next tasks start. If it waits too long, it times out and fails.

Example code

python
wait = S3KeySensor(
    task_id="wait_for_file",
    bucket_name="partner-drop",
    bucket_key="orders/{{ ds }}.csv",
    poke_interval=300,
    timeout=6 * 60 * 60,
    mode="reschedule",
)
An Airflow sensor that checks every 5 minutes for the day's file. It gives up after 6 hours.
A sensor sits in the middle. It keeps checking until the thing it waits for shows up.

Common mistake

Leaving out the timeout. A sensor that waits forever can hold a worker slot all day.

Key takeaways

  1. 1
    Wait for a sign. A file, a row or a finished job.
  2. 2
    Check on a gap. It looks again every few minutes.
  3. 3
    Time out. It fails if the sign never comes.