🗓️Sensor Task
Also known as Airflow Sensor, Dagster Sensor
A task that waits for a file or a table. Then the rest go on.
Think of it like
It is like a dog by the door. It sits and waits until the mail drops in.
Example
A partner drops a file on a server some time between 1 and 4 am. A sensor waits for it, and only then does the load start.
Why it matters
Other teams rarely deliver on a fixed clock. A sensor starts the work when the data is really there, not when you guessed it would be.
Definition
Example code
python
wait = S3KeySensor(
task_id="wait_for_file",
bucket_name="partner-drop",
bucket_key="orders/{{ ds }}.csv",
poke_interval=300,
timeout=6 * 60 * 60,
mode="reschedule",
)Common mistake
Key takeaways
- 1Wait for a sign. A file, a row or a finished job.
- 2Check on a gap. It looks again every few minutes.
- 3Time out. It fails if the sign never comes.