Skip to content
BENIBIX Learn

Airflow Provider Package

Data🗓️ OrchestrationApache Airflow

Also known as Airflow Provider, Provider Package, apache-airflow-providers

An add-on for Airflow. It brings the code to work with one outside system.

Think of it like

It is like an add-on pack for a board game. It brings new pieces for one new place.

Example

Jack's DAGs read from PostgreSQL and write to Amazon S3. He installs two providers, for postgres and for amazon, and nothing else.

Why it matters

Airflow itself stays small, and each system's code is updated on its own schedule. You get fixes for one system without moving all of Airflow.

Definition

A provider package adds support for one outside system to Airflow. It holds the hooks, the ready-made tasks and the sensors for it. Each one is installed on its own, apart from Airflow. Their names all start the same way, like apache-airflow-providers-amazon for AWS. The people who build Airflow look after more than 80 of them. You install only the ones your DAGs use.

Example code

console
$ pip install apache-airflow-providers-postgres
This adds the PostgreSQL hook and its tasks to Airflow.
The dashed box is the provider package. It ships the hook and the operator together.

Common mistake

Thinking all tasks come with Airflow. If an import fails, the package for that system is not installed yet.

Key takeaways

  1. 1
    One system each. A package per service.
  2. 2
    Hooks and operators. Plus sensors for that system.
  3. 3
    Install what you use. Nothing extra comes with it.