Skip to content
BENIBIX Learn

Airflow XCom

Data🗓️ OrchestrationApache Airflow

Also known as XCom, Cross-Communication

A small note one Airflow task leaves for another. It might be a file name or a count.

Think of it like

It is like a sticky note left on the fridge for the next person. It holds a few words, not a whole book.

Example

An extract task writes a file to the cloud and pushes its path. The load task pulls the path and reads the file from there.

Why it matters

Tasks may run on different machines, so they cannot share memory. XCom is the safe way to hand over a pointer to the data.

Definition

XCom is how one Airflow task passes a small value to another. XCom is short for cross-communication. A task pushes a value, such as a file name or a row count. A later task pulls it by the first task's ID. When a task function returns a value, Airflow pushes it for you. XComs are kept in Airflow's database by default, so they are meant to be small.

Example code

python
@task
def count_rows():
    return 1250          # pushed to XCom

@task
def report(n):
    print(f"{n} rows")   # pulled from XCom

report(count_rows())
The return value of count_rows travels to report through XCom. No file is written.
XCom is the small note passed between task instances in the same run, shown at the bottom.

Common mistake

Passing a whole table through XCom. It bloats Airflow's database. Pass the file path instead.

Key takeaways

  1. 1
    Small values. IDs, paths and counts, not data.
  2. 2
    Push and pull. One task writes, a later one reads.
  3. 3
    Kept in the database. Stored with the run by default.