Skip to content
BENIBIX Learn

Fan-Out and Fan-In

Data🗓️ OrchestrationWorkflow Orchestration

Also known as Fan-Out, Fan-In, Fan-Out/Fan-In, Parallel Tasks

One task splits into many that run at once. Then one task waits for them all.

Think of it like

It is like one teacher handing out three worksheets. Three pupils work at once, then hand them all back.

Example

A pipeline loads Europe, Asia and the Americas as three tasks at once. The report task waits for all three. The run takes 20 minutes instead of an hour.

Why it matters

Running parts side by side cuts the wait. It also means one slow region does not hold up the others.

Definition

Fan-out is when one task starts many tasks that run at the same time. Fan-in is when one task waits for all of them to finish. Together they split a big job into parts and join the results. The parts must not depend on each other, or they cannot run at once. A pipeline that loads each region in its own task is a common case.
One task fans out to three loads that run at once. The report fans them back in.

Common mistake

Fanning out to hundreds of tasks at once. The database they all hit can slow down or fall over.

Key takeaways

  1. 1
    Split. One task starts many.
  2. 2
    Run at once. Parts must not need each other.
  3. 3
    Join. One task waits for all of them.