Skip to content
BENIBIX Learn

Lakeflow Jobs

Data🧰 PlatformsData Platforms

Also known as Databricks Jobs, Databricks Workflows, Databricks Job

A Databricks tool that runs your steps in order, at set times.

Think of it like

It is like a to-do list that does itself. Each task starts when the one before it is done.

Example

Every night at 2 AM, a job runs three tasks. It loads new files, then a pipeline cleans them, then a notebook updates a report table.

Why it matters

Work that runs only when someone clicks run is not a pipeline yet. A job makes it run on its own, every day.

Definition

Lakeflow Jobs is the tool that runs work on its own inside Databricks. It used to be called Databricks Workflows. A job is made of tasks, such as a notebook, a pipeline or a Python script. A task can wait for others to finish, so the steps run in the right order. A job can start at a set time or when new files land in storage. One job can hold up to 1,000 tasks.
The job is the row of three tasks in the middle. A clock starts it, and each task waits for the last.

Common mistake

Putting every step in one giant notebook. Split the steps into tasks, so a failed step can be run again on its own.

Key takeaways

  1. 1
    Tasks. Notebooks, pipelines and scripts.
  2. 2
    Order. A task waits for the ones before it.
  3. 3
    Triggers. A set time, or new files landing.