Skip to content
BENIBIX Learn

Databricks Compute

Data🧰 PlatformsData Platforms

Also known as Databricks Cluster, All-Purpose Compute, Job Compute, Databricks Serverless Compute

The machines that run your code in Databricks. You pay while they are on.

Think of it like

It is like renting a car. You pay for each hour you have it, so you give it back when you are done.

Example

A team sets its all-purpose cluster to shut down after 30 idle minutes. It no longer runs all night with nobody using it.

Why it matters

Compute is most of what Databricks costs. The kind and size you choose decide the bill.

Definition

Databricks compute is the set of machines that runs your code in Databricks. Classic compute is a cluster of cloud machines that you pick and size yourself. All-purpose compute is for notebooks and can be shared by a team. Job compute starts for one job and shuts down when the job ends. Serverless compute is run by Databricks, so there are no machines for you to pick. SQL warehouses are the kind used for SQL and BI.
Two kinds of compute sit at the bottom. One stays on for notebooks, one lives only as long as the job.

Common mistake

Running nightly jobs on the team's shared cluster. Give each job its own job compute, which stops as soon as the job is done.

Key takeaways

  1. 1
    Classic or serverless. You size it, or Databricks does.
  2. 2
    All-purpose or job. Shared for notebooks, or one job.
  3. 3
    Auto stop. Idle clusters shut down after a set time.