Skip to content
BENIBIX Learn

Batch Inference

AI⚙️ MLOps & DeploymentModel Deployment

Also known as Offline Inference, Batch Prediction, Batch Scoring

A model runs on lots of data at set times. The results are saved.

Think of it like

Think of it like marking a whole stack of tests at night. The scores are ready in the morning.

Example

Each night Lucy's job scores all 2 million customers for how likely they are to leave. In the morning the sales team sees a list of the top 500 to call.

Why it matters

Many uses don't need an answer this second. Batch work is far cheaper and easier to run than a live service.

Definition

Batch inference runs a model on a big set of data all at once, at set times, and saves the results for later. A job reads many rows, such as every customer. It runs the model on each one and writes the scores to a table or file. Apps then read the saved scores and do not call the model. It is cheap and simple, since the work runs when machines are free and no one is waiting. The cost is that the results are only as fresh as the last run.
Each night a job scores every customer and saves the results for the next day.

Common mistake

Using batch scores for something that changes by the minute. A score from last night can't see this morning's fraud.

Key takeaways

  1. 1
    All at once. Many rows.
  2. 2
    On a schedule. Nightly or weekly.
  3. 3
    Saved for later. Only as fresh as the run.

Compare