Skip to content
BENIBIX Learn

Real-Time Inference

AI⚙️ MLOps & DeploymentModel Deployment

Also known as Online Inference, Online Prediction, Live Inference

A model runs the moment a request comes in. You wait for the answer.

Think of it like

Think of it like a crossing guard. They look and decide on the spot, while you wait at the curb.

Example

When Jack pays with his card, a fraud model scores the payment in about 50 milliseconds. The bank approves it before the shop's machine has finished beeping.

Why it matters

Some choices can't wait for tonight's batch. Real-time inference lets a model act on what is happening now.

Definition

Real-time inference runs a model the moment a request comes in. It sends back the answer while the user waits. Each request is scored on its own, most often in tens or hundreds of milliseconds. You need it when the answer depends on what is going on right now, such as a card payment or a search. It costs more than batch work, since the model has to be loaded and ready at all times. Speed limits often decide which models can be used at all.
The fraud model scores a payment before it is approved. It takes less than a second.

Common mistake

Making everything real time by default. If the answer can wait, a batch job is cheaper and simpler.

Key takeaways

  1. 1
    Answer on request. One at a time.
  2. 2
    Fast. Milliseconds count.
  3. 3
    Always on. Costs more.

Compare