Batch vs Real-Time Inference
People often think real-time inference is just the better kind and batch is the old way. They fit different needs. Batch inference scores lots of data at set times and saves it. That is cheap, but only as fresh as the last run. Real-time inference answers each request as it comes. That is fresh, but costs more to keep running.
Batch Inference
A model runs on lots of data at set times. The results are saved.
Batch inference runs a model on a big set of data all at once, at set times, and saves the results for later. A job reads many rows, such as every customer. It runs the model on each one and writes the scores to a table or file. Apps then read the saved scores and do not call the model. It is cheap and simple, since the work runs when machines are free and no one is waiting. The cost is that the results are only as fresh as the last run.
Real-Time Inference
A model runs the moment a request comes in. You wait for the answer.
Real-time inference runs a model the moment a request comes in. It sends back the answer while the user waits. Each request is scored on its own, most often in tens or hundreds of milliseconds. You need it when the answer depends on what is going on right now, such as a card payment or a search. It costs more than batch work, since the model has to be loaded and ready at all times. Speed limits often decide which models can be used at all.
In one line
- Batch Inference
- A model runs on lots of data at set times. The results are saved.
- Real-Time Inference
- A model runs the moment a request comes in. You wait for the answer.