💻Inference Engine
Also known as LLM Runtime, Model Runtime, Inference Server
The tool that runs a model and gets the answer out of it.
Think of it like
Think of it like a record player. The record holds the song, and the player makes it play.
Example
Jack's company tests two engines with the same open model. One is easy to set up on laptops, and the other serves five times more users on their GPU server.
Why it matters
The engine sets how fast a model runs. It also sets how many people one box can serve.
Definition
Common mistake
Key takeaways
- 1Loads and runs. Weights in, tokens out.
- 2Fits the hardware. CPU, GPU or phone.
- 3Laptop or server. Different jobs.