Skip to content
BENIBIX Learn

vLLM

AI💻 Running LocallyLocal Runtimes

Also known as vLLM Server, vllm

A free tool that serves one AI model to lots of people at once.

Think of it like

Think of it like a busy bus. It picks up many people on one trip, so no seat is wasted.

Example

Lucy's company hosts an open model for 2,000 staff. On the same GPU server, vLLM serves many more requests a second than a laptop tool could.

Why it matters

Serving many users is where the cost adds up. vLLM gets far more answers out of each graphics card.

Definition

vLLM is a free engine that serves language models from GPU servers to many users at once. It came out of a university lab in 2023. Its key idea, called PagedAttention, keeps each request's memory in small pages. So little space is wasted. It also adds new requests to the batch as old ones finish, which keeps the card busy. Its API has the same shape as common cloud model APIs, so apps can switch to it with ease.

Example code

bash
vllm serve <model-name> --port 8000
Start a server for one model. Apps call it on port 8000.
Lots of requests are packed together on one GPU server. Lots of answers come out at once.

Common mistake

Using it for one person on a laptop. It is built for big servers and many users, and simpler tools fit a single user better.

Key takeaways

  1. 1
    Many users at once. From one server.
  2. 2
    Paged memory. Little space wasted.
  3. 3
    A familiar API. Easy to switch.

Compare