💻vLLM
Also known as vLLM Server, vllm
A free tool that serves one AI model to lots of people at once.
Think of it like
Think of it like a busy bus. It picks up many people on one trip, so no seat is wasted.
Example
Lucy's company hosts an open model for 2,000 staff. On the same GPU server, vLLM serves many more requests a second than a laptop tool could.
Why it matters
Serving many users is where the cost adds up. vLLM gets far more answers out of each graphics card.
Definition
Example code
bash
vllm serve <model-name> --port 8000Common mistake
Key takeaways
- 1Many users at once. From one server.
- 2Paged memory. Little space wasted.
- 3A familiar API. Easy to switch.