Skip to content
BENIBIX Learn

llama.cpp

AI💻 Running LocallyLocal Runtimes

Also known as llama-cpp, llama cpp

A free engine that runs AI models on most PCs and laptops.

Think of it like

Think of it like a small, tough motor that fits in just about any car. It runs on most things.

Example

Lucy builds llama.cpp on an old office PC with no graphics card. A small 4-bit model writes about 8 tokens per second, which is enough for her notes tool.

Why it matters

It made local AI work on normal machines. Much of the local AI world is built on it.

Definition

llama.cpp is a free, open-source inference engine that runs language models on everyday machines. It first came out in March 2023. It is written in C and C++ and needs very few other parts. It runs on CPUs, on many kinds of graphics cards and on Apple chips. It reads models in the GGUF format, which it made. Many local AI apps use it inside, often without showing its name.

Example code

bash
llama-cli -m model-Q4_K_M.gguf -p "Write a haiku about rain."
Run a GGUF model with one prompt from the command line.
llama.cpp reads one GGUF file and runs it on a CPU, a graphics card or an Apple chip.

Common mistake

Using a very old build. It changes fast, and new models often need a recent version to load at all.

Key takeaways

  1. 1
    Runs almost anywhere. CPU, GPU, Apple.
  2. 2
    Made GGUF. Its own format.
  3. 3
    Inside many apps. Often unnamed.

Compare