Skip to content
BENIBIX Learn

GGUF

AI💻 Running LocallyModel Compression

Also known as GGUF File, GGUF Format, .gguf

One file that holds a whole model. It is ready to run on your own PC.

Think of it like

Think of it like a lunch box with a label. Everything you need is packed inside one box.

Example

Emma finds a model in several GGUF files on a model hub. She picks the Q4_K_M one at about 5 GB, because it fits her laptop and keeps good quality.

Why it matters

One file with all you need in it is easy to share, download and run.

Definition

GGUF is a file format that keeps a whole model in one file, ready to run on your own machine. It came out in 2023. It took the place of an older format called GGML. A GGUF file holds the weights, often shrunk, plus all that is needed to run them: the model's settings, its tokenizer and its prompt format. Names like Q4_K_M or Q8_0 in the file name tell you how it was shrunk. Many local tools read GGUF files.

Example code

text
model-8B-Instruct-Q4_K_M.gguf
A common file name. It gives the model, its size, and Q4_K_M, which means 4 bits per weight.
One GGUF file: a header with the model's details, then the weights.

Common mistake

Picking a GGUF file by size alone. The Q label tells you the quality trade, so read it before you download.

Key takeaways

  1. 1
    One file. Everything inside.
  2. 2
    Weights and settings. Tokenizer too.
  3. 3
    Q label. Tells the quantization.

Compare