Skip to content
BENIBIX Learn

GGUF Quantization Type

AI💻 Running LocallyModel Compression

Also known as Q4_K_M, Q8_0, Q5_K_M, Q6_K, Q3_K_M, Q2_K, K-Quants, Quant Type, Quantization Level

The label on a shrunk model file, like Q4_K_M. It says how much it was shrunk.

Think of it like

It is like photo sizes on a phone. You pick small, medium or large.

Example

Lucy sees one model in many files on a model hub. She picks Q4_K_M, about 5 GB, which fits her 8 GB card. Q8_0 is about 8.5 GB and would not fit.

Why it matters

It is the one choice that decides if a model fits, how fast it runs and how good it stays.

Definition

A GGUF quantization type is a label, like Q4_K_M or Q8_0, that says how hard a model was shrunk. The number after the Q is about how many bits each weight keeps. The K marks a newer, smarter way of shrinking. The last letter, S, M or L, gives a small, medium or large file within that level. For an 8 billion weight model, Q4_K_M keeps about 4.9 bits a weight. Its file is about 5 GB, against about 16 GB at 16 bits.

Example code

bash
llama-quantize model-bf16.gguf model-Q4_K_M.gguf Q4_K_M
This shrinks a 16-bit model file to the Q4_K_M type.
Q8_0 and Q4_K_M are two of the steps. Q4_K_M is a third the size of 16 bits.

Common mistake

Picking the smallest file every time. The lowest types, like Q2_K, can make answers clearly worse, so test before you settle.

Key takeaways

  1. 1
    The number is bits. Q4 keeps about four bits a weight.
  2. 2
    K is newer. A smarter way to shrink.
  3. 3
    S, M or L. Small, medium or large file.