💻GGUF Quantization Type
Also known as Q4_K_M, Q8_0, Q5_K_M, Q6_K, Q3_K_M, Q2_K, K-Quants, Quant Type, Quantization Level
The label on a shrunk model file, like Q4_K_M. It says how much it was shrunk.
Think of it like
It is like photo sizes on a phone. You pick small, medium or large.
Example
Lucy sees one model in many files on a model hub. She picks Q4_K_M, about 5 GB, which fits her 8 GB card. Q8_0 is about 8.5 GB and would not fit.
Why it matters
It is the one choice that decides if a model fits, how fast it runs and how good it stays.
Definition
Example code
bash
llama-quantize model-bf16.gguf model-Q4_K_M.gguf Q4_K_MCommon mistake
Key takeaways
- 1The number is bits. Q4 keeps about four bits a weight.
- 2K is newer. A smarter way to shrink.
- 3S, M or L. Small, medium or large file.