# GGUF Quantization Type

> The label on a shrunk model file, like Q4_K_M. It says how much it was shrunk.

**Field:** Artificial Intelligence  
**Category:** Running Locally  
**Also known as:** Q4_K_M, Q8_0, Q5_K_M, Q6_K, Q3_K_M, Q2_K, K-Quants, Quant Type, Quantization Level

## Think of it like

It is like photo sizes on a phone. You pick small, medium or large.

## Definition

A GGUF quantization type is a label, like Q4_K_M or Q8_0, that says how hard a model was shrunk. The number after the Q is about how many bits each weight keeps. The K marks a newer, smarter way of shrinking. The last letter, S, M or L, gives a small, medium or large file within that level. For an 8 billion weight model, Q4_K_M keeps about 4.9 bits a weight. Its file is about 5 GB, against about 16 GB at 16 bits.

## Example

Lucy sees one model in many files on a model hub. She picks Q4_K_M, about 5 GB, which fits her 8 GB card. Q8_0 is about 8.5 GB and would not fit.

## Example code

```bash
llama-quantize model-bf16.gguf model-Q4_K_M.gguf Q4_K_M
```

This shrinks a 16-bit model file to the Q4_K_M type.

## Why it matters

It is the one choice that decides if a model fits, how fast it runs and how good it stays.

## Common mistake

Picking the smallest file every time. The lowest types, like Q2_K, can make answers clearly worse, so test before you settle.

## Three principles

- **The number is bits.** Q4 keeps about four bits a weight.
- **K is newer.** A smarter way to shrink.
- **S, M or L.** Small, medium or large file.

## Related terms

- [GGUF](https://learn.benibix.com/term/ai/gguf)
- [Quantization (AI)](https://learn.benibix.com/term/ai/quantization)
- [llama.cpp](https://learn.benibix.com/term/ai/llama-cpp)
- [Model Memory Requirements](https://learn.benibix.com/term/ai/model-memory-requirements)
- [Model Precision](https://learn.benibix.com/term/ai/model-precision)
- [Model Hub](https://learn.benibix.com/term/ai/model-hub)

![Bars for one 8 billion weight model saved at different precisions. At 32 bits it is about 32 GB. At 16 bits, as FP16 or BF16, about 16 GB. Q8_0 keeps about 8.5 bits a weight, about 8.5 GB. Q4_K_M keeps about 4.9 bits, about 5 GB, and is highlighted. A dashed line marks an 8 GB graphics card.](https://learn.benibix.com/diagrams/ai4-bits-per-weight.svg)

---

Source: https://learn.benibix.com/term/ai/gguf-quantization-type
Author: Behnia Zarei (a developer in Vienna, Austria, who builds and teaches data, AI and business intelligence at BENIBIX)
Publisher: BENIBIX Learn (https://learn.benibix.com/)
Last updated: 2026-11-19
Licence: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Quote freely with attribution.
