# Quantization (AI)

> A way to shrink a model. Each weight is kept with fewer bits.

**Field:** Artificial Intelligence  
**Category:** Running Locally  
**Also known as:** Model Quantization, LLM Quantization, 4-bit Quantization

## Think of it like

Think of it like rounding prices to the nearest dollar. The list is shorter, and you lose just a bit.

## Definition

Quantization makes a model smaller by storing each weight with fewer bits. Weights are usually trained as 16-bit numbers, and quantization rounds them to 8 bits, 4 bits or even less. An 8 billion weight model drops from about 16 GB at 16 bits to about 5 GB at 4 bits. The model also runs faster, because there is less to read from memory for each token. At 8 or 4 bits the loss in quality is usually small, but it grows quickly below that.

## Example

Lucy's laptop has 16 GB of memory. The model she wants is 16 GB at full size, so she downloads a 4-bit version of about 5 GB, and it runs well.

## Why it matters

It is the main reason good models fit on normal laptops and phones at all.

## Common mistake

Going as low as possible to save space. Below about 4 bits, answers often get noticeably worse, so test before you settle.

## Three principles

- **Fewer bits per weight.** 16 down to 4.
- **Smaller and faster.** Less to read.
- **Small loss at first.** Big loss if pushed.

## Related terms

- [Model Weights File](https://learn.benibix.com/term/ai/model-weights)
- [GGUF](https://learn.benibix.com/term/ai/gguf)
- [Model Pruning](https://learn.benibix.com/term/ai/model-pruning)
- [Knowledge Distillation](https://learn.benibix.com/term/ai/knowledge-distillation)
- [Local AI Model](https://learn.benibix.com/term/ai/local-ai-model)
- [Quantized Low-Rank Adaptation (QLoRA)](https://learn.benibix.com/term/ai/qlora)
- [Model Precision](https://learn.benibix.com/term/ai/model-precision)

## Told apart from

- [Quantization vs Model Pruning](https://learn.benibix.com/compare/quantization-vs-model-pruning): People often think quantization and pruning are the same way to shrink a model. Quantization keeps every weight but saves each one with fewer bits, like rounding. Pruning takes out the weights or parts that matter least, so there are fewer of them. The two work in their own ways, so they are often used together.

![The same weight stored three ways: with 16 bits, with 8 bits and with 4 bits. The file gets smaller each time, and the number gets a little rougher.](https://learn.benibix.com/diagrams/lo-quantization.svg)

---

Source: https://learn.benibix.com/term/ai/quantization
Author: Behnia Zarei (a developer in Vienna, Austria, who builds and teaches data, AI and business intelligence at BENIBIX)
Publisher: BENIBIX Learn (https://learn.benibix.com/)
Last updated: 2026-11-19
Licence: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Quote freely with attribution.
