Quantization vs Model Pruning
People often think quantization and pruning are the same way to shrink a model. Quantization keeps every weight but saves each one with fewer bits, like rounding. Pruning takes out the weights or parts that matter least, so there are fewer of them. The two work in their own ways, so they are often used together.
Quantization (AI)
A way to shrink a model. Each weight is kept with fewer bits.
Quantization makes a model smaller by storing each weight with fewer bits. Weights are usually trained as 16-bit numbers, and quantization rounds them to 8 bits, 4 bits or even less. An 8 billion weight model drops from about 16 GB at 16 bits to about 5 GB at 4 bits. The model also runs faster, because there is less to read from memory for each token. At 8 or 4 bits the loss in quality is usually small, but it grows quickly below that.
Model Pruning
A way to shrink a model. The parts it needs least are cut out.
Model pruning makes a model smaller by taking out the parts that matter least. It can take out single weights that are close to zero. Or it can take out whole parts, such as neurons, attention heads or layers. After pruning, the model is often trained a bit more, so it can make up for what was lost. Taking out whole parts is what makes it run faster on normal hardware. Pruning is often used with quantization and distillation.
In one line
- Quantization (AI)
- A way to shrink a model. Each weight is kept with fewer bits.
- Model Pruning
- A way to shrink a model. The parts it needs least are cut out.