LoRA vs QLoRA
People think QLoRA is a different way to tune a model, and it is not. Both lock the model and train two thin tables beside each big one. QLoRA also squeezes the locked model into a 4-bit form first, about a quarter of the usual size. That lets a big model be tuned on one chip, at some cost in speed.
Low-Rank Adaptation (LoRA)
A cheap way to fine-tune. The big model is locked and small parts are trained.
Low-rank adaptation is a cheap way to fine-tune a large model. The model's own weights are locked. Small extra pieces are added next to them, and only those are trained. These pieces are often less than 1 percent of the model's size. The result is a small file that can be swapped in and out on top of the same base model.
Quantized Low-Rank Adaptation (QLoRA)
LoRA on a squeezed 4-bit copy of the model. Now it fits on one card.
QLoRA is a way to fine-tune a large model on a single graphics card. It first squeezes the frozen model down to 4-bit numbers, so it takes far less memory. Then it trains small LoRA add-ons on top, in the usual way. The paper that brought it in tuned a model with 65 billion weights on one card with 48 GB of memory. The results matched full 16-bit tuning on its tests.
In one line
- Low-Rank Adaptation (LoRA)
- A cheap way to fine-tune. The big model is locked and small parts are trained.
- Quantized Low-Rank Adaptation (QLoRA)
- LoRA on a squeezed 4-bit copy of the model. Now it fits on one card.