Skip to content
BENIBIX Learn

Quantized Low-Rank Adaptation (QLoRA)

AI💬 LLMsLanguage Model Training

Also known as QLoRA, Quantized LoRA

LoRA on a squeezed 4-bit copy of the model. Now it fits on one card.

Think of it like

It is like packing a big coat in a small bag. Then you sew a patch on it.

Example

Emma wants to tune a large open model, but she has one 48 GB card. With QLoRA the squeezed model fits, and only the small add-ons are trained.

Why it matters

Now people can tune big models at home, with no data centre.

Definition

QLoRA is a way to fine-tune a large model on a single graphics card. It first squeezes the frozen model down to 4-bit numbers, so it takes far less memory. Then it trains small LoRA add-ons on top, in the usual way. The paper that brought it in tuned a model with 65 billion weights on one card with 48 GB of memory. The results matched full 16-bit tuning on its tests.
QLoRA trains the same add-on as LoRA. It does it on a squeezed 4-bit copy of the model.

Common mistake

Thinking the squeezed model is what you ship. The 4-bit copy is for training, and you pick the format you serve later.

Key takeaways

  1. 1
    Squeeze the base. Frozen weights in 4-bit.
  2. 2
    Train LoRA on top. The same small add-ons.
  3. 3
    One card. Big models fit in less memory.

Compare

Learn it properly