Skip to content
BENIBIX Learn

Low-Rank Adaptation (LoRA)

AI💬 LLMsLanguage Model Training

Also known as LoRA, LoRA Fine-Tuning

A cheap way to fine-tune. The big model is locked and small parts are trained.

Think of it like

Think of it like you add a small sticker to a big map. The map stays the same.

Example

Jack fine-tunes an 8B model with LoRA on one graphics card in an afternoon. The result is a 40 MB file he loads on top of the base model.

Why it matters

It lets one person with one graphics card fine-tune a model. Before, that took a lab.

Definition

Low-rank adaptation is a cheap way to fine-tune a large model. The model's own weights are locked. Small extra pieces are added next to them, and only those are trained. These pieces are often less than 1 percent of the model's size. The result is a small file that can be swapped in and out on top of the same base model.
The big grid is locked. Only the two thin strips beside it are trained, then added on top.

Common mistake

Thinking the file works on any model. It only fits the exact base model it was trained on.

Key takeaways

  1. 1
    Base stays locked. Its weights do not change.
  2. 2
    Small add-on pieces. Often under 1 percent.
  3. 3
    Swap in and out. A small file per skill.

Compare

Learn it properly