Skip to content
BENIBIX Learn

Direct Preference Optimization (DPO)

AI💬 LLMsLanguage Model Training

Also known as DPO, Direct Preference Optimisation

A model learns from pairs of answers. People picked the best one in each pair.

Think of it like

It is like a coach who shows you two tries and says which one was best.

Example

A team has 10,000 pairs of answers, each with a winner picked by a person. They run DPO on their chat model, and it starts to answer more like the winners.

Why it matters

It gives most of the gain of RLHF with far less setup, so smaller teams can use it.

Definition

Direct preference optimization is a way to teach a language model which answers people like best. It uses pairs of answers to the same prompt. In each pair, a person marked one as better. The model learns to make the better answer more likely and the worse one less likely. Unlike RLHF, it needs no separate reward model and no trial and error loop. So it is simpler and steadier to run.
DPO sits in the middle step, where the model learns which answer people prefer.

Common mistake

Thinking the pairs can be sloppy. The model learns whatever the picks reward, mistakes included.

Key takeaways

  1. 1
    Pairs of answers. One marked better.
  2. 2
    No reward model. It learns from the pairs.
  3. 3
    Simpler than RLHF. One training step, not a loop.

Compare

Learn it properly