Skip to content
BENIBIX Learn

DPO vs RLHF

People think DPO is just a kind of RLHF, and it is not. Both teach a model from pairs of answers that people ranked. RLHF first trains a reward model to score answers. Then it uses trial and error to push the chat model toward high scores. DPO skips both steps and trains the chat model on the pairs straight away. So it is simpler to run.

AI💬 LLMs

Direct Preference Optimization (DPO)

A model learns from pairs of answers. People picked the best one in each pair.

Direct preference optimization is a way to teach a language model which answers people like best. It uses pairs of answers to the same prompt. In each pair, a person marked one as better. The model learns to make the better answer more likely and the worse one less likely. Unlike RLHF, it needs no separate reward model and no trial and error loop. So it is simpler and steadier to run.

Example
A team has 10,000 pairs of answers, each with a winner picked by a person. They run DPO on their chat model, and it starts to answer more like the winners.
Why it matters
It gives most of the gain of RLHF with far less setup, so smaller teams can use it.
Full entry
AI💬 LLMs

Reinforcement Learning from Human Feedback (RLHF)

A model learns which answers people like best. Then it gives more of those.

Reinforcement learning from human feedback trains a model to give the kind of answers people prefer. First, people compare pairs of answers and pick the better one. A reward model learns from those picks to score any answer. Then the language model is trained to write answers that score high. It makes replies more helpful and polite, and less likely to cause harm.

Example
Workers see two replies to 'How do I fix a flat tyre?' and pick the clearer one, 100,000 times over. The model then learns to write the clearer kind.
Why it matters
It is a big reason chat helpers are kind and useful, not odd or rude.
Full entry

In one line

Direct Preference Optimization (DPO)
A model learns from pairs of answers. People picked the best one in each pair.
Reinforcement Learning from Human Feedback (RLHF)
A model learns which answers people like best. Then it gives more of those.