DPO vs RLHF
People think DPO is just a kind of RLHF, and it is not. Both teach a model from pairs of answers that people ranked. RLHF first trains a reward model to score answers. Then it uses trial and error to push the chat model toward high scores. DPO skips both steps and trains the chat model on the pairs straight away. So it is simpler to run.
Direct Preference Optimization (DPO)
A model learns from pairs of answers. People picked the best one in each pair.
Direct preference optimization is a way to teach a language model which answers people like best. It uses pairs of answers to the same prompt. In each pair, a person marked one as better. The model learns to make the better answer more likely and the worse one less likely. Unlike RLHF, it needs no separate reward model and no trial and error loop. So it is simpler and steadier to run.
Reinforcement Learning from Human Feedback (RLHF)
A model learns which answers people like best. Then it gives more of those.
Reinforcement learning from human feedback trains a model to give the kind of answers people prefer. First, people compare pairs of answers and pick the better one. A reward model learns from those picks to score any answer. Then the language model is trained to write answers that score high. It makes replies more helpful and polite, and less likely to cause harm.
In one line
- Direct Preference Optimization (DPO)
- A model learns from pairs of answers. People picked the best one in each pair.
- Reinforcement Learning from Human Feedback (RLHF)
- A model learns which answers people like best. Then it gives more of those.