💬Direct Preference Optimization (DPO)
Also known as DPO, Direct Preference Optimisation
A model learns from pairs of answers. People picked the best one in each pair.
Think of it like
It is like a coach who shows you two tries and says which one was best.
Example
A team has 10,000 pairs of answers, each with a winner picked by a person. They run DPO on their chat model, and it starts to answer more like the winners.
Why it matters
Definition
Common mistake
Key takeaways
- 1Pairs of answers. One marked better.
- 2No reward model. It learns from the pairs.
- 3Simpler than RLHF. One training step, not a loop.