# Direct Preference Optimization (DPO)

> A model learns from pairs of answers. People picked the best one in each pair.

**Field:** Artificial Intelligence  
**Category:** LLMs  
**Also known as:** DPO, Direct Preference Optimisation

## Think of it like

It is like a coach who shows you two tries and says which one was best.

## Definition

Direct preference optimization is a way to teach a language model which answers people like best. It uses pairs of answers to the same prompt. In each pair, a person marked one as better. The model learns to make the better answer more likely and the worse one less likely. Unlike RLHF, it needs no separate reward model and no trial and error loop. So it is simpler and steadier to run.

## Example

A team has 10,000 pairs of answers, each with a winner picked by a person. They run DPO on their chat model, and it starts to answer more like the winners.

## Why it matters

It gives most of the gain of RLHF with far less setup, so smaller teams can use it.

## Common mistake

Thinking the pairs can be sloppy. The model learns whatever the picks reward, mistakes included.

## Three principles

- **Pairs of answers.** One marked better.
- **No reward model.** It learns from the pairs.
- **Simpler than RLHF.** One training step, not a loop.

## Related terms

- [Reinforcement Learning from Human Feedback (RLHF)](https://learn.benibix.com/term/ai/reinforcement-learning-from-human-feedback)
- [Reward Model](https://learn.benibix.com/term/ai/reward-model)
- [Post-Training](https://learn.benibix.com/term/ai/post-training)
- [Instruction Tuning](https://learn.benibix.com/term/ai/instruction-tuning)
- [Fine-Tuning](https://learn.benibix.com/term/ai/fine-tuning)

## Told apart from

- [DPO vs RLHF](https://learn.benibix.com/compare/dpo-vs-rlhf): People think DPO is just a kind of RLHF, and it is not. Both teach a model from pairs of answers that people ranked. RLHF first trains a reward model to score answers. Then it uses trial and error to push the chat model toward high scores. DPO skips both steps and trains the chat model on the pairs straight away. So it is simpler to run.

## Taught in

- [Post-Training and RLHF](https://learn.benibix.com/learn/ai-llms/post-training-and-rlhf): LLMs, 8/15

![A flow from left to right. Pretraining reads a huge pile of text, guessing the next word. It leads into a highlighted post-training box with three steps: show it good answers to copy, teach it which answer people prefer, and reward it for right, checkable work. That leads to the assistant you chat with. A note says step two is where RLHF and DPO sit, and the order and mix change from lab to lab.](https://learn.benibix.com/diagrams/ai2-training-stages.svg)

---

Source: https://learn.benibix.com/term/ai/direct-preference-optimization
Author: Behnia Zarei (a developer in Vienna, Austria, who builds and teaches data, AI and business intelligence at BENIBIX)
Publisher: BENIBIX Learn (https://learn.benibix.com/)
Last updated: 2026-11-18
Licence: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Quote freely with attribution.
