DPO: learning human preferences without a reward model
DPO (direct preference optimization) teaches a language model to favour some answers over others by training it directly on pairs of a preferred and a rejected response. It targets the same goal as RLHF, with a procedure that is far simpler to run.
Data as of October 22, 2026
Key figures
| Level | Advanced |
|---|---|
| Origin | 2023 |
| Introduced by | Rafailov et al. |
DPO in one sentence, then in detail
Instead of first learning to score answers and then optimising the model against that score, DPO directly makes the preferred answer more likely and the rejected one less likely.
In classic RLHF, you train a reward model on human comparisons, then optimise the main model with reinforcement learning (often the PPO algorithm), penalising it for drifting too far from a reference model. DPO starts from that same mathematical objective and shows it has an exact solution that can be written as a simple classification loss over pairs. For each pair, the model compares the probability it assigns to both answers with the probability a frozen reference model assigns to them. A hyperparameter, β, controls how far the model may move away from its starting point.
A concrete example
Prompt: “Summarise this rental agreement in five bullet points.” Answer A: five clear bullets covering term, rent, deposit, notice period and service charges. Answer B: one long paragraph that copies clauses verbatim. An annotator picks A. After many such pairs, a DPO-trained model produces structured summaries on its own, without any separate “judge” ever being trained.
Origins and why it matters
The method was introduced in May 2023 by Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher Manning and Chelsea Finn at Stanford, in the paper “Direct Preference Optimization: Your Language Model is Secretly a Reward Model”. The authors describe it as more stable and computationally lighter than PPO-based RLHF, because it does not need to sample answers during fine-tuning and requires little hyperparameter tuning.
That simplicity explains its popularity in the open-model ecosystem: the Tülu 3 recipe from the Allen Institute for AI, for instance, chains supervised fine-tuning, DPO and RLVR.
Common misconceptions
- “DPO has nothing to do with reward models.” Quite the opposite: as the paper’s title says, the language model implicitly plays that role itself.
- “No reinforcement learning means no risk of drift.” DPO can overfit quirks of the preference data, for example favouring longer answers if annotators often preferred them.
- “DPO is enough for alignment.” It only learns what the pairs contain. A model’s alignment also depends on data, evaluations and safeguards.
Related terms
- RLHF (reinforcement learning from human feedback) · A training method where people compare model answers, and the model learns to favour the responses humans judge helpful and safe.
- Reward model · An auxiliary model that scores an answer by predicting human preference, used as the training signal for another model.
- Fine-tuning · Continuing the training of an already trained model on a small, targeted set of examples to adapt it to a task, style or output format.
- AI alignment · The research and methods aimed at making an AI system actually pursue the intentions and values of the people who build and use it.
- RLVR (reinforcement learning with verifiable rewards) · Reinforcement learning in which the reward comes from an automatic check, such as a correct answer or passing code tests, not from a judgement.
Full glossary
Frequently asked questions
Has DPO replaced RLHF?
Not entirely. Many recipes chain several stages: supervised fine-tuning, DPO, then sometimes online reinforcement learning or verifiable rewards. DPO is often picked for its simplicity and stability, not because it always beats the alternatives.
What data do I need for DPO?
A set of preference pairs: for each prompt, one answer judged better and one judged worse. The pairs can come from human annotators or be scored by another model. Their quality matters more than their sheer number.
Can I run DPO on an open-source model?
Yes, and that is one of its most common uses, because it needs neither a separate reward model nor answer generation during training. The most widely used fine-tuning libraries support it.
Data sources
Data as of October 22, 2026
Report an error
Does a figure, a date or a fact look wrong? Tell us: every report is checked.