RLHF: teaching an AI what people prefer
RLHF (reinforcement learning from human feedback) is a training method in which people compare several answers from a model, so that it learns to produce the ones they judge most helpful and safe. It is one of the steps that turned raw language models into conversational assistants.
Data as of October 22, 2026
Key figures
| Level | Intermediate |
|---|---|
| Origin | 2017 |
| Introduced by | Christiano et al. |
A plain definition
After pre-training, a model can continue text but does not necessarily answer helpfully. RLHF teaches it what a good answer looks like, based on human judgement. Rather than writing the ideal answer to every question, which is slow and hard, people are asked to compare several answers and pick the best. Comparing is much easier than writing.
How it works, in three steps
- Demonstrations. Annotators write examples of good answers, used for an initial round of fine-tuning.
- Comparisons. The model produces several answers to the same question and people rank them. Those rankings train a reward model that learns to predict which answer a person would prefer.
- Reinforcement. The main model is then trained to score well with that reward model, while staying close to its original behaviour.
A concrete example
Asked “How do I settle a baby who cries at night?”, a raw model might reply with a list of forum questions, an off-topic anecdote or a blunt instruction. Annotators will prefer a structured, careful answer that suggests checking with a health professional if there is any concern. After a great many comparisons like this, the model adopts that style on its own.
Origins and role in assistants
The method was described in 2017 by Christiano et al. in “Deep reinforcement learning from human preferences”, using video games and simulated robots. OpenAI applied it to language models with InstructGPT, presented in March 2022 in “Training language models to follow instructions with human feedback”: evaluators preferred answers from a much smaller model trained with RLHF over those from GPT-3. OpenAI says it trained ChatGPT using the same methods.
Limits and myths
- “RLHF makes the model honest.” It makes the model match the evaluators’ preferences. If they unconsciously favour confident or flattering answers, the model learns that too, which is a known source of sycophancy.
- “It’s the only alignment method.” RLHF is just one tool for alignment. DPO and constitutional AI are variants or alternatives.
- “It’s cheap.” Recruiting and training annotators, then checking the quality of their judgements, takes considerable work.
Related terms
- Reward model · An auxiliary model that scores an answer by predicting human preference, used as the training signal for another model.
- DPO (direct preference optimization) · An alignment method that trains a model directly on pairs of preferred and rejected answers, with no separate reward model.
- Fine-tuning · Continuing the training of an already trained model on a small, targeted set of examples to adapt it to a task, style or output format.
- AI alignment · The research and methods aimed at making an AI system actually pursue the intentions and values of the people who build and use it.
- Pre-training · The first training phase of a language model: it learns to predict the next piece of text across a huge corpus, with no specific task.
- Constitutional AI · An Anthropic training method in which an AI critiques and revises its own answers against a written list of principles called a constitution.
Full glossary
Frequently asked questions
Am I taking part in RLHF when I click thumbs up?
Your ratings may be used to improve future models, depending on the provider's terms and your settings. RLHF itself, however, relies mainly on trained annotators who compare answers against detailed guidelines.
Does RLHF make a model smarter?
Not really in terms of knowledge, which mostly comes from pre-training. It improves how that knowledge is used: following instructions, picking the right tone, avoiding certain harmful answers.
Are there alternatives to RLHF?
Yes. DPO (direct preference optimisation) does away with a separate reward model. Other approaches use AI-generated feedback based on written principles, such as constitutional AI, or rewards that can be checked automatically.
Data sources
Data as of October 22, 2026
Report an error
Does a figure, a date or a fact look wrong? Tell us: every report is checked.