Reward model explained: definition, role and limits

Level	Advanced
Origin	2017
Introduced by	Christiano et al.

Level

Advanced

Origin

2017

Introduced by

Christiano et al.

How it works

First, annotators compare two answers to the same prompt and pick the better one. The reward model learns to reproduce those choices: it must give the preferred answer a higher score than the rejected one. Technically, this usually relies on a Bradley-Terry loss, which turns the gap between the two scores into the probability that the first answer is preferred.

Once trained, this “judge” can score any new answer. In RLHF, the main model generates answers, the reward model scores them, and the reinforcement learning algorithm nudges the main model towards the higher-scoring ones.

Origins

Learning a reward function from human comparisons was formalised in 2017 by Christiano et al. in “Deep reinforcement learning from human preferences”, using games and simulated robots. It was then applied to language, notably by Stiennon et al. in 2020 for text summarisation and by OpenAI with InstructGPT in 2022.

A concrete example

For a cooking assistant, annotators see two pancake recipes: one gives exact quantities and resting time, the other stays vague. The reward model learns that precision is valued. During training it scores detailed recipes higher, so the main model produces more of them.

Limits and misconceptions

“The reward model knows the truth.” It predicts human preference, biases included: length, a confident tone, flattery. Those biases are passed on to the trained model.
“The more you optimise, the better.” Gao, Schulman and Hilton showed in 2022 that beyond a certain point, optimising against a reward model degrades real quality: Goodhart’s law, also known as reward hacking. Hence the penalty that keeps the model close to its starting point.
“You can’t do without one.” DPO does away with it as a separate model, and RLVR replaces it with an automatic check whenever the answer can be verified.

A further distinction separates models that score only the final answer from those that score each step of a chain of reasoning, which are useful for reasoning models.

Related terms

RLHF (reinforcement learning from human feedback) · A training method where people compare model answers, and the model learns to favour the responses humans judge helpful and safe.

DPO (direct preference optimization) · An alignment method that trains a model directly on pairs of preferred and rejected answers, with no separate reward model.

RLVR (reinforcement learning with verifiable rewards) · Reinforcement learning in which the reward comes from an automatic check, such as a correct answer or passing code tests, not from a judgement.

AI alignment · The research and methods aimed at making an AI system actually pursue the intentions and values of the people who build and use it.

LLM-as-a-judge · An evaluation method in which a language model scores or compares another model's answers against a rubric, standing in for a human grader.

Frequently asked questions

Is a reward model a language model?

Usually, yes: you start from a pre-trained language model and swap its normal output for a head that produces a single number, the score. It understands text, but it does not write answers.

How is it different from an LLM judge?

A reward model is trained specifically on preference data and returns a numerical score. An LLM judge is a general-purpose model that is prompted to assess an answer. The two roles increasingly overlap.

Why not just have people do the scoring?

Reinforcement learning needs a vast number of answers scored continuously. People label a sample; the reward model generalises their judgements to everything else, far faster and more cheaply.

Reward model: the automatic judge that steers training

Key figures