Section 11

Reward models

Bradley–Terry and what an RM really learns

Statistical foundation: Bradley & Terry, Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons (Biometrika, 1952)

Step 2 of the RLHF recipe asks for something that sounds almost paradoxical: turn a pile of human “this beats that” judgments into a function that can score any response with a single number, including responses no human ever ranked. That function is the reward model, and this chapter is about what it is, how it’s trained, and (just as important) what it secretly is and isn’t.

What a reward model is in practice

A reward model is commonly a language-model backbone with a scalar head: a final output layer which outputs a single number. The backbone may be initialized from a pre-trained or SFT checkpoint; the usual language-model vocabulary head is replaced or bypassed for scoring. A linear head maps a sequence representation (often the hidden state at the final response token) to one number, the reward $r_\theta(x, y)$ for response $y$ given prompt $x$ .

Starting from a capable language model is useful because the reward model needs to interpret the prompt and response. The scalar head is normally initialized afresh, while the backbone is then fine-tuned as well; implementations differ in whether they start from a base or SFT checkpoint.

The Bradley–Terry model: from scores to preferences

Here’s the central problem. Our data is comparisons ( $y_w$ beat $y_l$ ) but we want to output a scalar score. We need a bridge that says: given two scores, what’s the probability a human prefers one over the other? The Bradley–Terry model, from 1952, is exactly that bridge.

It says the probability that the winner is preferred to the loser is the softmax (here in its two-item form, the logistic sigmoid $\sigma$ ) of the difference in their scores:

P(y_w \succ y_l) = \sigma\big(r(y_w) - r(y_l)\big) = \frac{1}{1 + e^{-(r(y_w) - r(y_l))}}

First, the notation: $P(y_w \succ y_l)$ reads “the probability that $y_w$ is preferred to $y_l$ ”. The $\succ$ symbol is the standard preference-ordering symbol (a curly “greater than”: preferred to, rather than numerically bigger than), and the subscripts follow the dataset convention from above: $w$ for the winning response, $l$ for the losing one.

Read it intuitively. If the two scores are equal, the difference is zero and $\sigma(0) = 0.5$ , a coin flip, as it should be. As the winner’s score pulls ahead, the difference grows positive and the probability climbs toward 1. The model is a smooth, probabilistic statement that bigger score means more likely to be preferred, with the gap controlling how confident the preference is.

The logistic sigmoid, up close

Since $\sigma$ carries the whole bridge, it’s worth pausing on what this function actually is. The logistic sigmoid is

\sigma(z) = \frac{1}{1 + e^{-z}},

a machine that takes any real number (a score gap of $-3$ , $0.2$ , $+40$ , anything) and squashes it into a probability strictly between 0 and 1. Plotted, it’s an S-shaped curve: near-zero on the far left, climbing steeply through $\sigma(0) = 0.5$ in the middle, then flattening out near 1 on the far right. Four of its properties do all the work in this chapter:

It’s monotonic. A bigger input always means a bigger output, so “higher reward gap” always translates to “more likely to be preferred”, never the reverse. That is the minimum sanity requirement for a preference model.
It’s symmetric around the midpoint: $\sigma(-z) = 1 - \sigma(z)$ . Swap winner and loser and the probability flips to its complement, so $P(y_w \succ y_l) + P(y_l \succ y_w) = 1$ automatically. The two possible outcomes always account for all the probability, with no extra bookkeeping.
It saturates. Once the gap is large (around $|z| > 4$ , where $\sigma(z)$ passes $0.98$ ), pushing it further barely changes the probability. A reward model that scores the winner miles above the loser gets almost no additional credit over one that scores it comfortably above: the loss cares about getting the order right with a decent margin, not about inflating the gap forever.
Its inverse is the log-odds. If $p = \sigma(z)$ , then $z = \log\frac{p}{1-p}$ . So the score gap isn’t just some number that happens to produce a probability: it is the log-odds of the preference. A gap of $0$ means even odds, a gap of about $2.2$ means 9-to-1, a gap of $4.6$ means 99-to-1. This gives reward-model score differences concrete meaning: units of log-odds of winning a human comparison.

Why the sigmoid, and not some other S-curve?

The sigmoid isn’t an arbitrary squashing choice; it falls out of the Bradley–Terry model’s one assumption. Give each response a positive “strength” $e^{r}$ (exponentiating guarantees positivity), and say the human picks a response with probability proportional to its strength. Then $P(y_w \succ y_l) = \frac{e^{r_w}}{e^{r_w} + e^{r_l}}$ (writing $r_w$ and $r_l$ as shorthand for the scores $r(y_w)$ and $r(y_l)$ ): the winner’s share of the total strength. Divide top and bottom by $e^{r_w}$ and you get $\frac{1}{1 + e^{-(r_w - r_l)}}$ : the sigmoid of the score difference, exactly the formula above. This is also why the chapter calls $\sigma$ the two-item softmax: run the softmax over exactly two scores and this expression is what it computes. The same function shows up as the neuron activation in binary classification, and for the same underlying reason: it is the canonical way to turn an unbounded score difference into the probability of a two-way outcome.

Training the reward model

Now training is a one-liner. We have a labeled preference $(x, y_w, y_l)$ : we observed the human prefer $y_w$ . The Bradley–Terry model gives us the probability our reward function assigns to that observation. We just maximize that probability, which is the same as minimizing its negative log, a maximum-likelihood fit. For a single example:

\mathcal{L}(\theta) = -\log \sigma\big(r_\theta(x, y_w) - r_\theta(x, y_l)\big)

This is the entire reward-model objective. It is exactly a binary cross-entropy loss on the preference label, and it does precisely what you’d want: it pushes the score of the chosen response up and the score of the rejected response down, until the gap between them is large enough that $\sigma$ of the difference is close to 1. Average it over your whole dataset of triples, run gradient descent, and you have a reward model.

Where the loss comes from

The derivation is three short moves.

(1) Model the data: assume each comparison follows Bradley–Terry, so $P(y_w \succ y_l) = \sigma(r_\theta(x,y_w) - r_\theta(x,y_l))$ .

(2) Write the likelihood: the probability of the whole dataset is the product of that term over every observed preference (assuming independence).

(3) Take negative log: products become sums, and maximizing the likelihood becomes minimizing $\sum -\log\sigma(\Delta r)$ . Reading that formula left to right: the $\sum$ runs over every preference pair in the dataset; $\Delta r = r_\theta(x, y_w) - r_\theta(x, y_l)$ is the winner’s score minus the loser’s; $\sigma$ squashes that gap into a probability between 0 and 1; and $-\log\sigma(\cdot)$ is the penalty, near 0 when the model already scores the winner well above the loser (so $\sigma \to 1$ ), and large when it gets the pair wrong. Minimizing the sum therefore pushes the winner’s score above the loser’s on every comparison.

That’s it: the RM loss is just maximum-likelihood estimation under the Bradley–Terry model, identical in form to logistic regression where the “feature” is the score difference the network itself learns to compute.

What the reward model actually learns, and a subtlety

It’s tempting to think the reward model learns “the quality” of a response in some absolute sense. It doesn’t. Look again at the loss: only the difference $r_\theta(x, y_w) - r_\theta(x, y_l)$ ever appears. If you added the same constant to every score the model produces, every difference (and therefore every preference probability and the entire loss) would be completely unchanged.

This means the reward model is only identified up to an additive function of the prompt: adding the same $c(x)$ to every candidate response for prompt $x$ leaves every within-prompt comparison unchanged. Its absolute scale and offset are therefore not directly identified by pairwise data; relative scores for comparable candidates are what the loss constrains. Implementations may normalize or calibrate scores for optimization, but that is an additional convention rather than information recovered from Bradley–Terry labels.

And what the model has actually learned is a proxy for human judgment: a compression of thousands of human “this beats that” clicks into a function that generalizes to new responses. It is not truth, not a ground-truth quality oracle; it’s a learned approximation, and a flawed one. That flaw is not a footnote. Optimizing too hard against an imperfect proxy is reward hacking, the failure mode that gets its own chapter (19), and measuring how good a reward model even is launched a benchmark, RewardBench, that we cover in chapter 31.

Try it

The widget below makes Bradley–Terry tangible. Set an underlying score for each of two responses and watch how the preference probability $P(A \succ B) = \sigma(s_A - s_B)$ responds. Notice the two facts from above: equal scores give exactly 0.5, and only the gap between the scores matters. Shift both by the same amount and the probability doesn’t budge.

Bradley–Terry: scores to preferences

A reward model gives each response a hidden scalar score. The chance a human prefers one over the other is the logistic of the score gap.

Response A · underlying score s_A = 1.20

Response B · underlying score s_B = -0.40

−30+3

P(A ≻ B) = σ(s_A − s_B)

score gap s_A − s_B = 1.60

P(A beats B)

83.2%

P(B beats A) = 16.8%

A reward model turns pairwise human preferences into scalar scores via exactly this Bradley–Terry / logistic relationship. Pressing Train on "A > B" applies one gradient step of the loss −log σ(s_A − s_B): it nudges s_A up and s_B down, and the step shrinks as the model already becomes confident that A wins.

Where this is going

We now have a learned, automatic stand-in for human preference: a scalar reward we can query on any response. Chapter 12 asks what happens when even the comparisons feeding this model come from an AI rather than a human: the move to RLAIF and Constitutional AI. After that, Section 4 finally opens the other black box from the recipe: the reinforcement learning that turns this reward signal into an improved policy.