Optimizing against the reward

Chasing a reward without breaking the model

The previous chapter gave us a learned reward: a number that scores how good a response is, distilled from human comparisons. That’s half of RLHF. The other half is the part we’ve carefully avoided naming so far: the algorithm that takes that reward and actually changes the model to earn more of it. That algorithm is Proximal Policy Optimization (PPO), and the next several chapters lean on it constantly. This short chapter gives you a working understanding of what it does and why, so the name stops being a black box. The full, rigorous construction (where the update rule comes from) is built from the ground up later, in the RL fundamentals section; here we just want the shape of the idea.

The goal: make high-reward responses more likely

Recall the framing from chapter 2: the model is a policypolicyIn RL, the thing that chooses actions — here, the language model itself, viewed as a distribution over next tokens given the context. RL post-training optimizes the policy.See in glossary → $\pi_\theta$ that, given a prompt, defines a probability distribution over responses. We now have a reward $r(y)$ for any response $y$. The objective is exactly what you’d guess. Produce responses the reward likes:

Let’s read that off piece by piece. $\text{maximize}_\theta$ means “adjust the parameters $\theta$ to make the following quantity as large as possible.” $\theta$ is the only thing we get to change. $\mathbb{E}$ is an expectation (an average) and the subscript $y \sim \pi_\theta$ says what we’re averaging over: responses $y$ sampled from the policy $\pi_\theta$ (the "$\sim$" is read “drawn from”). Inside the brackets, $r(y)$ is the reward of one such response. Put together: it’s the expected reward the policy earns, averaged over all the responses it might generate, and we want to pick the weights that push that average up. Nothing here is new; it’s the expectation from chapter 2, now with a real reward attached.

The basic move to increase it is intuitive, and you don’t need any heavy machinery to picture it. Sample some responses from the current model. Score each with the reward. Then adjust the weights so that the responses that scored well become a little more probable, and the ones that scored badly become a little less probable. Repeat. Over many rounds, probability mass flows toward the kind of answers the reward rewards. (That “nudge up what worked, down what didn’t” is the seed of the policy gradientpolicy gradientA family of RL methods that directly adjust the policy’s parameters in the direction that increases expected reward, using the score-function (REINFORCE) estimator.See in glossary →, which the RL section explains in full.)

The danger: optimize too hard and the model breaks

If that were the whole story, you could just crank the optimizer. The catch is that the reward is a learned, imperfect proxy, and an optimizer pushed hard enough will find and exploit its flaws. Take steps that are too large, or run too long, and the policy drifts off into reward-hackingreward hackingWhen a policy finds ways to score high on the reward model without actually being better — exploiting quirks of an imperfect proxy. A central danger of RL post-training.See in glossary → territory: text that scores 9.8 under the reward model while being repetitive, sycophantic, or outright gibberish to a human. The optimizer is doing its job perfectly; the measure is the thing that’s wrong.

So the real problem RLHF has to solve isn’t “increase the reward”: that part is easy. It’s “increase the reward without wandering off into nonsense.” Everything distinctive about PPO is machinery for taking that walk carefully.

PPO’s answer: small steps, on a leash

PPOPPOProximal Policy Optimization (Schulman, 2017) — nudges the policy toward higher reward in small, clipped steps with a KL leash to a reference model. The RLHF workhorse: stable, simple, widely used.See in glossary → is the algorithm that does this carefully, and its name is the whole idea: proximal means stay close to where you are. It keeps two safeguards bolted onto the basic “nudge toward reward” loop:

• Clipped updates. PPO refuses to move any single response’s probability too far in one step. Even if the reward says “make this way more likely,” the update is clipped to a modest change. This keeps each step inside a region the model can trust, so the optimizer takes many small, safe steps instead of a few reckless ones.
• A KL penaltyKL penaltyA term added to the RLHF reward that subtracts β times the KL divergence from the reference policy, keeping the optimized model from drifting too far while chasing reward.See in glossary → to a reference. A frozen copy of the model from before RL — the reference modelreference modelA frozen copy of the policy (usually the SFT model) that RLHF and DPO stay close to via a KL penalty, preventing the optimized policy from drifting into degenerate text.See in glossary → (usually the SFT checkpoint) — acts as an anchor. PPO adds a penalty that grows as the policy’s output distribution drifts away from that reference, measured by the KL divergenceKL divergenceKullback–Leibler divergence — a measure of how far one probability distribution is from another. Used in post-training as a "leash" that keeps a model close to a reference policy.See in glossary → we met in chapter 3. The reward pulls the policy toward higher scores; the KL penalty pulls it back toward fluent, sensible language. The balance between the two is the leash.

Put together, the thing PPO actually optimizes is, in spirit:

Here $\rho(y) = \pi_\theta(y) / \pi_{\theta_{\text{old}}}(y)$ is the probability ratio: how much more (or less) likely the update in progress has made response $y$, compared to before the step. The $\operatorname{clip}$ is the first safeguard written in symbols. Once $\rho$ leaves the band $[1-\varepsilon,\, 1+\varepsilon]$ (with $\varepsilon$ typically around $0.2$), the clipped term goes flat, and the $\min$ takes that cautious branch, so the objective stops rewarding any further movement of this response’s probability in this step. That is all “clipped updates” means: the gradient simply shuts off outside the band. (The exact objective swaps $r(y)$ for a per-token advantage; chapter 17 builds it properly.) The coefficient $\beta$ sets how tight the leash is. The slider below lets you feel that tradeoff: too loose and the policy sprints off to game the reward; too tight and it never improves; the useful gains live in the middle. The step buttons show the other safeguard at work: clipped updates creep along the frontier toward the target, while a single unclipped leap overshoots it and lands in gibberish.

The takeaway

That’s the working picture of PPO. Strip the names away and it’s just careful hill-climbing on the reward: sample responses, score them, nudge the policy toward the ones that scored well, but in proximal (small, clipped) steps, kept on a leash by a KL penalty to a frozen reference, so the policy gets better at the reward without drifting into nonsense. Two words — proximal and leashed — and the name stops being a mystery.

We’ve taken the update rule itself largely on faith here: why nudging toward higher-scoring samples is the right move, exactly what it means to “clip a step,” and how to judge whether a single response was good. Those are real questions with real answers, but the intuition above is enough to use “PPO” confidently as the optimizer that turns a reward into a better model.