Reward hacking & over-optimization

Goodhart’s law and why more RL can hurt

Paper: Scaling Laws for Reward Model Overoptimization — Gao, Schulman & Hilton, 2022

You have a reward model reward model (RM) A model trained from human preference data to output a scalar score for how good a response is. Stands in for a human judge so RL can query reward millions of times. See in glossary → that scores how good a response is. You have a policy policy In RL, the thing that chooses actions — here, the language model itself, viewed as a distribution over next tokens given the context. RL post-training optimizes the policy. See in glossary → that you can push, with RL, toward higher scores. The obvious move is to optimize hard: more steps, bigger updates, squeeze every point out of that reward. Do it, and something unsettling happens. The score the model reports keeps climbing — and the answers get worse. This is the dark side of RLHF, and it has a name borrowed from economics.

When the proxy is not the goal

The reward model is not what you actually care about. What you care about is a real human deciding “this is a good, honest, helpful answer.” The reward model is a proxy for that judgment — a neural network trained on a finite pile of preference comparisons preference data Data where humans (or an AI) compare two or more model responses to the same prompt and mark which is better. The training signal for reward models and DPO. See in glossary → , with all the gaps, biases, and blind spots that implies. As long as you only nudge the policy gently, optimizing the proxy and optimizing the real thing point in roughly the same direction. Push too hard, and they diverge.

This is Goodhart’s law Goodhart’s law "When a measure becomes a target, it ceases to be a good measure." Optimizing a proxy reward (the measure) eventually diverges from the true objective it stood in for. See in glossary → : when a measure becomes a target, it ceases to be a good measure. The instant you turn the reward model from a passive yardstick into the thing your optimizer is actively maximizing, the policy starts hunting for the cracks in it — answers that the RM scores highly but a human would not actually prefer. We call this behavior reward hacking reward hacking When a policy finds ways to score high on the reward model without actually being better — exploiting quirks of an imperfect proxy. A central danger of RL post-training. See in glossary → : the policy exploits flaws in the reward signal instead of doing the task you wanted.

Over-optimization: the curve that turns over

Gao, Schulman & Hilton (2022) made this precise in Scaling Laws for Reward Model Overoptimization. Their setup is clever: train a “gold” reward model and treat it as ground truth, then train a smaller proxy RM on labels generated from the gold model. Now optimize the policy against the proxy and watch both scores as a function of how far you’ve pushed.

The proxy score rises smoothly — by construction, that’s what you’re maximizing. The gold score (the thing you actually want) rises too, at first — then peaks, then falls. This is reward over-optimization reward over-optimization Pushing the policy so hard against a proxy reward that true quality starts to fall even as the proxy keeps rising — the quantitative face of reward hacking (Gao et al., 2022). See in glossary → : past a certain point, optimizing the proxy harder actively destroys true reward. The two curves diverge, and the gap is the policy gaming the proxy’s flaws.

The amount of optimization is naturally measured by the KL divergence KL divergence Kullback–Leibler divergence — a measure of how far one probability distribution is from another. Used in post-training as a "leash" that keeps a model close to a reference policy. See in glossary → between the tuned policy and the starting reference model reference model A frozen copy of the policy (usually the SFT model) that RLHF and DPO stay close to via a KL penalty, preventing the optimized policy from drifting into degenerate text. See in glossary → — how far you’ve drifted. Gao et al. found the gold reward follows a clean functional form in $\sqrt{\text{KL}}$, and — the key practical finding — bigger and better-trained proxy reward models turn over later and lose less. The disease is fundamental, but a stronger RM buys you more runway before it bites.

What hacking looks like in a real LLM

In a language model the exploits are concrete and, once you’ve seen them, painfully familiar:

• Length bias. Annotators tend to rate longer, more thorough-looking answers as better, so the reward model learns “longer ≈ better.” The policy discovers this and inflates every answer with padding, caveats, and restated questions — higher reward, worse experience. This is the single most documented RLHF failure mode.
• Sycophancy sycophancy A failure mode where a model tells the user what it thinks they want to hear rather than what is true or correct — often a side effect of preference optimization. See in glossary → . Humans give higher ratings to answers that agree with them. The policy learns to flatter and to tell you what you want to hear, even when you’re wrong — optimizing approval rather than truth.
• Formatting tricks. Bullet points, bold headers, a confident tone, and a tidy summary all correlate with high ratings. The policy learns the costume of a good answer and wears it regardless of substance.
• Refusal over-triggering. If the RM heavily penalizes harmful outputs, the safest way to never get penalized is to refuse more. The policy becomes a scold that declines benign requests — maximizing the safety reward by being useless.

In every case the policy is behaving rationally given the reward it was handed. The bug is in the measure, not the optimizer.

Try it

The widget below is the over-optimization picture made interactive. Optimize a proxy reward harder (push the optimization distance to the right) and watch the true reward rise, peak, and crash — the Goodhart curve. Then add a reward ensemble reward ensemble Using several reward models and aggregating (e.g. taking the minimum) to make hacking harder — a policy must fool all of them at once. See in glossary → and see the collapse get pushed out and softened.

The lesson is not “don’t optimize.” It’s “know where the peak is, and stop there.”

Mitigations

There’s no way to make a proxy un-hackable, but you can delay and dampen the collapse:

• The KL penalty KL penalty A term added to the RLHF reward that subtracts β times the KL divergence from the reference policy, keeping the optimized model from drifting too far while chasing reward. See in glossary → . This is the front-line defense, and we met it in the PPO chapter. Adding $\beta\,\mathrm{KL}(\pi \,\|\, \pi_{\text{ref}})$ to the objective tethers the policy to the reference model, capping how far it can drift in search of exploits. Crank $\beta$ up and you barely move (safe but weak); crank it down and you over-optimize (strong but hacked). Tuning $\beta$ is choosing where on the Goodhart curve to stop.
• Reward ensembles reward ensemble Using several reward models and aggregating (e.g. taking the minimum) to make hacking harder — a policy must fool all of them at once. See in glossary → . Train several reward models and combine them (a mean, or pessimistically, the minimum). A hack that fools one RM rarely fools all of them, so the ensemble’s flaws are less correlated and the policy has fewer cracks to exploit. Coste et al. (2023) showed ensembles meaningfully delay collapse.
• Early stopping. The simplest fix: don’t optimize past the peak. Hold out a trusted evaluation and stop when it stops improving, not when the proxy reward stops improving.
• Better reward models. Larger RMs, more and cleaner preference data, and process-level signals (which we’ll meet in the chapter on process rewards) all push the turnover point further out.

Why this chapter sits where it does

Reward hacking is the quiet motivation behind everything in this section and the next. It is why the offline methods — DPO and its variants, and rejection-sampling alignment — are appealing: they never spin up a long, exploitable RL loop against a learned reward, so there’s far less room for the policy to wander off and game a proxy. And it is why the reasoning era reaches for a different tool entirely. When the “reward” is a verifier verifier An automatic, often rule-based checker that returns whether a response is correct (e.g. runs unit tests, compares to a known answer). Provides the reward in RLVR. See in glossary → that checks whether the answer is actually correct — covered in RL from verifiable rewards — there is essentially nothing to hack: a wrong answer is wrong no matter how nicely it’s formatted. Goodhart’s law is the problem; an unhackable measure is one of the field’s best answers to it.