RLHF scales to language

Summarization, InstructGPT, and the 3-step recipe

Papers: Learning to Summarize from Human Feedback — Stiennon et al., 2020 · Training Language Models to Follow Instructions with Human Feedback (InstructGPT) — Ouyang et al., 2022

The previous chapter gave us the principle: learn a reward from comparisons, then optimize against it. But a principle that works on Atari and toy text continuations is a long way from a principle that builds a useful assistant. This chapter is the story of how RLHF crossed that gap (first on a single hard language task, then on the open-ended job of following any instruction) and arrived at a three-step recipe that became the template for the entire industry.

Stiennon 2020: the proof that it scales

An influential demonstration on a real NLP task was Learning to Summarize from Human Feedback (Stiennon et al., 2020). The training task was abstractive summarization of Reddit TL;DR posts, where faithful, concise summaries are hard to specify with a single automatic metric. The authors also evaluated transfer to CNN/DailyMail news articles without news-specific fine-tuning.

The recipe was exactly the loop from the last chapter. Collect human comparisons between candidate summaries, train a reward modelreward model (RM)A model trained from human preference data to output a scalar score for how good a response is. Stands in for a human judge so RL can query reward millions of times.See in glossary → to predict which summary a human would prefer, then fine-tune the summarizer with PPOPPOProximal Policy Optimization (Schulman, 2017) — nudges the policy toward higher reward in small, clipped steps with a KL leash to a reference model. The RLHF workhorse: stable, simple, widely used.See in glossary → to maximize that reward (with the KL-to-reference leash holding it steady). The striking result was that the human-feedback policies outperformed the supervised baselines in the study’s human-preference evaluations. On the Reddit task, their 1.3B feedback model received a higher raw preference score against the dataset reference summaries than a 13B supervised model. Those references are useful baselines, not an unqualified measure of human-summary quality.

That result made the value of preference data concrete: it could improve the model under the study’s human-evaluation protocol beyond supervised baselines trained on the same task. It does not show that a model universally exceeds human writers; both the preference labels and the reference-summary benchmark have limitations.

InstructGPT 2022: from one task to following instructions

Summarization is one task. The real prize was a model that follows any instruction — answer this, rewrite that, explain this, refuse that — the behavior we now expect from a chat assistant. That’s what InstructGPT (Ouyang et al., 2022) delivered, and in doing so it crystallized the canonical RLHF recipe into three crisp steps.

Step 1: Supervised fine-tuning

Start from a pre-trained base modelbase modelA model straight out of pre-training — a powerful text continuator that has not yet been taught to follow instructions, hold a conversation, or refuse harmful requests.See in glossary → and fine-tune it on a dataset of human-written demonstrations: prompts paired with high-quality responses that show the model what a good answer looks like. This is plain supervised fine-tuningsupervised fine-tuning (SFT)Training a pre-trained model on curated (prompt, response) pairs with the ordinary next-token objective, so it imitates demonstrated assistant behavior. The first stage of post-training.See in glossary → (the subject of the previous section) and it gives the model a basic grasp of the instruction-following format. The result is the SFT model, the starting point for everything that follows.

Step 2: Train a reward model

Sample several responses from the SFT model for each of many prompts, and have human labelers rank them from best to worst. Break those rankings into pairwise comparisonspairwise comparisonAsking a labeler which of two responses is better, rather than scoring each on an absolute scale. Easier and more reliable for humans, and the basis of preference learning.See in glossary → and train a reward modelreward model (RM)A model trained from human preference data to output a scalar score for how good a response is. Stands in for a human judge so RL can query reward millions of times.See in glossary → to assign a higher scalar score to the responses humans preferred. The RM is a learned, automatic proxy for human judgment, and the next chapter is devoted entirely to how it works.

Step 3: Optimize the policy with PPO

Now run reinforcement learning. The SFT model becomes the policypolicyIn RL, the thing that chooses actions — here, the language model itself, viewed as a distribution over next tokens given the context. RL post-training optimizes the policy.See in glossary →; it generates responses, the reward model scores them, and PPOPPOProximal Policy Optimization (Schulman, 2017) — nudges the policy toward higher reward in small, clipped steps with a KL leash to a reference model. The RLHF workhorse: stable, simple, widely used.See in glossary → nudges the policy toward responses the RM rates highly. Crucially, the objective also includes a KL penaltyKL penaltyA term added to the RLHF reward that subtracts β times the KL divergence from the reference policy, keeping the optimized model from drifting too far while chasing reward.See in glossary → that pulls the policy back toward the frozen SFT model (the reference modelreference modelA frozen copy of the policy (usually the SFT model) that RLHF and DPO stay close to via a KL penalty, preventing the optimized policy from drifting into degenerate text.See in glossary →) so it improves on the reward without drifting into degenerate text that merely games the RM.

The result that shocked everyone

The headline finding of InstructGPT is still widely cited. On the paper’s held-out prompt distribution and preference protocol, its human labelers preferred outputs from a 1.3-billion-parameter InstructGPT model to outputs from the 175-billion-parameter GPT-3 baseline, a model more than 100× larger. This is evidence that post-training can matter enormously for that target behavior; it is not a general conversion rate between alignment and parameter count.

The lesson is foundational for everything in this explainer: post-training can substantially change how a base model’s capabilities are elicited and expressed. It can also introduce new skills and trade-offs through its data and objectives. InstructGPT made GPT-3-style models more likely to follow instructions under its evaluation protocol; it did not establish a clean separation between “intelligence” and behavior.

Where this is going

We now have the recipe (SFT, then a reward model, then PPO against it with a KL leash) and the historical proof that it works and scales. The next chapters open up the two pieces we’ve leaned on without fully building. First, in chapter 11, the reward model: how a single scalar head, trained with the Bradley–TerryBradley–Terry modelA simple rule for turning "A beats B" comparisons into a single score per item: the bigger an item's score over another, the more likely it wins. A reward model produces exactly such a score.See in glossary → loss, learns to stand in for human judgment. Then the RL machinery itself in Section 4: the full construction behind the working picture of PPO from chapter 9, and why it’s shaped the way it is.