RLAIF & Constitutional AI

AI feedback and scalable oversight

Papers: Constitutional AI: Harmlessness from AI Feedback — Bai et al., 2022 · Llama 2: Open Foundation and Fine-Tuned Chat Models — Touvron et al., 2023 · RLAIF vs. RLHF — Lee et al., 2024

The three-step recipe can have an expensive, slow ingredient: human comparisons. They are costly, can be inconsistent, and may not scale to every task. The natural question is whether a capable model can supply some of those judgments. This chapter is about replacing some human labels with AI labels, the evidence for when that can work, and the new problems it creates.

RLAIF: swap the labeler

Canonical RLAIFRLAIFReinforcement Learning from AI Feedback — replace human preference labels with labels from another model (or the model itself), making the feedback loop cheap and scalable.See in glossary → (Reinforcement Learning from AI Feedback) mirrors an RLHF pipeline but uses AI-generated preferences in place of some human comparisons. A capable model is prompted to judge candidate responses; those judgments train a reward or preference model, which can then support policy optimization. Variants can use soft labels, direct model scores, or a direct preference loss, so a reward model and PPO are not mandatory parts of RLAIF.

Is an AI judge good enough? RLAIF vs. RLHF (Lee et al., 2024) compared the two approaches on summarization, helpful dialogue, and harmless dialogue. In its human evaluations, RLAIF was comparable to RLHF on the first two tasks and performed better on the paper’s harmlessness measure. That is encouraging evidence in those settings, not proof that AI labels preserve quality for every target behavior or judge model.

Constitutional AI: the model critiques itself

The idea was pioneered, and given its sharpest form, in Constitutional AI (Bai et al., 2022). The goal was harmlessness without armies of humans labeling toxic content, work that is both expensive and genuinely unpleasant. The method has two phases, and the elegant move is that the model improves itself against a short written document called a constitution: a list of plain-language principles like “choose the response that is least harmful” or “prefer the answer that is honest and non-evasive.”

The first phase is supervised self-revision. The model generates a response to a red-teaming prompt, then is asked to critique its own answer against a constitutional principle, and finally to revise it to better satisfy that principle. Fine-tuning on the revisions supplies synthetic demonstrations of the intended behavior. Humans still supply the constitution and prompts, and the original CAI work also retained human helpfulness data.

The second phase is RLAIF proper. The model generates pairs of responses and an AI labeler picks which one better satisfies the constitution, producing AI preference datapreference dataData where humans (or an AI) compare two or more model responses to the same prompt and mark which is better. The training signal for reward models and DPO.See in glossary →. In the original CAI work, these harmlessness labels were combined with human helpfulness labels to train a preference model, then PPO optimized the policy. It replaced direct human labels for harmfulness rather than eliminating human influence altogether.

Scalable oversight: why this matters beyond cost

The deeper motivation isn’t just saving money. It’s scalable oversightscalable oversightThe challenge of supervising models on tasks too hard or numerous for humans to label directly — addressed by AI feedback, critiques, and verifiers.See in glossary →: the problem of supervising tasks that are too hard, too numerous, or too specialized for humans to reliably judge. How do you human-label the correctness of a 2,000-line program, a subtle mathematical proof, or a million-response firehose? You often can’t, at least not fast enough or well enough. If models can help judge other models, oversight can scale alongside capability instead of being throttled by human bandwidth. AI feedback is the first rung on that ladder, and the same impulse runs straight through to the verifiable-reward methods later in this explainer.

The open case study: Llama 2-Chat

The most detailed open account of an RLHF-plus-AI-feedback pipeline at scale is Llama 2 (Touvron et al., 2023), and it’s worth knowing as the canonical example. A few choices stand out:

• Two reward models, not one. Llama 2-Chat trains a separate reward modelreward model (RM)A model trained from human preference data to output a scalar score for how good a response is. Stands in for a human judge so RL can query reward millions of times.See in glossary → for helpfulness and another for safety, because optimizing a single blended score tends to let one objective quietly swallow the other. Keeping them apart lets the team balance “be useful” against “be safe” explicitly.
• Rejection sampling and PPO. Each round, the policy samples many candidate responses; the best-scoring ones are selected and fine-tuned on using rejection samplingrejection samplingGenerate several candidate responses, keep only the best-scoring one(s) by some reward or verifier, and fine-tune on those. A simple, stable, RL-free way to improve a model.See in glossary →. Later iterations applied PPO on top of rejection-sampling checkpoints. The paper treats the two as complementary alignment tools, not rival algorithms.
• Iterative rounds. The whole loop runs repeatedly, with fresh preference data collected on each improved model: Christiano’s original “loop and refine” structure, at production scale.

Llama 2 is a detailed open account of iterative RLHF: human preference data, specialized reward models, rejection sampling, and PPO. It is not, however, a clean RLAIF case study.

The catch: who writes the constitution?

AI feedback is cheap, fast, scalable, and (usefully) consistent, since a model applies the same criteria to every example without the fatigue and drift that plague human labelers. But the benefits come with a real cost. The AI labeler’s biases get baked in. If the judge model has a blind spot, a stylistic preference, or a subtle misjudgment, that flaw is now stamped onto every preference label and propagated into the reward model and the policy. You’ve automated the judge, including its mistakes.

And the constitution itself raises the uncomfortable question the method can’t escape: who writes it? The principles encode values (about what’s harmful, what’s honest, what’s worth refusing) and those choices are made by a small group of people and frozen into a document that then shapes the model’s behavior at scale. Scalable oversight makes the mechanism of alignment cheaper; it does nothing to settle the content of what we’re aligning to.

Where this is going

We’ve now spent this section on the preference half of RLHF (comparisons, reward models, and where the labels come from, human or AI) and we sketched the optimizer at a working level back in chapter 9: PPO takes small, clipped steps on a KL leash. But its internals arrived by assertion: where does the clipped update come from, and why does it work? Section 4 finally opens that box. We’ll build the RL machinery from the ground up and answer the question the intuition left open: what is the algorithm that turns a reward signal into a better model?