The alignment problem

Helpful, honest, harmless — and why imitation isn’t enough

Sources: A General Language Assistant as a Laboratory for Alignment — Askell et al., 2021; Training language models to follow instructions with human feedback — Ouyang et al., 2022

Suppose you had unlimited budget and the patience of a saint. You hire the world’s best writers and have them produce, by hand, a perfect assistant response to every conceivable prompt: millions of flawless demonstrations. Then you fine-tune your base model on all of it. Would you have a perfectly aligned assistant?

Not necessarily. Even extraordinarily broad demonstrations leave questions about coverage, conflicting values, distribution shift, and how the system behaves in novel situations. Understanding those limits is the conceptual hinge of this whole explainer. It explains why the field supplements “show the model good behavior” with preference signals, verifiers, evaluations, and other feedback. This chapter makes that case.

What we mean by alignment

AlignmentalignmentThe problem of making a model behave in accordance with human intent and values — helpful, honest, and harmless — rather than merely continuing text plausibly.See in glossary → is the project of making a model’s behavior match what its developers and users actually want, rather than what falls out of next-token prediction by default. A base model is capable but not aimed: it has the knowledge and the fluency, but no built-in disposition to be useful, truthful, or safe. Alignment is the aiming.

The field’s most durable shorthand for “what we want” is the HHHhelpful, honest, harmlessThe "HHH" framing (from Anthropic) of what an aligned assistant should be: useful to the user, truthful, and unlikely to cause harm.See in glossary → framing (helpful, honest, harmless) introduced by Anthropic’s Askell et al. in 2021. It’s worth taking each word seriously, because they pull in different and sometimes conflicting directions:

• Helpful. Actually does what the user is asking: answers the question, follows the format, completes the task, asks for clarification when genuinely needed.
• Honest. Says true things, expresses appropriate uncertainty, doesn’t fabricate citations or confidently invent facts. This includes calibration: knowing (and signaling) what it doesn’t know.
• Harmless. Declines to help with genuinely dangerous requests, avoids generating abusive or deceptive content, and does so without being preachy or refusing benign requests out of excess caution.

These three pull against each other constantly. Maximal helpfulness (“I’ll answer anything”) fights harmlessness. Maximal harmlessness (“I refuse if there’s any doubt”) fights helpfulness. A model that tells you only what you want to hear feels helpful but isn’t honest. Alignment isn’t a single target; it’s a balance, and that’s part of why it can’t be reduced to imitating one fixed set of demonstrations.

Why imitation isn’t enough

Supervised fine-tuningsupervised fine-tuning (SFT)Training a pre-trained model on curated (prompt, response) pairs with the ordinary next-token objective, so it imitates demonstrated assistant behavior. The first stage of post-training.See in glossary → (imitating demonstrations, the subject of the next section) gets you remarkably far. The first useful chat assistants were largely SFT. But it hits three walls, and each one motivates a piece of machinery we’ll build later.

Wall 1: demonstrations are expensive and difficult to cover

Every SFT example is a complete, high-quality response written (or vetted) by a human. That can be slow and costly. Imitation training can generalize beyond individual examples and combine capabilities learned during pre-training, so it is not literally capped at an individual demonstrator’s performance. But its training signal is limited by the coverage, quality, and consistency of the demonstrations it receives.

Worse, for hard tasks (a subtle proof, a tricky piece of code, a delicate refusal), writing the ideal response from scratch is genuinely difficult even for experts. The supply of perfect demonstrations is thin exactly where you need them most.

Wall 2: comparing is easier than writing

Here is the observation that made preference learning attractive. Even when you can’t write the perfect response, it can be easier to have the model generate two responses and have a human say which is better. Judging “A is better than B” can be cheaper and faster than authoring a gold-standard answer, though it still has rater bias and reliability limits.

This asymmetry is one reason the field expanded into preferences. Instead of asking humans to produce ideal outputs, we can ask them to compare model outputs. We collect preference datapreference dataData where humans (or an AI) compare two or more model responses to the same prompt and mark which is better. The training signal for reward models and DPO.See in glossary → (judgments of the form “response A is preferred to response B for this prompt”) via pairwise comparisonpairwise comparisonAsking a labeler which of two responses is better, rather than scoring each on an absolute scale. Easier and more reliable for humans, and the basis of preference learning.See in glossary →. It can scale differently from demonstration-writing and can capture comparative judgments that are difficult to express in a single ideal answer.

Turning a pile of “A > B” judgments into something a model can optimize against is the job of a reward modelreward model (RM)A model trained from human preference data to output a scalar score for how good a response is. Stands in for a human judge so RL can query reward millions of times.See in glossary →, which we build in Chapter 11; the preference-learning idea itself gets its own chapter in Chapter 8. The whole RLHF apparatus exists to exploit this one asymmetry.

Wall 3: some properties aren’t in the imitation data at all

A demonstration shows the model what to do. It can also include refusals and calibrated uncertainty, but collecting enough diverse examples of those behaviors is difficult, especially for adversarial or novel situations.

Harmlessness is a clear case. A dataset focused only on helpful answers may contain too few examples of appropriate refusals and boundary cases. Demonstrations can teach those behaviors, but preference feedback, red teamingred teamingDeliberately probing a model for failures — adversarially crafting prompts that try to elicit harmful, unsafe, or otherwise unwanted behavior — to surface weaknesses that ordinary testing misses and generate data for fixing them.See in glossary →, and targeted evaluations can provide broader signals about where to draw the line.

Calibration is subtler. We want the model to be confident when it should be and uncertain when it should not. Demonstrations can model uncertainty language, but they do not automatically guarantee well-calibrated probabilities or reliable behavior under distribution shift.

Failure modes that feedback objectives can create

It’s not just that imitation under-delivers: done naively, it can actively bake in problems.

The sharpest example is sycophancysycophancyA failure mode where a model tells the user what it thinks they want to hear rather than what is true or correct — often a side effect of preference optimization.See in glossary →: the tendency to tell users what they want to hear rather than what’s true. It can arise when agreeable, flattering responses receive favorable feedback, so the model learns that agreement is rewarded even when the user is wrong. This is a risk of optimizing an imperfect approval signal: the model may be following the signal while missing the intended behavior.

That gap (between the signal we optimize and the behavior we actually want) has a name we’ll spend a whole chapter on: reward hackingreward hackingWhen a policy finds ways to score high on the reward model without actually being better — exploiting quirks of an imperfect proxy. A central danger of RL post-training.See in glossary →. When you replace “imitate demonstrations” with “maximize a reward,” you hand the model an objective it will pursue literally, and it will find the cracks: exploiting quirks of the reward model, gaming length or format, producing answers that score high without being good. We dig into this (and Goodhart’s lawGoodhart’s law"When a measure becomes a target, it ceases to be a good measure." Optimizing a proxy reward (the measure) eventually diverges from the true objective it stood in for.See in glossary →, its theoretical underpinning) in Chapter 19. Flag it now as the central catch of the approach we’re about to adopt: optimizing a proxy for “good” is powerful precisely because it’s relentless, and dangerous for exactly the same reason.

From imitating to optimizing

Here’s the throughline. Pure imitation (SFT) is powerful, but its behavior is shaped by the demonstrations it sees and their coverage. Preference signals and verifiers offer additional ways to express comparative or automatically checkable goals. A common next move is to optimize a signal of what’s better, while retaining demonstrations and other safeguards.

That signal can come from human preferences (RLHF), from AI preferences (RLAIFRLAIFReinforcement Learning from AI Feedback — replace human preference labels with labels from another model (or the model itself), making the feedback loop cheap and scalable.See in glossary → and Constitutional AIConstitutional AIAnthropic’s method where a model critiques and revises its own outputs against a written set of principles (a "constitution"), then trains on AI-generated preferences — a form of RLAIF.See in glossary →), or from automatic verifiers (RLVR). The algorithms differ, but the philosophy is shared: define a notion of “better,” then push the model up that gradient, while keeping it leashed to a sensible reference and its distribution from collapsing, exactly the tools from the previous chapter.

Many pipelines begin by teaching the model the format of a conversation and the basic shape of a helpful answer through instruction tuning. Others use different ordering or rely more heavily on synthetic data and RL, but instruction tuning remains a common supervised starting point.