Section 06

The SFT stage in practice

Demonstrations, chat templates, and data quality

Paper: Training language models to follow instructions with human feedback (InstructGPT) — Ouyang et al., 2022

The previous chapter established why instruction tuning works. This one is about the unglamorous machinery of actually doing it in a production pipeline, the part labeled “Step 1: SFT” in every modern post-training diagram. There is very little new math here. The interesting questions turn out to be about data and plumbing: what format you feed the model, which tokens you compute loss on, and how few examples you can get away with.

SFT is just next-token training, on curated data

When InstructGPT (Ouyang et al., 2022) laid out the influential three-step recipe (SFT → reward model → RL optimization), the first step was deliberately ordinary. Supervised fine-tuning takes the pre-trained base model and continues training it with the same basic autoregressive next-token prediction machinery, maximizing the likelihood of target text.

The only thing that changes is the diet. Instead of raw web text, the model sees curated (prompt, response) demonstrations: high-quality examples of a prompt followed by the ideal answer, often written or edited by human labelers. For a demonstration with prompt $x$ and response $y = (y_1, \ldots, y_T)$ , the loss is the familiar negative log-likelihood:

\mathcal{L}_{\text{SFT}} = -\sum_{t=1}^{T} \log p_\theta\big(y_t \mid x,\, y_{<t}\big)

That is it. No reward, no sampling, no RL. SFT is teaching by demonstration: here is a good answer; make answers like this more probable.

Look closely at what that sum runs over, though: only the response tokens $y_1, \ldots, y_T$ , never the prompt. In practice this is enforced by loss masking. The full sequence (system + user + assistant turns) is fed through the model so it can condition on the whole context, but the prompt positions are excluded from the loss. The model is graded only on the tokens it is supposed to produce (the assistant’s reply), not on the tokens it was given. We want it to learn to answer, not to predict the user’s question; computing loss on the prompt would spend capacity modeling user inputs and reward the model for memorizing prompts instead of responding well. Masking points the entire training signal at the behavior we actually care about.

What the loss is computed on

role:    user ───────────────────────────────────────    ─ assistant ─
token:   What   is   the   capital   of   France   ? │   Paris   .
loss?:    ✗     ✗    ✗      ✗        ✗     ✗      ✗  │    ✓      ✓
         └────────────── masked (context only) ──────┘  └─ trained ──┘

Every token is read; only the assistant-response tokens (✓) contribute gradients. The prompt is effectively “read-only”: it shapes the prediction without being a prediction target.

The chat template: turning a document model into a conversation

A base model only knows how to continue a document. To make it act like a turn-taking assistant, we need a fixed convention for marking who is speaking: a chat template. The template wraps each message in special tokens: reserved symbols, added to the vocabulary, that act as role and turn markers the model learns to recognize and emit.

A typical exchange, serialized, looks like this:

A tiny chat template

<|system|>
You are a helpful, concise assistant.
<|user|>
What is the capital of France?
<|assistant|>
Paris.<|end|>

The role tags (<|system|>, <|user|>, <|assistant|>) and the turn terminator (<|end|>) are illustrative markers; a real template may encode each marker as one or several tokens. At inference time the runtime appends the assistant-turn marker and lets the model generate until a configured stop condition. The model learns the convention from correctly formatted examples during SFT. Real templates differ substantially by model, though many encode role boundaries and turn endings explicitly.

That opening <|system|> block is the system prompt: an instruction, often hidden from the end user, that sets persona, rules, and constraints for the conversation. Training and template conventions can make models more likely to follow it over user instructions, but this is a learned behavior rather than a hard security boundary; product-level controls may also be needed.

Loss masking: only learn the assistant’s words

One important practical detail is loss masking. The serialized example above contains the system prompt, the user’s question, and the assistant’s reply. Computing language-modeling loss over all tokens trains the model to model the entire transcript; many chat-SFT pipelines instead focus the loss on text the assistant should produce.

The fix, as we explained briefly above, is loss masking (sometimes “prompt masking”): you compute the next-token loss only on the assistant’s response tokens, and mask out (zero the loss on) the system and user tokens. The prompt tokens are still fed in as context (the model conditions on them), but they contribute nothing to the gradient. Concretely, the masked objective trains only on the spans the assistant is responsible for producing:

\mathcal{L}_{\text{SFT}} = -\sum_{t \in \text{assistant}} \log p_\theta\big(y_t \mid \text{context}_{<t}\big)

Assistant-only masking is common rather than universal: the best choice depends on the data format, objectives, and model. It concentrates the supervised signal on assistant behavior while retaining the prompt as conditioning context. This masking is also one concrete line between SFT and mid-training: mid-training is still full-sequence language modeling (loss on every token), whereas SFT typically masks the loss down to just the assistant’s reply.

Data quality over quantity

The instinct from pre-training (more tokens is better) does not carry over cleanly to SFT. The landmark demonstration was LIMA (“Less Is More for Alignment,” Zhou et al., 2023): the authors fine-tuned a strong base model on just 1,000 carefully curated prompt–response pairs and got a model competitive with ones tuned on hundreds of thousands of examples.

LIMA supports the hypothesis that a strong base model can benefit greatly from a small, carefully curated alignment set. It does not establish a universal data law: SFT can also teach new facts, skills, tools, and formats, and the useful data scale depends on the base model and target behavior. In practice, high-quality examples often outperform a comparable amount of noisy data, but larger diverse datasets can also help.

Why SFT alone plateaus

SFT directly pushes the model toward imitating its demonstrations. That creates important limitations, though generalization and pre-training mean they are not absolute ceilings. Three consequences follow.

1. Coverage constrains the signal. If demonstrations omit difficult behaviors or disagree about them, SFT has no direct preference signal to resolve that ambiguity.

2. It has no explicit comparison signal. Positive demonstrations can include refusals and counterexamples, but pairwise data can more directly state that one response is better than another.

3. It can teach an overconfident style. Demonstrations that state facts crisply can encourage the style of confidence even on questions where uncertainty is appropriate, without guaranteeing that outcome.

This plateau is exactly the gap that learning from human preferences (comparisons, rewards, and RL) was invented to close.

With the supervised stage understood, the next question is where all those high-quality demonstrations come from when you don’t have an army of labelers. That is the story of synthetic and self-generated data, the next chapter.