Synthetic & self-generated data

Self-Instruct, Alpaca, and distillation

Papers: Self-Instruct: Aligning Language Models with Self-Generated Instructions (Wang et al., 2022) · Stanford Alpaca (Taori et al., 2023)

The last chapter ended on a quiet problem: SFT often uses large numbers of high-quality (prompt, response) demonstrations, and writing them by hand is slow and expensive. InstructGPT used a specialized labeler workforce; almost nobody else had comparable resources. A technique that helped unlock the 2023 wave of open instruction-tuned models was to generate at least part of the demonstration data with a language model.

Self-Instruct: bootstrapping data from the model itself

Self-Instruct (Wang et al., 2022) asked a provocative question: can a model generate its own instruction-tuning data? The pipeline is a bootstrap loop:

1. Seed. Start with a small pool of human-written seed tasks: the original paper used 175.
2. Generate. Prompt the model to write new instructions in the style of the seeds, then to generate input–output pairs for each new instruction.
3. Filter. Throw out instructions too similar to existing ones (to keep diversity), and drop malformed or low-quality generations.
4. Fine-tune. Use the surviving examples as instruction-tuning data. Iterating generation and fine-tuning is a common extension, but not a required part of the original recipe.

This process can manufacture tens of thousands of self-generatedSelf-InstructA method that bootstraps instruction-tuning data from a model itself: seed it with a few tasks, have it generate many more, filter, and fine-tune. Made instruction data cheap and synthetic.See in glossary → examples from a few hundred human seeds. Filtering for novelty and quality is important because unchecked generation can produce near-duplicates, malformed tasks, and systematic errors.

Alpaca: distillation from a stronger teacher, for $600

Stanford Alpaca (Taori et al., 2023) took Self-Instruct and made one pragmatic change that turned a clever idea into a movement: instead of having the target model generate its own data, it used a stronger model as the teacher. The recipe was almost comically cheap:

• Take a capable base model (LLaMA-7B, the original 2023 LLaMA, later retroactively called LLaMA 1).
• Use Self-Instruct’s bootstrap, but have GPT-3.5 (text-davinci-003) generate the 52,000 instruction-following examples.
• Fine-tune LLaMA-7B on those 52K examples.

Total cost: under $600 (roughly $500 of API calls and a few hours of GPU time). The resulting model behaved qualitatively like a much larger, much more expensive assistant. Alpaca’s release (weights, data, and training code) set off a Cambrian explosion of open instruction-tuned models built on the same template.

The benefits, and the catch

The upside is obvious and was the whole point: synthetic instruction data is cheap, fast, and scalable. You can generate a million examples overnight, target any domain you like, and never schedule a human labeler. For getting an open model to behave like an assistant, it works remarkably well.

But there are three structural problems, and they matter more the more you rely on this data.

• You inherit the teacher’s flaws. Every bias, factual error, and stylistic tic in the teacher’s outputs gets baked into the student. Distillation copies the bad with the good.
• You are constrained by the teacher signal. Pure output imitation does not directly provide a signal for correcting the teacher’s mistakes. A student can still generalize differently or outperform a teacher on some evaluations, but that is not guaranteed by distillation alone.
• Legal and terms-of-service issues. API terms, licenses, and applicable law vary and change. Training on commercial-model outputs requires reviewing the relevant terms and permissions; open-data projects often prefer permissively licensed or human-sourced data.

Going beyond imitation in two ways

That last point is the thread that ties this whole section to the rest of the explainer. Imitation has a ceiling, and the rest of the explainer is largely about the two ways past it.

The first is to stop imitating and start optimizing a preference or reward signal — which answer is better? — rather than copy this answer. That is RLHFRLHFReinforcement Learning from Human Feedback — train a reward model on human preference comparisons, then optimize the policy against that reward with RL (typically PPO), with a KL leash to a reference.See in glossary → and the preference era, where the next several chapters go.

The second is more surprising, and it’s a forward reference worth planting now: self-generated data returns in the reasoning era. There, instead of distilling from a stronger external teacher, a model generates many candidate solutions, keeps ones that pass a verifier, and fine-tunes on them: rejection samplingrejection samplingGenerate several candidate responses, keep only the best-scoring one(s) by some reward or verifier, and fine-tune on those. A simple, stable, RL-free way to improve a model.See in glossary → and STaRSTaRSelf-Taught Reasoner (Zelikman, 2022) — generate chain-of-thought rationales, keep those that reach the correct answer, fine-tune on them, and repeat. Bootstraps reasoning from a model’s own correct attempts.See in glossary →-style bootstrapping (Chapter 23). A reliable correctness filter supplies a selection signal beyond raw imitation, but it does not guarantee that retained rationales are sound or that every training run improves. Self-generated data becomes much more useful when the selection criterion actually captures the target behavior.