Where training starts

Where the numbers start

The training loop needs a starting point. Before the first step, every weightweightOne adjustable number that scales an input inside a model — how strongly that input pushes the prediction up or down. The weights (plus biases) are the parameters training adjusts.See in glossary → and biasbiasThe constant term added to a weighted sum — the model's baseline output when the inputs contribute nothing. It lets a line shift up and down rather than being pinned to the origin. (Unrelated to "bias" in the fairness sense.)See in glossary → in the network has to hold some number. It’s tempting to shrug and set every weight to zero: clean, neutral, no bias in any direction. That instinct is not just suboptimal; it breaks training outright, and seeing why reveals a constraint that shapes every real initializationinitializationThe scheme for setting parameters before training starts. Good initialization keeps activations and gradients at sane scales through a deep network so training can get going.See in glossary → scheme. Biases are often allowed to start at zero; the dangerous part is making units’ weights identical.

Why not zero? The symmetry problem

Consider a hidden layerhidden layerA layer of a neural network between the input and the output. "Hidden" because its values are internal scratch space you don't directly observe. Stacking hidden layers (with nonlinearities) is what gives networks their power.See in glossary → where every unit starts with identical weights. Each unit sees the same inputs, so with identical weights each computes the exact same output. During backpropagationbackpropagationThe algorithm that computes the loss gradient for every parameter efficiently by applying the chain rule backward through the network, reusing intermediate results from the forward pass.See in glossary → they then receive the exact same gradient, so they update identically and stay identical forever. A layer of a thousand units behaves like a single unit wearing a thousand masks: all that capacity, permanently wasted.

This is the symmetry-breaking problem. To let units specialize, they must start different. The cure is to initialize weights from small random values, so each unit begins life computing something slightly its own and the gradients pull them apart. Randomness here isn’t a hack; it’s the thing that lets a wide layer be wide.

The scale is what really matters

So the weights usually start random, but their scale turns out to matter enormously. A signal passing through a deep stack is repeatedly changed by each layer’s weights and activation function. Those same pieces affect gradients on the way back, so poorly chosen scales can make either direction unstable:

• Too small and the values flowing forward may shrink toward zero; gradients can shrink too, so early layers learn slowly.
• Too large and the values or gradients may explode into huge numbers, overflow, and produce NaNs. Some activations instead flatten at extreme inputs and kill their gradients.
• Well scaled and signals and gradients begin at useful sizes, though training can still push them out of balance later.

Slide the initialization scale and watch a signal travel through a ten-layer chain:

The useful scale isn’t the same for every network: it depends on how many inputs feed into each layer and on the activation. Under a simple approximation where those inputs vary independently around zero, adding $n$ of them makes the typical result grow like $\sqrt{n}$. Starting each weight at a size around $1/\sqrt{n}$ counteracts that growth.

Notice what the scale section was really about: signals growing or shrinking as they pass through many layers. With good initialization the network starts balanced, but push it deep enough and keeping it balanced becomes the single hardest part of training at all. That problem, and the handful of ideas that solved it and unlocked truly deep networks, is what the next section confronts head-on.