DEEP LEARNINGPresentation & practice sheets
0 of 5 sections complete0%
← All presentation & practice sheets
WEIGHT INITIALIZATION DYNAMICS

Initialize
with purpose.●

Before a network learns anything,
its weights have already set the stage.

4 visual explorations + a diagnosis11 checksExplore at your own pace
01 / UNDERSTANDRead the theory. Make a prediction.
02 / EXPERIMENTChange a parameter. Follow the signal.
03 / EXPLAINCheck your reasoning. Keep going.
BEFORE YOU START

What are we trying to preserve?

Initialization chooses the weights before the first training update. We want different neurons to learn different features, while keeping both forward signals and backward gradients at a useful scale.

One layer

z = Wa + b, then anext = φ(z). The weighted sum z is the preactivation; φ applies the nonlinearity.

Width and weight scale

nin counts inputs per unit; nout counts outputs. σ is weight standard deviation, and σ² is weight variance.

The signal we measure

q = E[a²] is the second moment, or average squared activation. Var(a) = q − E[a]². They coincide only for zero-mean activations.

Model assumptions. The plots start at q₀ = 1 and use independent zero-mean random weights, zero biases, and equal hidden widths. Gaussian preactivations approximate a wide network. Symmetry uses a separate five-unit calculation. These are forward-pass models, not training runs.

01
ZERO & CONSTANT INITIALIZATION

Different neurons.
The same story.

Complete ✓
IN ONE MINUTE

Identical hidden units cannot learn different features while their incoming weights, biases, and outgoing connections remain symmetric. Randomness gives each neuron a different starting point.

∂L / ∂Wij = δi hj Equal δ across units → equal row updates

Read the gradient. L is the loss; Wij connects incoming feature j to unit i. δi = ∂L/∂zi measures how sensitive the loss is to that unit’s preactivation. hj is the incoming activation. Their product tells us how a small change in this weight changes the loss. Equal δ values give matching gradients across rows for the same input j; different input columns can still have different gradients.

THEORY · BEFORE THE VISUALIZATION

Symmetry: why identical units stay alike

  1. Start with the computation

    A row of W contains all the incoming weights for one hidden unit. Equal rows and equal biases give equal z values for the same input, so those units produce the same activation.

  2. Follow the learning signal

    The gradient is ∂L/∂Wᵢⱼ = δᵢaⱼ. If outgoing connections are also symmetric, the units receive equal δ values. Gradient descent applies the same row update and cannot separate the units.

  3. Break symmetry without losing scale

    Independent random weights give units different starting computations. Biases may still start at zero. Randomness solves symmetry; choosing its scale is a separate problem.

WORKED EXAMPLE

For input [1, 2] and each weight row [c, c], every unit receives z = 3c. Zero initialization gives tanh(0) = 0; c = 2 gives tanh(6) ≈ 0.999988. Nonzero weights still give identical outputs.

Predict before you play: If every zero weight is replaced by 2, will the neurons become different?

No. All rows still match. The output changes, but feature diversity does not. With tanh, the larger constant also drives saturation.

WHAT TO LOOK FOR

First compare Zero and Constant with perturbation off. Then turn perturbation on. Look for equality across the five numerical outputs; the overlapping paths are a metaphor for repeated features.

3D METAPHOR / SYMMETRY LOCK

Five neurons, one path.

INTERACTIVE
Drag to orbit · arrow keys to rotate

An analogy for identical features, not a simulated loss surface.

YOUR EXPERIMENT

Start them alike.

Inputs [1, 2], five units per layer, tanh activation, zero bias. Values below the scene show the final layer.

TRY THIS

Switch from zero to constant. Does a nonzero value create different features?

At the board Why identical rows stay identical

For h = φ(Wx + b), identical rows of W and equal biases give identical preactivations. If the downstream connections are also symmetric, backpropagation gives the same δi for every hidden unit.

∂L / ∂Wij = δixj   →   Wi: = Wk: after every update

The update is identical across rows, not necessarily across columns: different xj can produce different column gradients. Equal incoming weights alone do not guarantee equal gradients if outgoing weights differ. Zero biases are fine when random weights have already broken symmetry.

Worked example. x = [1, 2] and δ = [0.5, 0.5, 0.5] produce three identical gradient rows [0.5, 1.0]. This specified δ need not arise from an entirely zero-initialized deep network.

02
NAIVE RANDOM INITIALIZATION

Random is a start.
Scale is the difference.

Complete ✓
IN ONE MINUTE

Random weights break symmetry. Their scale decides whether a signal grows or fades. In a linear network, even a small mismatch compounds with depth. With tanh, large inputs cause saturation; the activations themselves stay bounded.

qℓ = nin σ² qℓ−1Linear activation · q = E[a²] · independent, zero-mean weights

Read the recurrence. ℓ labels the current layer, and qℓ−1 is the incoming average squared activation. Each of the nin inputs contributes σ²qℓ−1; adding them gives the outgoing q for a linear layer. The factor ninσ² is the per-layer multiplier: above 1 means growth, below 1 means decay. This recurrence assumes independent zero-mean weights and does not pass unchanged through tanh.

THEORY · BEFORE THE VISUALIZATION

Random weights: a small scale error compounds

  1. Add independent contributions

    Each unit sums nᵢₙ weighted inputs. With independent zero-mean weights, the expected preactivation second moment is E[z²] = nᵢₙσ²qᵢₙ. Since E[z] = 0, this is also Var(z).

  2. Apply the activation

    For linear activation, qₒᵤₜ = nᵢₙσ²qᵢₙ. For tanh, calculate E[tanh²(z)] instead: outputs stay bounded, and saturated units have small local derivatives.

  3. Repeat through depth

    In an equal-width linear network, qᴸ = (nσ²)ᴸq₀. A multiplier above 1 grows the signal; below 1 it fades. A well-scaled forward signal alone does not prove gradients are healthy.

WORKED EXAMPLE

At width 64, σ = 1 gives a per-layer multiplier of 64. Choosing σ = 0.125 gives 64 × 0.125² = 1. Choosing σ = 0.0625 gives 0.25, leaving q₃ = 0.25³ = 0.015625 after three linear layers.

Predict before you play: If we halve the weight standard deviation, does the linear layer multiplier halve?

It becomes one quarter: the multiplier depends on σ². Across three layers, the output second moment becomes 1/64 of its previous value.

WHAT TO LOOK FOR

Use the two scale presets with Linear selected. Inspect one layer at a time, then switch to tanh. On the log chart, equal vertical steps represent equal multiplicative changes, not equal additive changes.

3D METAPHOR / UNBALANCED STARTS

Watch the scale take over.

Drag to orbit · arrow keys to rotate

Terrain motion is illustrative. The chart below uses the stated propagation model.

YOUR EXPERIMENT

Tune the scale.

0.01 ← logarithmic scale → 2.00

TRY THIS

At width 64, compare σ = 1 with σ ≈ 0.125. Then switch to tanh.

At the board Why fan-in multiplies the variance

Write z = Σwjaj. Under independent zero-mean weights, independent input coordinates, and weights independent of inputs, cross terms vanish. Then E[z²] = nin Var(w) E[a²]. For centered inputs, E[a²] = Var(a).

n = 64, σ = 1, Var(x) = 1   ⇒   Var(z) = 64

For repeated equal-width linear layers, qL = (nσ²)L. For tanh, this linear recurrence no longer holds after the activation: q is bounded by 1, while tanh′(z) = 1 − tanh²(z) shrinks in saturated regions. Large weights and many layers make gradient behavior more subtle than activation variance alone.

03
XAVIER / GLOROT INITIALIZATION

Give the signal
room to travel.

Complete ✓
IN ONE MINUTE

Xavier balances the competing forward and backward scaling requirements. With equal widths and linear activations, it preserves the expected signal energy. For tanh, it is a useful starting point, not a promise of constant variance at every depth.

Var(w) = 2 / (nin + nout)Normal: σ = √Var(w) · Uniform: bounds ±√(6 / (nin + nout))

Read the scale. nin is the number of inputs per unit and nout is the number of output units. The formula specifies variance; take its square root to get the standard deviation used for normal sampling. Wider layers need smaller weights because they add more contributions. For uniform sampling on [−b, b], variance is b²/3, which explains the 6 inside the square root. These formulas use gain 1.

THEORY · BEFORE THE VISUALIZATION

Xavier: balance the forward and backward directions

  1. Set a forward target

    With activation derivatives near one, keeping the forward second moment steady asks for nᵢₙ Var(w) ≈ 1.

  2. Set a backward target

    Under similar independence assumptions, preserving gradient variance asks for nₒᵤₜ Var(w) ≈ 1. When widths differ, one variance cannot satisfy both targets exactly.

  3. Use a compromise

    Xavier chooses Var(w) = 2/(nᵢₙ + nₒᵤₜ). With equal widths and linear activation this gives a forward multiplier of 1. Tanh compresses the distribution, so gain-1 Xavier is not exact preservation for deep tanh networks.

WORKED EXAMPLE

At nᵢₙ = nₒᵤₜ = 128, Var(w) = 1/128 and σ ≈ 0.08839. At widths 256 → 512, Var(w) = 1/384 and σ ≈ 0.05103. That unequal-width layer has forward multiplier 2/3 and backward multiplier 4/3 under the linear approximation.

Predict before you play: If we double the equal hidden width, should the expected linear signal become larger?

No. Xavier halves the weight variance as the width doubles. The product n Var(w) stays 1, so the expected forward second moment stays fixed.

WHAT TO LOOK FOR

Compare widths with Linear selected, then use tanh. Each ridge has its own peak normalized to the same height: compare its horizontal spread. Use the chart and q readout to quantify that spread.

3D DISTRIBUTIONS / FORWARD SIGNAL

A view through the layers.

Drag to orbit · arrow keys to rotate

Each ridge is a sampled activation histogram, shown on a shared horizontal scale.

YOUR EXPERIMENT

Balance the widths.

A SINGLE LAYER, DIFFERENT WIDTHS

The network above stays at the hidden width. This calculator describes one separate layer.
At the board A compromise between two directions

Forward preservation asks for nin Var(w) ≈ 1. Backward preservation asks for nout Var(w) ≈ 1, assuming derivatives near one and independent gradients. These conditions coincide when widths match. Xavier uses 2/(nin + nout) as a compromise.

nin = 256, nout = 512   ⇒   σ = √(2/768) ≈ 0.0510

A uniform variable on [−b, b] has variance b²/3. Setting b = √(6/(nin + nout)) gives the same target variance. The explorer uses gain 1. Tanh-specific gain conventions change the scale, and sigmoid has a nonzero mean that can complicate deep signal propagation.

04
HE / KAIMING INITIALIZATION

Half the gate.
Twice the variance.

Complete ✓
IN ONE MINUTE

For symmetric preactivations, ReLU removes half the second moment, E[z²]. He initialization compensates with twice the weight variance. Signal energy can then remain steady through an equal-width ReLU network.

Var(w) = 2 / ninLeaky ReLU with slope α: Var(w) = 2 / ((1 + α²)nin)

Read the factor of two. nin counts the inputs summed by each unit. ReLU discards the negative half of a symmetric preactivation distribution, halving its second moment. Doubling the weight variance compensates: nin × (2/nin) × ½ = 1. In leaky ReLU, α is the negative slope; α² appears because we track squared signals. At α = 0 the leaky formula reduces to ordinary He initialization.

THEORY · BEFORE THE VISUALIZATION

He: compensate for the rectifier

  1. Track the squared signal

    For a symmetric preactivation distribution, positive and negative values contribute equally to E[z²]. ReLU removes the negative half, so E[ReLU(z)²] = ½E[z²].

  2. Compensate before the gate

    Choose Var(w) = 2/nᵢₙ. Weight mixing gives E[z²] = 2qᵢₙ, and ReLU gives qₒᵤₜ = ½ × 2qᵢₙ = qᵢₙ.

  3. Distinguish energy from variance

    ReLU outputs have a positive mean, so their centered variance is less than their second moment. For leaky ReLU with slope α, the retained squared-signal fraction is (1 + α²)/2, giving Var(w) = 2/((1 + α²)nᵢₙ).

WORKED EXAMPLE

At width 128 with qᵢₙ = 1, He gives preactivation variance 2 and ReLU output q = 1. For Gaussian preactivations the output mean is 1/√π, so Var(a) = 1 − 1/π ≈ 0.68169. Xavier instead gives q = 0.5 after one ReLU layer.

Predict before you play: With gain-1 Xavier and ReLU, how much second moment remains after four equal-width layers?

(½)⁴ = 1/16 = 0.0625 of the input value. He compensates for this factor in the stated expectation model.

WHAT TO LOOK FOR

Step through both initializers at the same depth. Read the “before activation” and “after activation” values, then compare their curves. Particle size is illustrative; the numbers report the model.

3D METAPHOR / BALANCED STARTS

Keep the signal in play.

COMPARE
Drag to orbit · arrow keys to rotate

The bowl illustrates balance. Initialization does not guarantee convergence or a global minimum.

YOUR EXPERIMENT

Match the activation.

LOOK CLOSELY

The curves show q = E[a²]. ReLU outputs have a positive mean, so their variance is smaller than q.

At the board Second moment is not variance

For a centered symmetric distribution, E[ReLU(z)²] = ½E[z²]. But ReLU(z) has a positive mean. If z ∼ N(0, v), then:

E[a] = √(v / 2π)
E[a²] = v / 2
Var(a) = v(½ − 1/2π)

With qin = 1 and He weights, v = 2. The output second moment is 1 while its variance is approximately 0.682. It is incorrect to say ReLU simply halves the centered variance.

Leaky ReLU gives E[a²] = (1 + α²)E[z²]/2, yielding the adjusted He variance. This derivation is specific to rectifiers; GELU and SiLU do not exactly satisfy the same rule.

Worked example. With nin = nout = 100, Xavier variance is 0.01 and He variance is 0.02. The ratio He/Xavier is 2.

05
DIAGNOSTIC PRACTICE

Read the symptoms.
Choose the beginning.

Complete ✓
CASE FILE / 50-LAYER CNN

The network barely learns.

You use ReLU in every hidden layer and initialize with Xavier normal, gain 1. Accuracy stalls; gradients in the first ten layers are around 10−15.

ACTIVATIONReLU
INITIALIZERXavier
EARLY GRADIENTS≈ 10⁻¹⁵

In the simplified equal-width model, the signal second moment at layer 50 is 2−50 ≈ 8.88 × 10−16. Diagnose the likely initialization mismatch, then choose a fix.

Your initializer field guide
InitializerWeight varianceBest starting contextWatch for
Zero / constant0Biases; not all hidden weightsSymmetric units
Naive normalσ²Controlled scale experimentsWidth-dependent growth or decay
Xavier2 / (nin + nout)Linear; tanh with appropriate gainReLU signal decay at gain 1
He2 / ninReLU; adjust for leaky ReLULinear signal growth
Transfer it to a real model What to measure next

Start by checking activation means, second moments, saturation rates, and gradient norms layer by layer. For convolutional layers, fan-in includes input channels × kernel height × kernel width. Residual connections, normalization, layer shapes, and activation-specific gains can change the picture.

Small gradients are evidence, not a unique diagnosis. The case above deliberately isolates an initialization mismatch; learning rate, architecture and data can also cause stalled training.

# PyTorch · W shaped [fan_out, fan_in]
torch.nn.init.xavier_normal_(linear.weight, gain=1.0)
torch.nn.init.kaiming_normal_(relu_layer.weight,
                             mode="fan_in", nonlinearity="relu")
torch.nn.init.zeros_(relu_layer.bias)

Start the checks again?

This clears the answers saved for this worksheet in this browser.