✦ Short idea, then a plot, then questions

Spend capacity
more slowly.

USEEach topic is a one-minute idea, a live visual, then theory and numerical MCQs in order. Textbook screenshots stay folded until you want them. Later sections stay locked. Nothing is scored.

01Concept in a minute

Read the teal box and the formula. That is the whole claim.

02Change the plot

Turn λ, p, γ, or the checkpoint. The numbers are the evidence.

03Answer in order

Theory first, then numbers. A miss is a hint. Open the book fold only if you need it.

motivation

Why add anything to the loss?

✓ complete

🔒

Section locked

Complete the section immediately above.

In one minuteTraining loss is what we differentiate. Test loss is what we care about. Regularisation is any extra preference that trades a little training fit for better test performance — without shrinking the architecture by hand.

the regularised objectiveẼ(θ) = E(θ) + λ Ω(θ)

E is the data term. Ω is a preference. λ is a hyperparameter: do not fit it by gradient descent on the training set. Tune it on validation. λ = 0 is ordinary training. Too large a λ (or too small a dropout p, or too early a stop) underfits.

L2

Shrink every weight toward zero.

ridge / weight decay

L1

Delete some weights.

lasso

Dropout

Delete activations for one step.

train mask

Batch norm

Re-scale a minibatch, then γ û + β.

running stats at test

Early stop

Delete training time.

θ at t*

Who picks λ? Training updates θ. Validation chooses λ, p, patience. Test is opened once.

Training set

Updates the weights. Do not use it to pick λ.

updates θ

Validation set

Chooses λ, dropout p, patience.

chooses λ

Test set

Opened once after the knobs are frozen.

final report

Optional · textbook figuresBishop PRML fig. 1.7 and the explicit / implicit split. Open if you want the long version.

Goodfellow ch. 7: strategies that reduce test error, possibly at the expense of training error. Prince UDL ch. 9: explicit penalties on weights versus implicit ones (optimiser, stopping time, SGD noise).

Polynomial fits as lambda varies — Bishop, PRML (2006), fig. 1.7 — polynomial fit as ln λ varies. Large λ underfits; small λ overfits.

theory You will compare λ = 0, λ = 0.1, and λ = 10 on the same wide network. Which split should choose λ?

your turn · numbers

Add the penalty

Ẽ(θ) = E(θ) + λ Ω(θ). Pick the expression first, then evaluate it.

you needE = 0.40λ = 0.10Ω = 2.0

Ẽ(θ) = ?

you need0.40 + (0.10)(2.0)

Ẽ(θ) = 0.40 + (0.10)(2.0) = ?

L2 · ridge · weight decay

Shrink every weight

✓ complete

🔒

Section locked

Complete the section immediately above.

In one minuteL2 adds λ‖w‖₂² to the loss. Every coordinate is pulled toward zero; none typically hits exactly zero. The closed form is ridge. The SGD implementation is weight decay: w ← (1 − ηλ) w − η ∇E.

ridgeŵ = (XᵀX + λ I)⁻¹ Xᵀ y

Adding λ to every eigenvalue of XᵀX shrinks unconstrained directions hardest. The constraint ‖w‖₂² ≤ η is a disk, so loss ellipses usually touch off-axis: both weights survive.

Optional · textbook figures and the board calculationGoodfellow fig. 7.1, Prince fig. 9.2, and the two-point worked example.

Bishop PRML §3.1.4: Ẽ(w) = (1/2) Σ_n (y_n − wᵀφ_n)² + (λ/2)‖w‖₂². Tiny eigenvalues of XᵀX are the ones shrinkage hits. λ → 0⁺ among interpolators selects the minimum Euclidean-norm solution. For Adam, fold λ into the loss and the adaptive denominator rescales the L2 gradient too — AdamW applies (1 − ηλ) after the adaptive step.

board calculation · ŷ = w x, two points (1,2) and (2,3)

SSE(w) = (2 − w)² + (3 − 2w)² = 5w² − 16w + 13
λ = 0: 10w − 16 = 0 ⇒ w = 1.6000, SSE = 0.2000
λ = 1: Ẽ = SSE + λ w² ⇒ w = 8/(5+λ) = 1.3333
Training SSE rose to 0.5556. |w| fell. A little bias, less variance.

L2 as balance of ellipses and circular penalty — Goodfellow, Bengio & Courville (2016), fig. 7.1 — L2 as a balance between data-loss ellipses and circular penalty contours.

L2 smoothing a small network — Prince, UDL, fig. 9.2 — as the L2 penalty grows, the fitted function loses kinks.

live plot

Turn the λ dial

Dashed = true sine. Dots = six noisy training points. Solid = degree-9 ridge fit.

penalty

How hard to pull toward 0

train MSE—

validation MSE—

Training interpolates. Validation does not.

theory As λ grows from 0 to 10 on the live plot, the fitted curve

your turn · numbers

Two-point ridge

ŷ = w x. Points (1, 2) and (2, 3). Closed form ŵ = 8 / (5 + λ). Pick the plug-in, then evaluate.

you needŵ = 8/(5+λ)λ = 0

ŵ = ?

you needŵ = 8/5

ŵ = 8/5 = ?

you needŵ = 8/(5+λ)λ = 1

ŵ = 8/(5+1) = ?

theory On the live plot, at λ = 10 both errors are high. The diagnosis is

L1 · lasso

Delete some of them

✓ complete

🔒

Section locked

Complete the section immediately above.

In one minuteReplace ‖w‖₂² by ‖w‖₁. The constraint is a diamond with vertices on the axes. Expanding loss ellipses typically hit a vertex first, so one coordinate is exactly zero. Ridge will not do this at any finite λ.

soft-thresholdingS_λ(z) = sign(z) max(|z| − λ, 0)

|z| > λ: move z toward 0 by λ. |z| ≤ λ: output 0. That is the exact minimiser of (1/2)(w − z)² + λ|w|.

Optional · textbook figures and the two-feature examplePRML fig. 3.4, Bishop & Bishop fig. 9.6, and lasso versus ridge on a decoy feature.

Ω_q = Σ |w_j|^q. q = 1 is still convex but has corners. Nondifferentiable at 0: the subgradient of |w| at 0 is [−1, 1]. Elastic net (L1 + L2) shares credit between correlated features; lasso prefers a vertex that keeps only one.

board calculation · five points, two features, decoy x₂

XᵀX = [[15, 1], [1, 2]], Xᵀy = [29.4, 2.3]
OLS: w = (1.9483, 0.1759)
Lasso, λ = 2: w = (1.8933, 0.0000)
Ridge at the same λ: (1.7209, 0.1448) — both smaller, neither zero

Constraint regions for q-norms — Bishop, PRML (2006), fig. 3.4 — constraint regions for Σ |w_j|^q ≤ η. q = 2 disk; q = 1 diamond.

L1 diamond and L2 disk — Bishop & Bishop (2023), fig. 9.6 — L1 diamond versus L2 disk.

live plot

Diamond versus disk

Grey ellipses: loss around OLS. Marker: first contact with the constraint. Drag η and watch L1 snap to an axis.

tight budgetloose budget

budget η

Where the contour first touches

L1 ŵ—

L2 ŵ—

Tighten the budget until the teal marker sits on an axis. The coral marker on the disk stays off-axis.

theory Which penalty can set a useless feature’s weight to exactly zero at a finite λ?

your turn · numbers

Soft-thresholding

S_λ(z) = sign(z) max(|z| − λ, 0). First write the expression, then evaluate.

you needλ = 2z = 1.5

S₂(1.5) = ?

you needsign(1.5) max(|1.5| − 2, 0)

S₂(1.5) = sign(1.5) max(−0.5, 0) = ?

you needλ = 2z = 3.0

S₂(3.0) = sign(3) max(3 − 2, 0) = ?

theory Two features are strongly correlated. Lasso tends to

dropout

Delete activations for a step

✓ complete

🔒

Section locked

Complete the section immediately above.

In one minuteInstead of deleting a weight forever, delete a random hidden unit for one step. Stored weights still exist. At test, use the full net. Inverted dropout puts the 1/p scale in training so test is a plain forward pass.

inverted dropouttrain: h̃ = (m ⊙ h) / p test: h̃ = h

A unit cannot rely on a particular neighbour being present, so fragile cancellations are discouraged. Typical keep probability p ≈ 0.5 on hidden units.

Optional · textbook figures and the four-unit tablePrince figs. 9.8–9.9, Goodfellow fig. 7.6, and inverted-dropout scaling.

Srivastava et al. (2014); Goodfellow §7.12. Draw m_i ~ Bernoulli(p). The original paper multiplied leftover weights by p at test. Inverted dropout divides by p when a unit is kept. Typical p: 0.8–0.9 on inputs, 0.5–0.8 on hidden. MC dropout (keep dropping at test) is a cheap uncertainty estimate — not the default.

board calculation · p = 0.5

inverted dropout
unit	h	m	h̃ = m h / p
1	1.2	1	2.4
2	−0.4	0	0
3	2.0	1	4.0
4	0.8	0	0

Kept units double. At test the same layer forwards (1.2, −0.4, 2.0, 0.8).

Network and thinned copies under dropout — Prince, UDL, fig. 9.8 — a network and three random thinned copies.

Co-adaptation kink — Prince, fig. 9.9 — a kink that needs two units acting together. Dropout makes that conspiracy unreliable.

Dropout as an ensemble — Goodfellow et al., fig. 7.6 — dropout as an ensemble of thinned networks that share weights.

live plot

A four-unit hidden layer

Small label = raw h. Big number = forwarded h̃. Toggle test: mask off, no 1/p.

keep probability

How often a unit survives

0.5

theory At test time, inverted dropout

your turn · numbers

Inverted dropout

Train: h̃ = m h / p. Write the expression for unit 1, evaluate it, then do unit 2.

you needp = 0.5h = 1.2m = 1

h̃₁ = ?

you need(1)(1.2) / 0.5

h̃₁ = (1)(1.2) / 0.5 = ?

you needp = 0.5h = −0.4m = 0

h̃₂ = (0)(−0.4) / 0.5 = ?

theory Two hidden units that only work as a cancelling pair will, after dropout training, be

batch normalisation

Re-scale the minibatch

✓ complete

🔒

Section locked

Complete the section immediately above.

In one minuteHidden values wander as earlier layers train. Batch norm standardises each channel over the current minibatch, then lets the net undo that with learned γ and β. At test, freeze running averages — a single image has no batch companions.

forward passû = (u − μ_B) / √(σ_B² + ε), y = γ û + β

Use the population variance (divide by N). One (γ, β) pair per channel. Train uses batch stats; test uses running stats.

Optional · textbook figures and the four-point calculationBishop & Bishop §7.4.2, Prince fig. 11.14, and why BN also regularises.

If the net wants the identity it can set γ = √(σ²+ε) and β = μ. Two stories: internal covariate shift (Ioffe & Szegedy) versus a smoother loss (Santurkar et al.). Operationally you need neither to implement the four lines. Smaller batches → noisier BN. Very small batches are why LayerNorm won in transformers.

board calculation · u = (1, 3, 5, 7), ε = 0, γ = 2, β = 1

μ_B = 16/4 = 4
σ_B² = (9+1+1+9)/4 = 5, √5 ≈ 2.2361
û ≈ (−1.3416, −0.4472, +0.4472, +1.3416)
y = 2 û + 1 ≈ (−1.6832, 0.1056, 1.8944, 3.6832)

Batch normalisation in the hidden pipeline — Bishop & Bishop, §7.4.2 — batch normalisation in the hidden pipeline.

BatchNorm versus LayerNorm, InstanceNorm, GroupNorm — Prince, fig. 11.14 — BatchNorm versus LayerNorm / InstanceNorm / GroupNorm.

live plot

μ, σ, then γ û + β

Navy band = ±σ around μ. Coral = raw u. Teal = y after scale-and-shift. Drag u₄.

u₄ = −2u₄ = 14

affine undo

Let the net keep a useful range

γ (scale)

β (shift)

μ used4.00

σ² used5.00

Drag u₄ in train mode and watch μ move. Switch to test: μ stays at the running value 4.

your turn · numbers

One batch of four

u = (1, 3, 5, 7), ε = 0, γ = 2, β = 1. Population variance: divide by N = 4. Expression, then number, then y.

you needu = (1, 3, 5, 7)N = 4

μ_B = ?

you need(1 + 3 + 5 + 7) / 4

μ_B = 16 / 4 = ?

you needμ_B = 4u = (1, 3, 5, 7)

σ_B² = [(1−4)² + (3−4)² + (5−4)² + (7−4)²] / 4 = ?

you needu₁ = 1μ = 4σ² = 5γ = 2β = 1

y₁ = γ (u₁ − μ) / √σ² + β = ?

theory You pass one image at test. Batch norm should normalise with

theory You leave BN in train mode and test one image at a time. What happens?

early stopping

Delete training time

✓ complete

🔒

Section locked

Complete the section immediately above.

In one minuteThe last epoch is often the wrong one to ship. Validation typically falls, then rises. The rise is overfitting in time. Save θ at the validation minimum. Patience P waits a window after that best point, then restores.

checkpointθ̂ = θ_t* with t* = arg min_t L_val(θ_t)

Never take the gradient of validation. Do not pick t* with the test set and then report that number. For a linear net with quadratic error, early stopping is roughly ridge with λ related to 1/(training time).

Optional · textbook figuresGoodfellow fig. 7.3, PRML figs. 5.11–5.12, Prince fig. 9.6.

Gradient descent on a quadratic is slowest along the small-eigenvalue directions — the same directions ridge damps. Stopping before those directions finish is like leaving them shrunk.

Training NLL falling while validation NLL rises — Goodfellow et al., fig. 7.3 — training NLL keeps falling; validation NLL minimises and then rises.

Training versus validation against epochs — Bishop, PRML, figs. 5.11–5.12 — training vs validation against epochs, and effective parameters growing with time.

Small network trained past the useful point — Prince, fig. 9.6 — a small network trained past the useful point: extra wiggles. Early stopping is the snapshot before those wiggles.

live plot

Watch the two losses separate

Coral = training MSE. Teal = validation MSE. The star is the checkpoint to keep.

checkpoint

When would you stop?

epoch

train MSE—

validation MSE—

Click through 1 → 20 → 80 → 300. Ask which error is still falling at 300, and which has already turned.

theory On the live plot, which checkpoint should early stopping keep?

your turn · numbers

Pick t*

L_val: 1 → 0.40, 20 → 0.18, 80 → 0.12, 300 → 0.21. Write the rule, then the epoch.

you needL_val(1)=0.40L_val(20)=0.18L_val(80)=0.12L_val(300)=0.21

t* = ?

you needarg min L_val0.40, 0.18, 0.12, 0.21

t* = ?

theory Early stopping regularises because

which knob

They are not interchangeable

✓ complete

🔒

Section locked

Complete the section immediately above.

In one minuteRead (L_train, L_val) first. High–high: relax the regularisers. Low–high: regularise, stop earlier, or add data. Then pick the tool that matches the model class.

One-line map. L1 deletes weights. L2 shrinks weights. Dropout deletes activations at train. BN re-scales activations. Early stopping deletes training time.

Optional · defaults by architectureConv net vs linear model vs transformer.

Residual conv net: batch norm (or LayerNorm) on, weight decay ~ 10⁻⁴, early stopping, dropout only if you still overfit. Linear model on many correlated features: lasso or elastic net, not dropout. Transformer: LayerNorm, residual dropout, weight decay, early stopping. The architecture already is a regulariser. Penalties sit on top.

live plot

Apply one knob

Same six points every run. Dashed = true sine. Preferred setting: low unseen error, not the lowest training loss.

experiment

Choose, then train

method

strength

Not evaluated

Training error—

Validation error—

Select a method and a strength, then train.

0 configurations evaluated

what “best” looks like on this toy L2 at mid strength: both errors low and close.

theory On this dataset, which setting has the lowest validation error with a small gap?

theory Which tool deletes activations for one training step, while leaving stored w dense?

Answer the previous question first.

Diagnose from evidence

✓ complete

🔒

Section locked

Complete the section immediately above.

In one minuteCompare seen and unseen error. Ask whether weights hit zero or merely shrink. Ask whether the noise was in w, in activations, in batch statistics, or in training time.

Four cases. A and B: high–high is underfit, low–high is overfit. C and D separate L1 from dropout.

train .210 val .238

Both losses plateau high. λ is huge.

theory Diagnosis

train .003 val .146

Degree 9, λ = 0, 300 epochs.

theory Diagnosis

w = (1.89, 0.00)

Two features; the second is a decoy.

numbers Which regulariser did this?

train mask · test full net

Predictions change if you forget to switch mode.

theory This train/test pair is

End of notes

You can now pick a regulariser from evidence — and explain why the diagnosis changes across λ, masks, batches, and time.

leave on screen

Six ideas to keep

L2 / ridge: Shrinks every weight. ŵ = (XᵀX + λI)⁻¹ Xᵀy. Almost never exact zeros. Weight decay is the SGD form.
L1 / lasso: Diamond corners. Soft-thresholding. Sparse w at finite λ.
Dropout: Random subnetwork each step; inverted scale 1/p at train; full net at test.
Batch norm: Normalise over the minibatch, then γ û + β. Test uses running statistics.
Early stopping: Keep θ at the best validation epoch. Time is capacity.
The split: Tune λ, p, patience on validation. Report test once.

Spend capacitymore slowly.

Why add anything to the loss?

L2

L1

Dropout

Batch norm

Early stop

Training set

Validation set

Test set

Add the penalty

Shrink every weight

Turn the λ dial

How hard to pull toward 0

Two-point ridge

Delete some of them

Diamond versus disk

Where the contour first touches

Soft-thresholding

Delete activations for a step

A four-unit hidden layer

How often a unit survives

Inverted dropout

Re-scale the minibatch

μ, σ, then γ û + β

Let the net keep a useful range

One batch of four

Delete training time

Watch the two losses separate

When would you stop?

Pick t*

They are not interchangeable

Apply one knob

Choose, then train

Diagnose from evidence

You can now pick a regulariser from evidence — and explain why the diagnosis changes across λ, masks, batches, and time.

Six ideas to keep

Spend capacity
more slowly.