Remember what
came before.
A word changes the meaning of a sentence. A frame changes the direction of a journey. Today we give a network a past—and discover why keeping that past is harder than it looks.
Commit to an answer before the reveal.
Follow the information and the numbers.
Change one thing. Explain what changes.
Can one clue survive an entire lecture?
Read this word once. Hide it. Keep it for the end.
ORCHIDNo tricks: we will ask for exactly this word.
A model faces a similar problem: an early clue can matter only after many later inputs.
- If we ignore order, what information disappears?
- How can the same network read a short or a long sequence?
- If a clue matters much later, can the model learn to keep it?
Keep the third question open. We need the first two ideas to answer it.
From the last lecture: a CNN reuses a detector across spatial locations. An RNN reuses a state-update rule across time steps. In both cases, sharing parameters is a useful assumption about the problem.
The order is part
of the evidence.
Same three words. Same three frames. Can they tell different stories?
A sequence is an ordered collection of observations. Its steps can be words, audio samples, sensor readings, clicks or video frames. A step may be one number or a whole vector. The index t tells us where the observation sits; it need not represent one second.
Think of sheet music. A bag containing all the right notes does not specify the melody. We need both the notes and their order. In the same way, a model may need “what happened before” to interpret “what is happening now”.
For a temperature forecast, 18 → 21 → 24 and 24 → 21 → 18 have the same mean but different trends. We should preserve order when it is relevant to the target. Some tasks, such as summing a shopping basket, deliberately ignore order.
order-blind mean: m = (1/T) Σt=1…T xt
T: number of steps · xₜ ∈ ℝᵈ: the d features at step t · rearranging the terms leaves the mean unchanged, even if the sequence's meaning changes.
Go deeper Sets, windows, and what a CNN can already do
Order sensitivity is a property of the whole pipeline
A mean or sum is permutation-invariant: reordering its inputs does not change it. If two different sequences map to the same summary, a deterministic model that only sees that summary cannot distinguish them. A flattened ordered window does retain positions. An MLP on that window can learn order-sensitive rules, but its input size is fixed unless we add a strategy for variable length.
RNNs are one solution, not the definition of sequence modelling
Temporal convolutions can learn local sequence patterns; their context depends on kernel size, depth, dilation and causal padding. Attention-based models can mix information across positions. A recurrent network offers another useful assumption: update a state as each observation arrives. We study it to make context and memory explicit.
Time comes with a measurement convention
In irregularly sampled measurements, equal index gaps need not mean equal elapsed time. Supply time gaps or use a suitable model when timing matters. We assume equally indexed steps in these demonstrations.
Worked mean: the centre row of our three image frames is [1,0,0], [0,1,0], [0,0,1]. Their mean is [⅓,⅓,⅓] in either order. The other two rows stay [0,0,0].
What is an input?
What is the question?
You have read “not very good”. Are you predicting a review label, a word's role, or the word that comes next?
Before selecting a model, specify the unit of input, the target, and the moment of prediction. A sentence classifier can wait for the last word. A live next-word predictor must operate on a prefix. A tagger produces one label per token; whether it may look ahead depends on the intended use.
Think of a commentator and a reviewer. A commentator must describe the match with only the events seen so far. A reviewer can wait until the match ends. Both see sequences, but they have different information budgets.
A token must become numbers. Here, the tiny vocabulary [not, very, good] becomes three one-hot vectors. “not” is [1,0,0]. The 1 marks an identity; it does not say the word is larger or more positive. Real systems often replace these sparse vectors with learned embeddings.
aligned output: ŷt = g(ht)
next token: p(xt+1 | x₁, …, xt)
hₜ: a summary of the prefix, introduced next · g: the output/readout function · ŷ: a prediction · a causal prediction at t has access to x₁…xₜ, not later inputs.
Go deeper Shapes, masks, shifted targets and leakage
Separate time, features and examples
One sequence has T vectors, each of dimension d. With batch size B, a common tensor layout is B × T × d. Some libraries use T × B × d. Neither T nor d is the batch size. In the animation, T = 3 and d = 3; that equality is accidental.
| Step | Input token | One-hot input | Next-token target |
|---|---|---|---|
| 1 | not | [1, 0, 0] | very |
| 2 | very | [0, 1, 0] | good |
| 3 | good | [0, 0, 1] | EOS |
EOS is a special end-of-sequence target. It adds an output category to this tiny next-token example. The input vocabulary here still contains only our three displayed words; a full language model usually includes special tokens in its vocabulary too.
Padding is storage, not evidence
To batch unequal lengths, we may pad short sequences. Mask padded targets out of the loss and use each sequence's last valid state for a final classification. Masking the loss alone does not stop an RNN from updating its hidden state on padding; use lengths, packed sequences or an explicit state-update mask when needed.
The future can leak through preparation
For forecasting, split according to the intended time horizon and source groups. Fit preprocessing on the training portion. Avoid overlapping train/test windows that expose the same future observations. A bidirectional model can use later context for offline tagging, but cannot use unavailable future words for a live causal prediction.
During next-token training, the true prefix is usually supplied. During generation, a model's previous generated token often becomes its next input. Mistakes can then affect later inputs. We will return to that difference in a later lesson.
Carry a summary.
Update it as you read.
The current input is identical. Why should the answers be different?
A recurrent neural network (RNN) maintains a hidden state, hₜ. At each step, it reads xₜ, combines it with hₜ₋₁, and produces a new state. A readout can turn that state into a prediction. “Hidden” means an internal representation—not a secret answer and not a separately labelled memory slot.
Think of a small running notebook. You read a new line, consult the note you already have, then rewrite your summary. The notebook has limited space. In a trained RNN, learning determines what the update should retain; no one assigns every hidden coordinate a meaning by hand.
The state changes while reading a sequence. The weights specify the update rule and are reused at every step. Training changes those weights across optimization steps. Processing a new token is not itself a weight update.
ztout = Wyht + by
Column vectors: xₜ ∈ ℝᵈ, hₜ ∈ ℝᴴ. Wₓ: H × d; Wₕ: H × H; Wᵧ: C × H. Biases have H and C entries. tanh acts elementwise. For classification, softmax turns the C output logits into probabilities. We set h₀ = 0.
Go deeper One complete matrix update, tanh and state boundaries
Read the numbers, not just the colours
Take d = 3, H = 2, xₜ = [1,0,0]ᵀ, hₜ₋₁ = [0.2,−0.1]ᵀ and zero bias. These chosen weights illustrate a calculation; they are not a trained language model.
| 1 | 0 | −1 |
| 0 | 1 | 1 |
| 0.5 | 0 |
| 0 | 0.5 |
Wₓxₜ = [1,0]ᵀ; Wₕhₜ₋₁ = [0.1,−0.05]ᵀ. Adding gives [1.1,−0.05]ᵀ. Applying tanh gives hₜ ≈ [0.8005,−0.0500]ᵀ. We use column vectors consistently; some textbooks transpose the same convention.
Why an activation?
tanh(z) = (eᶻ − e⁻ᶻ)/(eᶻ + e⁻ᶻ) maps real numbers into (−1,1). It introduces nonlinearity. Its derivative is 1 − tanh²(z), which is close to 1 near zero and small in saturated regions near ±1. Bounded states do not imply bounded parameter gradients.
Where does a new sequence begin?
Reset the state for independent examples. Carrying it between unrelated people, documents or shuffled windows can mix contexts. In stateful streaming, carry state deliberately across truly adjacent chunks. Resetting state, cutting gradient history, and updating weights are three different actions.
An RNN is first-order in its hidden state, but that state can depend on many earlier inputs. It is not restricted to using only the previous raw input. Finite state width, optimization and the data determine what information it actually keeps.
Draw the time.
Keep the same weights.
If a sentence doubles in length, do we need twice as many networks?
Unrolling replaces the loop with a sequence of operations. Compute h₁ from x₁ and h₀, then h₂ from x₂ and h₁, and so on. The time axis makes the computational dependencies visible. You cannot finish this standard recurrent forward pass for hₜ before its predecessor is available.
Think of one rubber stamp used on many pages. The stamp is the parameter set. Each impression is a new computation with its own input and state. More pages mean more uses, not more stamps. A stack of RNN layers would introduce a separate vertical depth axis.
Training uses the same graph in reverse. A final error sends credit through earlier states by the chain rule: backpropagation through time (BPTT). Every occurrence of a shared parameter contributes to the gradient of that one parameter. Backward derivatives are a training calculation; the forward model still sees only its permitted inputs.
P = Hd + H² + H + CH + C
∇θL = Σ contributions from shared uses
P: trainable parameter count for one simple recurrent layer plus linear output head, including biases; h₀ is fixed · d: input width · H: hidden width · C: output width · T does not appear in P.
Go deeper Count every parameter and trace training credit
| Object | Shape | d = 3, H = 2, C = 2 |
|---|---|---|
| Input weights Wₓ | H × d | 6 |
| Recurrent weights Wₕ | H × H | 4 |
| State bias bₕ | H | 2 |
| Output weights Wᵧ | C × H | 4 |
| Output bias bᵧ | C | 2 |
| Total | 18 |
Shared weights do not make long sequences free
Longer sequences require more cell evaluations. Standard full BPTT stores intermediate activations across the unroll, so training memory also grows with T. In streaming inference, the recurrence can carry only its current state if no earlier states are needed by the task. Exact runtime and storage depend on implementation.
One error can teach many earlier uses
For a final-only task, L = ℓ(g(hT), y). For aligned outputs, one common objective is L = (1/N)Σₜmₜℓ(g(hₜ),yₜ), where mₜ is 1 for a valid labelled step and 0 for padding, and N = Σₜmₜ. In either case, differentiate through every connected state dependency. Summing parameter-use contributions is required by weight sharing; whether the loss is a sum or mean is a separate scaling choice.
For a scalar illustration, h₁ = wx₁, h₂ = wh₁ and L = h₂, so L = w²x₁. Its derivative is 2wx₁: one contribution from each use of w. Do not update the “first w” and “second w” as different parameters.
Three different meanings of “more”
Increasing T adds time steps. Increasing H widens the state and changes the parameter count. Adding layers stacks distinct recurrent transformations. Unrolling alone changes neither H nor the number of layers.
A path to the past
is not a promise of memory.
“The key to the cabinets near the windows … is missing.” How far back must the model look to choose “is”?
An RNN has a computational route from early inputs to later states. But information may be overwritten or transformed along that route. During training, the error signal also has to cross many transformations to teach earlier computations what to retain.
Think of a message relayed across a room. The forward message is what gets carried. The backward message is a correction explaining how an earlier relay should change. The two messages travel in opposite directions and answer different questions.
The backward sensitivity is a product of local derivatives. Repeated contraction can make a distant influence tiny: a vanishing gradient. Repeated expansion can make it huge: an exploding gradient. For a real vector state, these factors are matrices whose directions and activation derivatives matter. Our scalar experiments isolate the multiplication.
∂hL/∂h₀ = JLJL−1…J₁
linear scalar: wL
tanh scalar: Jt = w(1 − ht²)
L: number of recurrent links in this isolated experiment (we use ℒ or the word “loss” when needed to avoid confusing the two) · xₜ = 0 · input values and weights held fixed · the loss gradient would multiply this sensitivity by a derivative from the readout/loss.
Go deeper Derivatives, saturation and what the toy does not prove
A complete scalar path
Let hₜ = whₜ₋₁, h₀ = 1 and all inputs be zero. Then hL = wᴸh₀ and ∂hL/∂h₀ = wᴸ. At w = 0.8 and L = 20, that sensitivity is 0.011529…, about 1.15% of the unit seed. At w = 1.2, it is 38.3376…. These are sensitivities, not probabilities or accuracies.
Why looking at w alone is insufficient
For hₜ = tanh(whₜ₋₁), the local derivative includes 1 − hₜ². At saturated states, that term is small. Even w greater than 1 can produce a shrinking product. In a vector RNN, Jₜ = diag(1 − hₜ²)Wₕ. The norm of a product is bounded by the product of norms, but that bound need not be tight. A single weight magnitude or eigenvalue is not a universal diagnosis.
Do not merge three different quantities
State magnitude describes the forward representation. State sensitivity describes how a perturbation influences a later state. Parameter gradient sums loss-dependent contributions from parameter uses. Our plot isolates state sensitivity. A small state is not proof of lost information; a finite-precision or noisy system can nevertheless struggle to use a very weak signal.
What helps—and what does each remedy leave unresolved? The bridge to the next lecture
| Idea | What it changes | Keep in mind |
|---|---|---|
| Gradient clipping | Bounds an overly large update signal. | Does not enlarge a vanished gradient. |
| Truncated BPTT | Limits how far gradients travel during a training chunk. | Can carry state forward while cutting credit to earlier chunks. |
| Gated memory (LSTM/GRU) | Learns controls for retaining and replacing information. | Improves available paths, not guaranteed unlimited recall. |
| Attention | Can access relevant states through shorter connections. | Still needs appropriate positions, masks and computational resources. |
Clipping: a brake, not an amplifier
Norm clipping uses g ← g · min(1, τ/‖g‖) for nonzero g and threshold τ > 0. Leave g = 0 unchanged. In our scalar version, clipping 38.338 at τ = 5 gives 5; clipping 0.011529 gives 0.011529.
Preview a path that can preserve a derivative
An LSTM includes an additive cell update cₜ = fₜ ⊙ cₜ₋₁ + iₜ ⊙ c̃ₜ. Along its direct cell-state connection, holding the gate values fixed, the derivative is fₜ. A forget gate near 1 can preserve that path's signal. The full derivative also includes indirect gate dependencies, and this is not the complete LSTM. Next time we will learn how the gates are computed.
Suspense to carry forward: can a network learn when to keep its notebook unchanged and when to write something new?
Explain the memory.
Then question its limits.
Your design challenge. A sensor sends one measurement per minute. After observing a variable-length stream, you must report whether an early warning occurred before a later fault. The same two events in the opposite order should produce a different answer.
Compare with the discussion guide
The target depends on event order, so an order-blind mean can merge examples with different labels. Read a measurement or event-feature vector per minute; produce one sequence-level binary label at the last valid step. The state should preserve whether a warning has occurred and whether a qualifying later fault has occurred. Reset between independent streams.
Possible explanations include lost/overwritten representations, weak training gradients, insufficient examples at long gaps, or a preprocessing/label bug. Evaluate matched examples with varied gaps; inspect gradient norms by time step, check target alignment, and compare a simple explicit event-state baseline. Poor accuracy alone does not identify vanishing gradients.
The word was ORCHID. Both tasks need a distant clue. Human memory uses many processes, rehearsal and retrieval cues; our scalar recurrence is a controlled mathematical analogy, not a model of human cognition.
We now have a past. Next, we need a way to decide what deserves to stay.
Keep exploring.
Original diagrams and numerical teaching examples. No downloaded model, external font, or network connection is needed for the worksheet.
- Dive into Deep Learning: Recurrent Neural Networks — recurrence, hidden states, output layers and parameter sharing. Our equations use column-vector notation.
- Dive into Deep Learning: Backpropagation Through Time — the unrolled graph and how shared-parameter contributions combine.
- Pascanu, Mikolov & Bengio (2013): On the difficulty of training recurrent neural networks — analysis of vanishing and exploding gradients and gradient clipping.
Answers stay in this browser on this device. Use “Save my answers” to keep a text copy. Progress supports practice; it is not a grade.