Average the centre rows [1,0,0], [0,1,0], [0,0,1]. What is the first cell of the mean row? A fraction is fine.
Answer: __________________________
My reasoning:
SEQUENCE MODELLING / STUDENT PRACTICE
01 · Order is information
The three 3×3 image frames have centre rows [1,0,0], [0,1,0], [0,0,1]. All other pixels are 0. Time runs in the stated frame order.
Which representation necessarily loses the distinction between our two directions?
□ All frames, in order · □ The time-averaged image
My reasoning:
“Only RNNs can learn order.” Could an MLP on a fixed, ordered window distinguish [1,0] from [0,1]?
□ Yes · □ No
My reasoning:
SEQUENCE MODELLING / STUDENT PRACTICE
02 · Inputs and targets
One-hot vocabulary: [not, very, good]. Sequence classification predicts a final label; token tagging aligns labels; next-token prediction shifts targets one step ahead.
Our input vocabulary is [not, very, good]. How many coordinates does each one-hot input vector have?
Answer: __________________________
My reasoning:
For next-token training on “not very good”, the model has just read “very”. What is its target?
□ very · □ good · □ negative
My reasoning:
At noon, a live forecast predicts the 1 pm temperature. May the input include the actual 1 pm measurement?
□ Yes · □ No
My reasoning:
SEQUENCE MODELLING / STUDENT PRACTICE
03 · A running state
Linear toy: hₜ = xₜ + 0.5hₜ₋₁, h₀ = 0. Tanh RNN: hₜ = tanh(Wₓxₜ + Wₕhₜ₋₁ + bₕ). States and weights are different objects.
For the linear toy hₜ = xₜ + 0.5hₜ₋₁, h₀ = 0 and inputs [1,0,1], what is h₂?
Answer: __________________________
My reasoning:
During an ordinary inference pass, which can change from one time step to the next?
□ The hidden state · □ The shared weights are retrained
My reasoning:
Using tanh, h₀ = 0, x₁ = 1, input weight 1, recurrent weight 0.5 and bias 0, what is h₁ to three decimals?
Answer: __________________________
My reasoning:
SEQUENCE MODELLING / STUDENT PRACTICE
04 · Unroll the RNN
One recurrent layer + output head: P = Hd + H² + H + CH + C. Count each shared weight and bias once; the initial state is fixed.
For d = 3, H = 2 and C = 2, how many parameters are in the recurrent layer plus output head, including biases?
Answer: __________________________
My reasoning:
The same model now reads T = 100 steps instead of T = 4. How many trainable parameters does it have?
Answer: __________________________
My reasoning:
Two time steps contribute gradients 0.2 and 0.3 to the same weight, with the loss scaling already included. What gradient updates that weight?
□ 0.3: keep the last · □ 0.5: add contributions · □ 0.06: multiply contributions
My reasoning:
SEQUENCE MODELLING / STUDENT PRACTICE
05 · Distant dependencies
Across a recurrent path, local derivatives multiply. For a scalar linear chain the sensitivity is wᴸ. For tanh each local derivative is w(1 − hₜ²).
For 20 linear recurrent links with derivative 0.8 each, what fraction of a unit sensitivity reaches the beginning? Give a decimal.
Answer: __________________________
My reasoning:
Can clipping a gradient of 0.0115 at threshold 5 restore it to 1?
□ Yes · □ No
My reasoning:
With tanh and w = 1.3, why can the returning sensitivity still become small?
□ It cannot: w > 1 guarantees explosion · □ The factors 1 − hₜ² can be small
My reasoning:
SEQUENCE MODELLING / EXIT TICKET
Design. Diagnose. Remember.
A sensor sends one measurement per minute. At the end of a variable-length stream, report whether an early warning happened before a later fault. Reversing those events should change the answer.