Keep the clue.
Read the context.

A word ID is a category label, not a meaningful scalar magnitude. A vocabulary of three words could use one-hot inputs [1, 0, 0], [0, 1, 0], [0, 0, 1]; an embedding layer maps IDs to learned vectors. The scalar inputs below deliberately isolate arithmetic.

Unrolling draws repeated uses of one cell, not independently parameterized layers. For a single recurrent layer with one bias, P = H(d + H + 1); making the sequence longer adds computation, not parameters.

For next-token training, [dog, bites] has targets [bites, person]. The target at a position must not also be supplied as future input to that prediction. Backpropagation through time sends derivatives backward during training; it does not give a forward RNN access to future inputs at inference.

Semicolons inside brackets mean concatenation; dots between equations separate formulas, not multiplication. σ(v) = 1/(1 + exp(−v)). Each W has shape H × (H + d); each bias has H values. Sigmoid produces values strictly between 0 and 1 for finite inputs; sliders include the useful idealized limits.

For cₜ₋₁ = 0.8, f = 0.75, i = 0.5 and g = 0.4, cₜ = 0.6 + 0.2 = 0.8. With o = 0.6, hₜ = 0.6 tanh(0.8) ≈ 0.398. With o = 0, hₜ = 0 while cₜ remains 0.8. The hidden value is neither a class probability nor a stored token.

The cell state is not forced into [−1, 1]; additions can accumulate. The standard hidden state is bounded by its tanh and output gate. This worksheet uses the modern forget-gate form, no peephole connections and no projection. For one bias per affine transform, P = 4H(d + H + 1), excluding a task head.

For a scalar vanilla RNN hₜ = tanh(w hₜ₋₁ + xₜ), ∂hₜ/∂hₜ₋₁ = w(1 − hₜ²). Even a weight above 1 may be paired with a small tanh derivative. The interactive comparison removes tanh from its RNN baseline so the two products can be inspected exactly.

With no writes and f = 0.95, an initial cell value retains a fraction 0.95ᴸ. Its half-life is log(0.5)/log(0.95) ≈ 13.5 steps; at f = 0.99 it is about 69 steps. Gate choices and useful timescales depend on the data.

The derivative from cₜ to exposed hₜ also contains oₜ(1 − tanh²(cₜ)). Saturation or a closed output gate can weaken that route even when the cell path preserves information. Gates also depend on prior hidden states. Therefore fᴸ is not the complete gradient of a loss or a parameter.

Gradient clipping limits large accumulated parameter gradients; it does not reconstruct a vanished signal. Truncated backpropagation limits how far training credit travels, even if hidden states are carried farther. LSTM does not remove finite memory, optimization difficulty or sequential computation.

With x = 0.2, old h = 0.8, r = 0.5, weights 1 and bias 0, the candidate is tanh(0.6) ≈ 0.537. At z = 0.75, hₜ = 0.75 × 0.8 + 0.25 × 0.537 ≈ 0.734. At r = 0, the candidate is tanh(0.2), but z = 1 still retains old h = 0.8.

Some references write (1 − z)hₜ₋₁ + z h̃ₜ, so their z means “write”. Some libraries use reset-after-projection, r ⊙ (Wₕh + bₕ), instead of Wₕ(r ⊙ h). These are not generally interchangeable for matrices. Match the implementation before copying weights or counting biases.

With one bias per affine transform, a GRU has P = 3H(d + H + 1), versus 4H(d + H + 1) for this LSTM. At equal H it has 25% fewer recurrent parameters. Frameworks with separate input and recurrent biases change this count. Fewer parameters can reduce storage, but measured speed depends on kernels and hardware; accuracy depends on the task and training.

For tokens [A, B, C], the backward computation visits C, then B, then A. Store or reorder its results by original token index before pairing forward and backward states. For token B, use [h→₂ ; h←₂], not [h→₂ ; h←₃].

At hidden width H = 4 per direction, each concatenated token representation has 8 features. A full-sequence summary can concatenate h→T with h←₁. Using the backward output aligned to the final token alone does not summarize the whole sequence in that direction. A bidirectional LSTM has 2 × 4H(d + H + 1) recurrent parameters under our bias convention; the head also gets a wider input.

Bidirectionality requires the chosen input window to be available. Offline transcription may use the whole recording; streaming can use a limited look-ahead window at a latency cost. Forecasting may bidirectionally encode a fully observed history window, but must never include the future being forecast. In translation, the whole source can be known while target generation remains causal.

Split related people, recordings or documents by source when appropriate; for forecasting use chronological evaluation and prevent overlapping future targets from leaking. Fit scalers on training data only. Padding is not meaningful time: mask padded outputs in the loss and prevent padded steps from contaminating state (for example, use lengths/packed sequences). Reset states between unrelated sequences. Carry state only when continuity and boundaries are intentional.