The vanilla (Elman) recurrent neural network is the simplest member of the RNN family: one hidden layer whose output is fed back as an input at the next time step. It is worth understanding in detail because its two famous weaknesses — vanishing and exploding gradients — explain every design decision in LSTM, GRU and, ultimately, attention.
At each step the cell combines the current input and the previous hidden state through two weight matrices, adds a bias, and squashes the result with tanh: ht = tanh(W xt + U ht−1 + b). An output layer yt = V ht + c maps the state to logits or a regression value. The initial state h0 is usually zeros. Parameter count is dh(din + dh + 1) for the recurrence plus the output layer. (The related Jordan network feeds back the output y instead of h.)
tanh keeps the state bounded in (−1, 1) and is zero-centered, which helps optimization. ReLU recurrences are possible (IRNN, with identity-initialized U) but are prone to unbounded activations.
The whole model fits in a few lines of NumPy, which makes the sequential dependency obvious: step t cannot start until step t−1 finishes.
import numpy as np
rng = np.random.default_rng(0)
d_in, d_h, d_out, T = 3, 16, 2, 20
W = rng.normal(0, 1 / np.sqrt(d_in), (d_h, d_in))
U = rng.normal(0, 1 / np.sqrt(d_h), (d_h, d_h))
V = rng.normal(0, 1 / np.sqrt(d_h), (d_out, d_h))
b, c = np.zeros(d_h), np.zeros(d_out)
def rnn_forward(xs, h0=None):
h = np.zeros(d_h) if h0 is None else h0
hs, ys = [], []
for x in xs: # strictly sequential
h = np.tanh(W @ x + U @ h + b)
hs.append(h)
ys.append(V @ h + c)
return np.array(hs), np.array(ys)
xs = rng.normal(size=(T, d_in))
hs, ys = rnn_forward(xs)
print(hs.shape, ys.shape) # (20, 16) (20, 2)
print("spectral radius of U:", max(abs(np.linalg.eigvals(U))))The loss is typically a sum of per-step losses L = Σ Lt. Each Lt depends on U through every earlier hidden state, so its gradient is a sum over paths of different lengths. A path of length t−k carries the Jacobian product Π diag(1 − hj2) U. Memory cost is linear in sequence length because all hidden states must be stored for the backward pass; truncated BPTT bounds it by backpropagating only k steps at a time.
Bound the Jacobian norm: ||diag(1 − h2) U|| ≤ ||U|| because the tanh derivative is at most 1. If the largest singular value of U is below 1, the contribution from n steps back shrinks at least like λn; saturated tanh units shrink it further. The gradient then carries almost no information about distant inputs, so the network effectively learns only short-range dependencies — typically on the order of ten or so steps in practice. If the gain is above 1, gradients can grow exponentially and a single batch can blow up the weights (Bengio et al., 1994; Pascanu et al., 2013).
Note the asymmetry: exploding gradients are loud (NaNs, loss spikes) and easy to fix by clipping; vanishing gradients are silent and require an architectural change.
U — starts with singular values at 1 so signals initially neither shrink nor grow.import torch
import torch.nn as nn
text = "hello world. " * 200
chars = sorted(set(text)); stoi = {c: i for i, c in enumerate(chars)}
data = torch.tensor([stoi[c] for c in text])
V, H, T = len(chars), 64, 32
class CharRNN(nn.Module):
def __init__(self):
super().__init__()
self.emb = nn.Embedding(V, 32)
self.rnn = nn.RNN(32, H, nonlinearity="tanh", batch_first=True)
self.head = nn.Linear(H, V)
def forward(self, x, h=None):
out, h = self.rnn(self.emb(x), h)
return self.head(out), h
model = CharRNN()
for name, p in model.rnn.named_parameters():
if "weight_hh" in name:
nn.init.orthogonal_(p) # gain-1 recurrence at init
opt = torch.optim.Adam(model.parameters(), lr=3e-3)
for step in range(500):
i = torch.randint(0, len(data) - T - 1, (16,))
x = torch.stack([data[j:j + T] for j in i])
y = torch.stack([data[j + 1:j + T + 1] for j in i]) # next-char targets
logits, _ = model(x)
loss = nn.functional.cross_entropy(logits.reshape(-1, V), y.reshape(-1))
opt.zero_grad(); loss.backward()
nn.utils.clip_grad_norm_(model.parameters(), 1.0)
opt.step()
# sample
x, h, out = torch.tensor([[stoi["h"]]]), None, "h"
with torch.no_grad():
for _ in range(30):
logits, h = model(x, h) # carry state, O(1) per step
x = torch.multinomial(logits[0, -1].softmax(-1), 1)[None]
out += chars[x.item()]
print(out)Use a vanilla RNN for teaching, for sequences where only the last few steps matter, or for extremely constrained hardware where gate overhead counts. For almost anything else, a GRU or LSTM costs a small constant factor more and trains far more reliably; for long-context or large-data problems, a Transformer or a modern state-space model is the usual choice.
Use a scalar RNN with W = 0.5, U = 0.8 and no bias. Feed the input sequence 1, 0, 0. Step 1: h1 = tanh(0.5·1 + 0.8·0) = tanh(0.5) ≈ 0.462. Step 2: h2 = tanh(0.8·0.462) = tanh(0.370) ≈ 0.354. Step 3: h3 = tanh(0.8·0.354) = tanh(0.283) ≈ 0.276. The trace of the first input shrinks by roughly a factor of 0.8 per step (a bit more, because tanh compresses), so after 20 steps it is around 1% of its original size.
The gradient tells the same story. ∂h3/∂h1 = U(1−h32) · U(1−h22) ≈ 0.8·0.924 · 0.8·0.875 ≈ 0.52. Each step multiplies by a factor below one, so the error signal from step t back to step 1 decays geometrically. With U = 1.5 and small activations, the same product would grow geometrically instead: exploding gradients. In matrix form, the largest singular value of U (times the tanh derivatives) plays the role of this factor.
The same cell supports several input-output arrangements. Many-to-one reads a whole sequence and uses the final state (or a pooled state) for a single prediction. Aligned many-to-many emits an output at every step, as in tagging or frame-level audio classification. One-to-many generates a sequence from a single input, feeding each output back as the next input. Sequence-to-sequence uses one RNN to encode and another to decode, the setup that attention was invented to improve (Bahdanau et al., 2015).
| Vanilla RNN | LSTM | GRU | |
|---|---|---|---|
| Parameters per layer | 1x (one weight set) | About 4x | About 3x |
| Long-range memory | Poor beyond tens of steps | Good, via additive cell state | Good, via update gate |
| Training stability | Needs careful init and clipping | Robust | Robust |
| Typical use today | Teaching, tiny models, reservoir computing | Speech, time series, legacy NLP | Same as LSTM, slightly cheaper |
The gates in LSTM and GRU exist precisely to replace the repeated multiplication by U with a path where information is copied forward almost unchanged, so the gradient does not shrink at every step. Newer state-space models (S4, Mamba) revisit the linear-recurrence idea with structured, carefully parameterized transitions that remain stable over very long sequences.