A recurrent neural network processes a sequence one element at a time while carrying a hidden state forward, so the output at step t can depend on everything seen before it. The same weights are applied at every step, which lets one model handle sequences of any length. RNNs — and especially their gated forms, LSTM and GRU — were the standard architecture for language, speech and time series until Transformers took over most large-scale sequence work around 2018. They remain useful for streaming, low-latency and small-data problems, and the core ideas reappear in modern linear-recurrent and state-space models.
This page is the hub for the recurrent family under Deep Learning (DL).
The simple Elman recurrence h = tanh(Wx + Uh + b), backpropagation through time, and why gradients vanish.
A gated cell state with input, forget and output gates that preserves information over long spans.
A lighter two-gate alternative to LSTM that merges cell and hidden state; often equally accurate.
Forward and backward passes concatenated so every position sees both past and future context.
Multiple recurrent layers on top of each other for hierarchical sequence features.
The attention-based successor that processes all positions in parallel.
The defining equation is ht = f(xt, ht−1). The hidden state h is a fixed-size summary of the prefix x1..t. Drawing the loop as a chain of copies (unrolling) shows that an RNN is just a very deep feedforward network whose layers share weights — depth equals sequence length. Weight sharing gives generalization across positions and a parameter count independent of length; the chain structure gives sequential processing with constant memory per step at inference, which is ideal for streaming.
The same cell supports several task shapes. Many-to-one reads the whole sequence and classifies from the final hidden state (or a pooled state) — sentiment, anomaly detection on a window. Many-to-many aligned emits an output per input step — part-of-speech or named-entity tagging, per-frame speech labels. One-to-many generates a sequence from a seed — image captioning, character-level text generation. Encoder-decoder (seq2seq) uses one RNN (blue) to encode the input into a state and another (yellow) to decode an output of different length; adding attention over encoder states (Bahdanau, 2014) was the step that led to Transformers.
Training unrolls the network over the sequence, computes the loss, and runs ordinary backpropagation through the unrolled graph — Backpropagation Through Time (BPTT). Because the same recurrent matrix appears at every step, the gradient from step T to step 1 contains a product of T−1 Jacobians. That product tends to shrink or grow exponentially, which is the vanishing / exploding gradient problem analyzed in detail on the Vanilla RNN page. Exploding gradients are handled by gradient-norm clipping; vanishing gradients motivated the gated architectures. For very long sequences, truncated BPTT backpropagates through a fixed window of steps while still carrying the hidden state forward.
| Variant | Idea | Typical use |
|---|---|---|
| Vanilla (Elman) RNN | Single tanh layer recurrence | Teaching, very short dependencies |
| LSTM | Additive cell state with forget/input/output gates | Long dependencies, time series, speech |
| GRU | Update and reset gates, no separate cell | Same as LSTM with fewer parameters |
| Bidirectional | Forward + backward pass, concatenated | Offline tagging, encoders |
| Stacked (deep) | Several recurrent layers | Harder tasks, larger data |
A many-to-one sequence classifier. Switching between RNN, LSTM and GRU is a one-line change; packing handles variable-length batches so padding does not leak into the final state.
import torch
import torch.nn as nn
from torch.nn.utils.rnn import pack_padded_sequence
class SeqClassifier(nn.Module):
def __init__(self, vocab, d_emb=64, d_hid=128, n_cls=2, cell="lstm"):
super().__init__()
self.emb = nn.Embedding(vocab, d_emb, padding_idx=0)
rnn_cls = {"rnn": nn.RNN, "lstm": nn.LSTM, "gru": nn.GRU}[cell]
self.rnn = rnn_cls(d_emb, d_hid, batch_first=True)
self.out = nn.Linear(d_hid, n_cls)
self.cell = cell
def forward(self, tokens, lengths):
x = self.emb(tokens) # (B, T, d_emb)
packed = pack_padded_sequence(x, lengths.cpu(), batch_first=True,
enforce_sorted=False)
_, h = self.rnn(packed)
if self.cell == "lstm":
h = h[0] # (h_n, c_n) -> h_n
return self.out(h[-1]) # last layer's final state
model = SeqClassifier(vocab=10_000, cell="gru")
tokens = torch.randint(1, 10_000, (8, 50))
lengths = torch.randint(5, 51, (8,))
for i, L in enumerate(lengths):
tokens[i, L:] = 0 # pad
logits = model(tokens, lengths)
loss = nn.functional.cross_entropy(logits, torch.randint(0, 2, (8,)))
loss.backward()
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0) # essential for RNNs
print(logits.shape) # torch.Size([8, 2])output[:, -1] is a padding position; use packed sequences or gather by length.A scalar RNN with ht = tanh(w·xt + u·ht−1), w = 1, u = 0.5, h0 = 0, fed the input sequence [1, 0, 0]:
h1 = tanh(1) ≈ 0.762h2 = tanh(0.5 × 0.762) = tanh(0.381) ≈ 0.364h3 = tanh(0.5 × 0.364) = tanh(0.182) ≈ 0.180The trace of the first input roughly halves each step, because u = 0.5. The gradient tells the same story: ∂h3/∂h1 = u(1 − h32) · u(1 − h22) ≈ 0.484 × 0.434 ≈ 0.21. Each step multiplies by u times a tanh derivative that is at most 1, so after 20 steps the signal from step 1 is around 10−6 of its size. Making u larger than 1 does not fix this: it makes gradients explode wherever the tanh is unsaturated and still vanish where it saturates. Gated cells sidestep the dilemma with an additive path whose multiplier the network controls — see the GRU and LSTM pages for the same calculation with gates.
The classical RNN's weakness for training is the nonlinearity between steps: ht cannot be computed until ht−1 is known. If the state update is linear in the previous state, the whole sequence of states can be computed with a parallel prefix scan (or, when the parameters do not depend on the input, as one long convolution), so training parallelizes across time like a Transformer. Inference can still run as a recurrence with a fixed-size state.
This is the idea behind structured state-space models such as S4, which parameterize A so that long-range memory is stable, and Mamba, which makes the parameters depend on the current input (a "selective" state space) so the model can choose what to store and what to ignore — recovering the content-based gating that LSTMs and GRUs had. RWKV and linear-attention variants reach a similar place from the Transformer side. Hybrids that interleave a few attention layers with many recurrent or state-space layers are a common design, because pure fixed-size states still struggle with exact recall of arbitrary earlier tokens, which is what attention does well.
| Situation | Reasonable first choice |
|---|---|
| Short sensor or tabular time series, little data | Gradient boosting on lag features, or a small GRU/LSTM; compare against a seasonal-naive baseline (see Time-Series Forecasting). |
| Streaming on a device with tight memory | GRU or LSTM, quantized; a state-space model if sequences are long. |
| Offline tagging of complete sequences | Bidirectional recurrent encoder or a fine-tuned encoder-only Transformer. |
| Large-scale language modeling | Decoder-only Transformer, possibly hybridized with state-space layers for long context. |
| Very long sequences (audio, genomics, logs) | State-space or linear-recurrent models, or Transformers with chunked or sparse attention. |
Whatever the architecture, evaluate with chronological splits for temporal data, report latency and memory at the sequence lengths you will serve, and check that the model actually uses long context (for example by truncating history and seeing whether accuracy drops) before paying for it.