A stacked (deep) LSTM places several LSTM layers on top of one another: the full output sequence of layer 1 becomes the input sequence of layer 2, and so on. Depth in this sense is vertical — separate from the horizontal depth that comes from unrolling over time. Stacking lets lower layers capture local, short-range patterns and higher layers build more abstract, longer-range representations, the same hierarchy that depth provides in CNNs. Production systems such as Google's 2016 neural machine translation model used 8-layer stacked LSTMs with residual connections.
Each layer is a complete LSTM with its own weights and its own hidden and cell states. Layer l at time t receives two inputs: its own previous state (hlt−1, clt−1) from the left and the hidden state hl−1t of the layer below. Only h moves upward; each layer's cell state c stays private to that layer. The top layer's outputs feed a prediction head, either at every step (tagging, sequence-to-sequence) or only at the last step (classification, one-shot forecasting).
With h0t = xt, each layer applies the standard LSTM update:
# for layer l = 1..L, time t = 1..T
i = sigmoid(W_i @ h[l-1][t] + U_i @ h[l][t-1] + b_i) # input gate
f = sigmoid(W_f @ h[l-1][t] + U_f @ h[l][t-1] + b_f) # forget gate
o = sigmoid(W_o @ h[l-1][t] + U_o @ h[l][t-1] + b_o) # output gate
g = tanh (W_g @ h[l-1][t] + U_g @ h[l][t-1] + b_g) # candidate
c[l][t] = f * c[l][t-1] + i * g
h[l][t] = o * tanh(c[l][t])Only the first layer's input matrices are sized to the input dimension; every higher layer takes the hidden size (or 2×hidden if the layers are bidirectional). Per layer, parameters are 4(dh·din + dh2 + dh) (PyTorch keeps two bias vectors, adding another 4dh). In PyTorch, nn.LSTM(..., num_layers=L) returns output of shape (B, T, dh) from the top layer only, and h_n, c_n of shape (L, B, dh) with the final state of every layer.
The shape above is a qualitative pattern, not a benchmark. In practice, going from one to two layers usually helps on non-trivial tasks; two to four layers is the common range; deeper plain stacks become harder to train because gradients must now traverse both time and depth. More data and stronger regularization shift the optimum deeper. For a new problem, start with two layers and tune hidden size first — width is often a cheaper lever than depth.
Two techniques make deeper stacks practical. Residual connections add each layer's input to its output, as in ResNet; they give gradients a short path through depth and let extra layers start near the identity. Inter-layer dropout (Zaremba et al., 2014) regularizes the vertical connections only; applying ordinary dropout to the recurrent connections disrupts the memory the cell is supposed to keep. PyTorch's dropout= argument on nn.LSTM implements exactly the inter-layer version (and is ignored for a single layer).
Built-in stacking is the fastest option (it uses fused cuDNN kernels). A manual stack with residuals is easy when you need it.
import torch
import torch.nn as nn
# 1) Built-in: 3 layers, dropout between layers, multi-step forecaster
class Forecaster(nn.Module):
def __init__(self, n_features, d_h=128, layers=3, horizon=24):
super().__init__()
self.lstm = nn.LSTM(n_features, d_h, num_layers=layers,
dropout=0.2, batch_first=True)
self.head = nn.Linear(d_h, horizon)
def forward(self, x): # x: (B, T, n_features)
out, (h_n, c_n) = self.lstm(x) # out: top layer only; h_n: (layers, B, d_h)
return self.head(h_n[-1]) # top layer, final step
# 2) Manual residual stack
class ResidualLSTM(nn.Module):
def __init__(self, d_in, d_h=128, layers=4, p=0.2):
super().__init__()
self.proj = nn.Linear(d_in, d_h) # match dims for the shortcut
self.layers = nn.ModuleList(nn.LSTM(d_h, d_h, batch_first=True) for _ in range(layers))
self.norms = nn.ModuleList(nn.LayerNorm(d_h) for _ in range(layers))
self.drop = nn.Dropout(p)
def forward(self, x):
h = self.proj(x)
for lstm, norm in zip(self.layers, self.norms):
out, _ = lstm(self.drop(h))
h = norm(h + out) # residual + LayerNorm
return h # (B, T, d_h)
x = torch.randn(32, 96, 7) # 96 past steps, 7 features
print(Forecaster(7)(x).shape, ResidualLSTM(7)(x).shape) # (32, 24) (32, 96, 128)# Keras: every layer except the last must return the full sequence
from tensorflow import keras
from tensorflow.keras import layers
model = keras.Sequential([
keras.Input(shape=(96, 7)),
layers.LSTM(128, return_sequences=True, dropout=0.2),
layers.LSTM(128, return_sequences=True, dropout=0.2),
layers.LSTM(64), # last layer: final state only
layers.Dense(24),
])
model.compile(optimizer="adam", loss="mse")8dh2 parameters; on small datasets a deeper stack mainly overfits.l at step t while layer l+1 handles t−1) helps on multi-GPU setups.return_sequences=False on an intermediate layer — the next LSTM receives a 2-D tensor and errors, or a RepeatVector hack hides the bug.h_n[0] — that is the bottom layer; the top layer is h_n[-1].(h_0, c_0) they must be (num_layers × num_directions, B, d_h).Take a three-layer stack with input size 32 and hidden size 128. Using the per-layer formula 4(dh·din + dh2 + dh):
sum(p.numel() for p in lstm.parameters()) reports 347,136.Compare a single layer with hidden size 256 on the same input: 4(256·32 + 256² + 256) = 295,936 (plus 1,024 for the extra bias). A one-layer, twice-as-wide LSTM has roughly the same parameter count as the three-layer stack. That makes it a fair baseline: if the deep stack does not beat the wide single layer at equal parameters, the depth is not earning its extra sequential latency.
For a batch of 64 sequences, the returned states h_n and c_n each have shape (3, 64, 128) — one slice per layer — while output is (64, T, 128) from the top layer only. If you need every layer's sequence of outputs (for example to feed a weighted combination of layers to a head, as ELMo did), you must build the stack from separate single-layer modules.
The best-known use of deep LSTM stacks was neural machine translation. Sutskever, Vinyals and Le (2014) used four-layer LSTMs for both encoder and decoder, and Google's production NMT system (Wu et al., 2016) used eight encoder and eight decoder layers with residual connections between layers — without the residuals, the authors reported that stacks beyond about four layers trained poorly. The usual wiring passes the final (h, c) of each encoder layer to the matching decoder layer, which requires the two stacks to have the same depth and hidden size (or a projection between them). Attention over the encoder's top-layer outputs was added on top of this, and later the transformer encoder-decoder replaced the recurrence altogether.
| Model | Parallel over time? | Long-range memory | Typical fit |
|---|---|---|---|
| Single-layer LSTM | No | Good | Small data, simple sequences |
| Stacked LSTM (2–4 layers) | No (pipelining across layers only) | Good, with more hierarchical features | Speech, handwriting, moderate-size sequence tasks |
| Stacked GRU | No | Similar on many tasks | Same as LSTM with about 25% fewer parameters |
| Bidirectional stack | No | Sees both directions | Tagging and classification when the whole sequence is available |
| Temporal convolution (TCN) | Yes | Limited by receptive field | Fast forecasting and audio |
| Transformer | Yes (training) | Direct access to all positions | Large data, long contexts |
Stacked LSTMs remain reasonable choices where data are modest, sequences are streaming and must be processed step by step with constant memory, or deployment targets are small devices: on-device keyword spotting, sensor and time-series forecasting, and online handwriting recognition are typical examples.
unit_forget_bias=True; PyTorch does not), which helps early training retain memory (Jozefowicz et al., 2015).pack_padded_sequence) so padding does not update states, or mask it in Keras.