Bidirectional LSTM

A bidirectional LSTM (BiLSTM) runs two independent LSTMs over the same sequence — one left-to-right, one right-to-left — and combines their hidden states at each position. Every output therefore depends on the entire input, past and future. Before Transformers, BiLSTMs were the default encoder for named-entity recognition, part-of-speech tagging, speech recognition and the encoder side of neural machine translation, and ELMo's contextual embeddings were built from them. The idea comes from bidirectional RNNs (Schuster and Paliwal, 1997).

1. Architecture

The forward LSTM computes →ht = LSTMf(xt, →ht−1) and the backward LSTM computes ←ht = LSTMb(xt, ←ht+1). The two have separate parameters and do not interact inside the layer; their states are combined only at the output, usually by concatenation ht = [→ht; ←ht], giving a vector of size 2dh. Parameter count and compute are exactly double a unidirectional LSTM of the same hidden size, but the two directions can run in parallel.

2. Why Future Context Helps

Many labeling decisions depend on what comes later: whether "Washington" is a person or a place, which phoneme a sound belongs to given the following frames, whether a token begins a multi-word entity. A unidirectional model must commit using only the prefix; a BiLSTM sees the full sentence at every position. Empirically this gives consistent gains on tagging and recognition tasks wherever the full input is available before predictions are needed.

3. Combining the Two Directions

• Concatenate (default in PyTorch and Keras): preserves both signals, doubles the output dimension.
• Sum or average: keeps dimension at dh; Keras exposes this via Bidirectional(..., merge_mode="sum").
• Sequence-level summary: for classification, use the forward direction's last state and the backward direction's first-position state (which is the final state of the backward pass), or pool over all positions. A common bug is taking output[:, -1], which gives the backward direction only one token of context.

4. BiLSTM-CRF for Sequence Labeling

A BiLSTM classifies each token independently given context, which can produce invalid label sequences (an I-ORG right after O). Adding a linear-chain Conditional Random Field on top scores whole tag sequences with learned transition weights, trains with the sequence log-likelihood (forward algorithm) and decodes the best path with Viterbi. BiLSTM-CRF with character-level features (Lample et al., 2016; Ma and Hovy, 2016) was the standard NER architecture before fine-tuned Transformer encoders such as BERT.

5. PyTorch Example

import torch
import torch.nn as nn
from torch.nn.utils.rnn import pack_padded_sequence, pad_packed_sequence

class BiLSTMTagger(nn.Module):
    """Per-token tagger (many-to-many) plus a sentence-level head (many-to-one)."""
    def __init__(self, vocab, n_tags, n_cls, d_emb=100, d_h=128):
        super().__init__()
        self.emb = nn.Embedding(vocab, d_emb, padding_idx=0)
        self.lstm = nn.LSTM(d_emb, d_h, batch_first=True, bidirectional=True)
        self.tag_head = nn.Linear(2 * d_h, n_tags)
        self.cls_head = nn.Linear(2 * d_h, n_cls)

    def forward(self, tokens, lengths):
        packed = pack_padded_sequence(self.emb(tokens), lengths.cpu(),
                                      batch_first=True, enforce_sorted=False)
        out, (h_n, _) = self.lstm(packed)
        out, _ = pad_packed_sequence(out, batch_first=True)    # (B, T, 2*d_h)
        tag_logits = self.tag_head(out)
        # h_n: (num_layers*2, B, d_h). Index -2 = last layer forward, -1 = last layer backward
        sent = torch.cat([h_n[-2], h_n[-1]], dim=-1)          # (B, 2*d_h)
        return tag_logits, self.cls_head(sent)

model = BiLSTMTagger(vocab=5000, n_tags=9, n_cls=3)
tokens = torch.randint(1, 5000, (4, 12)); lengths = torch.tensor([12, 9, 7, 3])
for i, L in enumerate(lengths): tokens[i, L:] = 0
tags, cls = model(tokens, lengths)
print(tags.shape, cls.shape)          # (4, 12, 9) (4, 3)

# token loss must ignore padding positions
gold = torch.randint(0, 9, (4, 12)); gold[tokens == 0] = -100
loss = nn.functional.cross_entropy(tags.reshape(-1, 9), gold.reshape(-1), ignore_index=-100)

In Keras the equivalent is layers.Bidirectional(layers.LSTM(128, return_sequences=True)). A CRF layer is available in packages such as pytorch-crf.

6. When Not to Use It

• Streaming or real-time prediction. The backward pass needs the whole sequence, so a BiLSTM cannot emit an output until the input ends. Use a unidirectional model, or a limited lookahead window.
• Forecasting and language modeling. Predicting the next value with access to future values is leakage, not context.
• Large-scale NLP. Pretrained Transformer encoders are bidirectional by construction (full self-attention) and usually stronger; a BiLSTM is still a fast, small alternative when compute or data is limited.

7. Common Pitfalls

• Wrong final state. Using output[:, -1, :] with padding or for the backward direction; use h_n as shown.
• Padding inside the backward pass. Without packing, the backward LSTM starts on pad tokens. Always pack variable-length batches.
• Dimension mismatch. Downstream layers must expect 2×hidden_size.
• Time-series leakage. Using a BiLSTM on a sliding window whose labels are in the future of the window end.

8. Worked Example: Shapes and Parameters

Consider a tagger with a 100-dimensional embedding, two BiLSTM layers with hidden size 128 per direction, and 9 output tags (BIO tags for four entity types plus O). Each LSTM direction has four gates, each with input weights, recurrent weights and biases, giving 4·(d_in·d_h + d_h² + d_h) parameters, plus a second bias vector in PyTorch's implementation. For layer 1, one direction has 4·(100·128 + 128·128 + 2·128) = 117,760 parameters, and both directions 235,520. Layer 2 receives the 256-dimensional concatenation, so one direction is 4·(256·128 + 128·128 + 256) = 197,632, both 395,264. The output layer adds 256·9 + 9 = 2,313. The whole recurrent part is well under a million parameters, which is why BiLSTMs remain attractive for small, fast on-device taggers.

A frequent bug is using h_n[-1] as the sentence vector: in a bidirectional model the last two entries of h_n are the top layer's forward and backward states, and both are needed. For sequence classification, concatenate h_n[-2] and h_n[-1], or pool over the output sequence.

9. Training Tips

• Variable lengths: use pack_padded_sequence so the backward LSTM starts at each sequence's real last token, not at padding. Without packing, the backward direction reads padding first and its states are polluted.
• Dropout: apply dropout to embeddings and between layers (the dropout argument only acts between stacked layers). Variational dropout, reusing the same mask at every step, works well for recurrent connections.
• Character features: for NER, a small character-level CNN or BiLSTM per word handles rare and unseen words, capitalization and suffixes (Lample et al., 2016).
• Gradient clipping at a global norm around 1–5 prevents occasional exploding updates.
• Optimizer: Adam with a learning rate near 1e-3 is a sensible default; SGD with a decaying rate was common in older NER papers.

10. ELMo: BiLSTMs as Contextual Embeddings

ELMo (Peters et al., 2018) trained a forward and a backward LSTM language model on a large corpus and used the hidden states as contextual word embeddings. Note that ELMo's two directions are trained as separate language models, so neither direction sees the token it predicts. Downstream tasks learn a softmax-normalized weighted sum of all layers plus a scale. Analysis in the paper found lower layers more useful for syntax and higher layers for word sense. Adding ELMo improved the state of the art across question answering, NER, sentiment and other tasks at the time, and it was the direct precursor of BERT, which replaced the two shallow concatenated LSTMs with a deep bidirectional Transformer trained with masking.

11. BiLSTM vs Transformer Encoder

Choose a BiLSTM when the model must be trained from scratch on modest data, run on constrained hardware, or process long sequences such as sensor streams where quadratic attention is costly. Otherwise a fine-tuned pretrained encoder is usually more accurate.

12. Further Reading

• Schuster, M. & Paliwal, K. (1997). Bidirectional recurrent neural networks.
• Graves, A. & Schmidhuber, J. (2005). Framewise phoneme classification with bidirectional LSTM and other neural network architectures.
• Huang, Z., Xu, W. & Yu, K. (2015). Bidirectional LSTM-CRF models for sequence tagging. arXiv:1508.01991
• Lample, G. et al. (2016). Neural architectures for named entity recognition.
• Peters, M. et al. (2018). Deep contextualized word representations (ELMo). arXiv:1802.05365

13. Related Topics

• LSTM (Long Short-Term Memory)
• Stacked LSTM — deep BiLSTMs stack bidirectional layers
• Recurrent Neural Networks (RNN)
• GRU (Gated Recurrent Unit) — also works bidirectionally
• Deep Learning (DL)
• Self-Attention