GRU (Gated Recurrent Unit)

The Gated Recurrent Unit, introduced by Cho et al. in 2014, is a simplified gated recurrent cell. Like the LSTM, it solves the vanishing-gradient problem of the vanilla RNN with learned gates that allow information to be copied forward almost unchanged. Unlike the LSTM, it has two gates instead of three and no separate cell state, so it has about 25% fewer parameters per unit. Comparative studies (e.g. Chung et al., 2014; Jozefowicz et al., 2015) found GRU and LSTM broadly comparable, with the winner depending on the task.

1. The GRU Cell

GRU Cell: Two Gates, One State h(t-1) x(t) Reset gate rsigmoid Update gate zsigmoid × Candidate h~tanh(W x + U(r⊙h)) Interpolate(1-z)⊙h + z⊙h~ h(t) Yellow: previous state paths

Given input xt and previous state ht−1:

z_t  = sigmoid(W_z x_t + U_z h_{t-1} + b_z)            # update gate
r_t  = sigmoid(W_r x_t + U_r h_{t-1} + b_r)            # reset gate
h~_t = tanh(W_h x_t + U_h (r_t * h_{t-1}) + b_h)       # candidate state
h_t  = (1 - z_t) * h_{t-1} + z_t * h~_t                # interpolate

The new state is a per-dimension convex combination of the old state and a candidate. There is no output gate and no separate memory cell: h is both the memory and the output. Parameter count per layer is 3(dh·din + dh2 + dh), versus 4 blocks for an LSTM. (Conventions differ: some papers and Keras' docs swap the roles of z and 1−z; the model is equivalent. PyTorch and cuDNN apply the reset gate after the matrix multiply, r ⊙ (Uhh + b), a slight variant of the original.)

2. What the Gates Do

Trained GRUs tend to develop a mix of units: some with update gates mostly closed (slow, long memory) and some mostly open (fast, local features).

3. Why It Fixes Vanishing Gradients

Backward Path Through One Step Vanilla RNN h(t) h(t-1) x diag(1-h²) U always multiplied: decays GRU h(t) h(t-1) x diag(1 - z) + ... z near 0: near identity The additive (1-z) h(t-1) term gives a gated shortcut through time, like a residual connection.

Differentiate the final line: ∂ht/∂ht−1 contains the direct term diag(1 − zt) plus terms through the gates and candidate. When the network learns to keep z small for a unit, that Jacobian is close to the identity and the gradient passes back across many steps without exponential decay. This is the same principle as the LSTM's additive cell update and as residual connections in deep feedforward networks. GRUs still benefit from gradient clipping, because the non-identity terms can grow.

4. GRU vs LSTM

GRU vs LSTM at a Glance GRU Gates: update z, reset r (2) State: h only Weight blocks: 3 (about 3/4 of LSTM) Output exposes full memory LSTM Gates: input, forget, output (3) State: h and cell c Weight blocks: 4 Output gate can hide memory

The GRU is cheaper per step and has fewer hyperparameter interactions, which often makes it train a little faster and do well on smaller datasets. The LSTM's separate, output-gated cell gives it slightly more expressive control; it has been shown to count more reliably on some formal-language tasks (Weiss et al., 2018), and it remains the more common choice in older production systems and literature. In practice the difference on real tasks is usually small compared with the effect of tuning hidden size, regularization and learning rate — try both if it matters.

5. Python Examples

A GRU cell from scratch (to make the equations concrete), checked against PyTorch's layer, followed by a typical time-series model.

import torch
import torch.nn as nn

class MyGRUCell(nn.Module):
    def __init__(self, d_in, d_h):
        super().__init__()
        self.x2g = nn.Linear(d_in, 3 * d_h)            # W_z, W_r, W_h stacked
        self.h2g = nn.Linear(d_h, 3 * d_h)             # U_z, U_r, U_h stacked
    def forward(self, x, h):
        xr, xz, xn = self.x2g(x).chunk(3, -1)
        hr, hz, hn = self.h2g(h).chunk(3, -1)
        r = torch.sigmoid(xr + hr)
        z = torch.sigmoid(xz + hz)
        n = torch.tanh(xn + r * hn)                     # PyTorch-style reset placement
        return (1 - z) * n + z * h                      # PyTorch: z gates the OLD state

# Compare with nn.GRUCell using identical weights (gate order r, z, n)
torch.manual_seed(0)
ref, mine = nn.GRUCell(4, 8), MyGRUCell(4, 8)
with torch.no_grad():
    mine.x2g.weight.copy_(ref.weight_ih); mine.x2g.bias.copy_(ref.bias_ih)
    mine.h2g.weight.copy_(ref.weight_hh); mine.h2g.bias.copy_(ref.bias_hh)
x, h = torch.randn(2, 4), torch.randn(2, 8)
print(torch.allclose(ref(x, h), mine(x, h), atol=1e-6))   # True

# Typical sequence model: 2-layer GRU forecaster
class GRUForecaster(nn.Module):
    def __init__(self, n_feat, d_h=64, horizon=12):
        super().__init__()
        self.gru = nn.GRU(n_feat, d_h, num_layers=2, dropout=0.1, batch_first=True)
        self.head = nn.Linear(d_h, horizon)
    def forward(self, x):
        _, h_n = self.gru(x)                           # GRU returns h_n only (no c_n)
        return self.head(h_n[-1])

model = GRUForecaster(n_feat=5)
print(model(torch.randn(16, 48, 5)).shape)             # torch.Size([16, 12])
print(sum(p.numel() for p in nn.GRU(64, 64).parameters()),
      sum(p.numel() for p in nn.LSTM(64, 64).parameters()))   # 24960 33280

Note the convention in this code: PyTorch writes ht = (1−z)⊙n + z⊙ht−1, so there z near 1 means "keep". It is the same model as Section 1 with z relabeled.

6. When to Use a GRU

For long-context language tasks with abundant data, Transformers are usually the better choice.

7. Common Pitfalls

8. Unrolling and a Worked Step

Unrolled GRU: One Cell, Shared Weights, Carried State GRU cell t = 1 GRU cell t = 2 GRU cell t = 3 GRU cell t = 4 x(1) x(2) x(3) x(4) Head linear / softmax Blue arrows carry h(t). The same W and U matrices are reused at every step; only the state changes.

Unrolling makes the cost model clear: a sequence of length T needs T sequential cell evaluations, each doing three matrix-vector products against the input and three against the previous state. The input products for all steps can be batched up front; the state products cannot, which is why recurrent models are latency-bound on long sequences no matter how fast the GPU is.

A single scalar step with hand-picked numbers shows the gates at work (Section 1 convention, where z near 1 means "overwrite"). Take ht−1 = 0.5, xt = 1.0, candidate weights Wh = Uh = 1, zero biases, and a reset pre-activation of 0 so r = σ(0) = 0.5.

The same numbers explain gradient flow. The direct Jacobian term per step is 1 − z. With z = 0.12 that is 0.88, and 0.8850 is roughly 0.002 — still a fast decay. With z = 0.01 it is 0.99, and 0.9950 is about 0.6. Long memory therefore requires units whose update gates sit very close to "keep", which is exactly what bias initialization tricks (below) encourage.

9. Hyperparameters and Variants

ChoiceTypical rangeGuidance
Hidden size32–512Start small for tabular or sensor series; grow only if validation loss is still falling.
Layers1–3A second layer often helps; beyond three, gains are rare and training slows.
Dropout0.1–0.3 between layersPyTorch's dropout argument applies only between stacked layers, never on the last.
Gradient clippingmax norm around 1–5Cheap insurance against the occasional exploding batch.
OptimizerAdam / AdamW, lr near 1e-3Lower the rate if loss spikes; pair with a plateau or cosine schedule.
Update-gate biasshift toward "keep"Biasing the gate so units start out copying state (analogous to the LSTM forget-gate bias of 1) helps long dependencies.

Variants. The original formulation applies the reset gate to the previous state before multiplying by Uh. The cuDNN-friendly variant applies it after the multiplication, r ⊙ (Uhh + b); this is what PyTorch implements and what Keras calls reset_after=True. The two are not weight-compatible, which is the root of most porting bugs. Other variants include the Minimal Gated Unit (a single gate), layer-normalized GRUs for stability on long sequences, and convolutional GRUs that replace the matrix products with convolutions for spatio-temporal data such as radar or video frames.

10. Training and Production Notes

11. Further Reading