Two small layers that make deep networks trainable and less prone to overfitting. This page explains batch, layer and RMS normalization, dropout, how each behaves differently in training and inference, and where they go in real architectures.
Normalization layers and dropout are the two most widely used ingredients that make deep networks train reliably and generalise. They solve different problems. Normalization rescales activations inside the network so that each layer sees inputs with a stable scale, which makes optimisation faster and less sensitive to learning rate and initialisation. Dropout randomly switches off units during training so the network cannot rely on any single path, which reduces overfitting.
Both are a few lines in any framework, but they behave differently in training and inference mode, and getting that wrong is one of the most common bugs in deep-learning code. This page builds both from first principles and shows where each belongs in CNNs and transformers.
Before batch normalization (Ioffe and Szegedy, 2015), training very deep networks required careful initialisation and small learning rates. BatchNorm let image models train with much higher learning rates and fewer steps, and became standard in CNNs. Layer normalization (Ba, Kiros and Hinton, 2016) works per example, so it suits sequence models and small or variable batches; every mainstream transformer uses LayerNorm or its simplification RMSNorm. Dropout (Srivastava et al., 2014) was a key regulariser in the networks that won image and speech benchmarks in the early 2010s, and it remains standard in fine-tuning and in smaller models. Large language model pre-training on huge datasets often uses little or no dropout, because the model sees each example about once and overfitting is less of a concern.
For a mini-batch of activations, BatchNorm computes, for each feature (each channel in a CNN), the mean μB and variance σB2 over the batch (and spatial positions), then normalises and applies a learnable scale and shift:
x̂ = (x − μB) / √(σB2 + ε) y = γ x̂ + β
A tiny numeric example: one feature takes the values 2, 4, 6 and 8 across a batch of four. The mean is 5 and the (population) variance is 5, so √5 ≈ 2.24 and the normalised values are about −1.34, −0.45, 0.45 and 1.34. With γ = 1 and β = 0 these pass straight to the next layer: centred on zero with unit variance, whatever the original scale was. If the same feature had been measured in thousands, the output would be identical.
γ and β let the network undo the normalisation if that is useful. During training the layer also keeps running averages of mean and variance; at inference it uses those stored values, so a single example gives a deterministic output. The original motivation was reducing "internal covariate shift"; later work (Santurkar et al., 2018) argued the benefit comes mainly from smoothing the loss landscape. Either way, the practical effect is well established.
LayerNorm uses the same formula but computes μ and σ over the features of each single example. It does not depend on the batch, so it behaves identically in training and inference and works with batch size 1 and variable sequence lengths. RMSNorm (Zhang and Sennrich, 2019) drops the mean subtraction and divides by the root mean square only, which is slightly cheaper and is used in models such as LLaMA. GroupNorm and InstanceNorm are intermediate options for vision with tiny batches.
During training each unit is kept with probability 1 − p and set to zero otherwise, with a fresh random mask for each example and step. Modern frameworks implement inverted dropout: surviving activations are multiplied by 1/(1 − p), so the expected value is unchanged and the layer becomes an identity at inference. Dropout can be seen as training an ensemble of many thinned sub-networks that share weights, and averaging them at test time. Typical rates are 0.1 in transformers and 0.2 to 0.5 in fully connected layers.
In CNNs, BatchNorm usually sits between convolution and activation; the convolution's bias becomes redundant because β replaces it. Transformers apply LayerNorm before each attention and feed-forward sub-layer (pre-LN, which is more stable for deep stacks than the original post-LN arrangement) and dropout on the sub-layer output before the residual addition, plus on attention weights.
The PyTorch snippet below shows the train/eval difference concretely, then trains a small model with both layers.
import torch, torch.nn as nn
torch.manual_seed(0)
x = torch.randn(8, 4) * 5 + 3 # batch of 8, 4 features, far from mean 0 / std 1
bn, ln = nn.BatchNorm1d(4), nn.LayerNorm(4)
print("BN per-feature mean:", bn(x).mean(0).detach().round(decimals=3)) # ~0 per column
print("LN per-row mean: ", ln(x).mean(1).detach().round(decimals=3)) # ~0 per row
drop = nn.Dropout(p=0.5)
ones = torch.ones(1, 10)
drop.train(); print("train:", drop(ones)) # zeros and 2.0s (scaled by 1/(1-p))
drop.eval(); print("eval: ", drop(ones)) # all ones: identity
model = nn.Sequential(
nn.Linear(20, 64), nn.BatchNorm1d(64), nn.ReLU(), nn.Dropout(0.3),
nn.Linear(64, 2))
opt = torch.optim.AdamW(model.parameters(), lr=1e-2)
X = torch.randn(512, 20); y = (X[:, 0] + X[:, 1] > 0).long()
for step in range(200):
model.train() # enable dropout, use batch statistics
loss = nn.functional.cross_entropy(model(X), y)
opt.zero_grad(); loss.backward(); opt.step()
model.eval() # disable dropout, use running statistics
with torch.no_grad():
acc = (model(X).argmax(1) == y).float().mean()
print(f"loss {loss.item():.3f} train acc {acc:.3f}")Forgetting model.eval() before evaluation is the classic bug: predictions become random (dropout) and depend on the composition of the test batch (BatchNorm).
| Technique | Normalises over | Batch dependent? | Typical use | Main caveat |
|---|---|---|---|---|
| BatchNorm | Batch (and spatial) per feature | Yes | CNNs, MLPs with large batches | Breaks with tiny batches; train/eval mismatch |
| LayerNorm | Features per example | No | Transformers, RNNs | Slightly more compute than RMSNorm |
| RMSNorm | Features per example, no mean | No | Recent LLMs | No re-centring |
| GroupNorm | Groups of channels per example | No | Detection, segmentation with small batches | Choose group count |
| Dropout | (regulariser, not a normaliser) | No | Fine-tuning, MLP heads, transformers | Slows convergence; must be off at eval |
| Weight decay | (regularises weights) | No | Almost everywhere | Interacts with normalisation scale |
model.eval() for inference and model.train() again afterwards.