Activation functions are the small non-linear step inside every neuron that lets neural networks learn more than straight lines. This page explains why they are needed, the math and shape of each common choice, how their derivatives govern trainability, and which to use where.
A neuron computes a weighted sum of its inputs plus a bias, z = w·x + b, then passes z through an activation function a = f(z). The activation is the only non-linear step. As the diagram notes, without it a stack of layers is mathematically equivalent to a single linear layer, no matter how deep, so the network could only learn linear decision boundaries.
With a non-linear f, a network with enough hidden units can approximate any continuous function on a bounded domain (the universal approximation theorem). In practice the specific choice of f matters less for what a network can represent than for how easily gradient descent can train it.
Training uses backpropagation: the gradient of the loss with respect to an early weight is a product of terms, one per later layer, and each term includes the activation's derivative f'(z). Three properties of f therefore decide whether deep training works:
f' is mostly much smaller than 1, gradients shrink exponentially with depth (vanishing gradients). If the effective factors exceed 1, they can explode.f' is near zero and the neuron stops learning.Cost also matters: an activation is evaluated for every unit of every example, so cheap functions such as ReLU are attractive.
Sigmoid: σ(z) = 1 / (1 + e−z), range (0, 1). Its derivative is σ(z)(1 − σ(z)), which peaks at 0.25 and approaches 0 for large |z|. Historically popular because it resembles a firing rate; today it is used mainly at outputs for binary probabilities and inside gates (LSTM, GRU).
Tanh: tanh(z) = (ez − e−z) / (ez + e−z) = 2σ(2z) − 1, range (−1, 1). Zero-centred with derivative up to 1, so it trains better than sigmoid, but still saturates. Common in RNN hidden states.
ReLU: max(0, z). Derivative is 1 for positive inputs and 0 for negative ones. It does not saturate on the positive side and is very cheap, which is a large part of why deep networks became trainable around 2010–2012. Its weakness is "dying ReLU": a unit whose input is negative for every example gets zero gradient and never recovers.
The ReLU family fixes that. Leaky ReLU uses max(αz, z) with small α such as 0.01; PReLU learns α; ELU uses α(ez − 1) for negative z, giving smooth, roughly zero-mean outputs.
GELU: z · Φ(z), where Φ is the standard normal CDF. It weights the input by the probability it is "kept", giving a smooth curve with a small negative dip. It is the default in BERT and GPT-style Transformers. SiLU / Swish (z · σ(z)) is a close relative, and gated variants such as SwiGLU are used in many recent LLM feed-forward blocks.
Softmax is different: it acts on a whole vector, softmax(z)i = ezi / Σj ezj, producing positive values that sum to 1. It belongs at the output of a multi-class classifier and inside attention, not in hidden layers.
The derivative plots explain the history. Through 10 sigmoid layers the activation contributes a factor of at most 0.2510, about one in a million, to the gradient reaching layer one (weights can partially compensate, but rarely enough). Tanh is better near zero but still collapses once inputs saturate. ReLU passes gradient through unchanged wherever a unit is active, which is why it, together with careful initialisation (He/Kaiming init for ReLU, Xavier/Glorot for tanh), normalisation layers and residual connections, made very deep networks practical.
import torch, torch.nn as nn, torch.nn.functional as F
x = torch.linspace(-3, 3, 7)
print("sigmoid", torch.sigmoid(x))
print("tanh ", torch.tanh(x))
print("relu ", F.relu(x))
print("leaky ", F.leaky_relu(x, negative_slope=0.01))
print("gelu ", F.gelu(x))
print("silu ", F.silu(x)) # x * sigmoid(x), a.k.a. Swish
# Softmax turns logits into a probability distribution over classes.
logits = torch.tensor([2.0, 1.0, 0.1])
print("softmax", F.softmax(logits, dim=-1)) # approx [0.659, 0.242, 0.099]
# Gradient magnitude through 20 stacked layers: sigmoid vs ReLU.
def grad_norm(act, depth=20, width=64):
torch.manual_seed(0)
layers = []
for _ in range(depth):
lin = nn.Linear(width, width)
nn.init.kaiming_normal_(lin.weight) if act is nn.ReLU else nn.init.xavier_normal_(lin.weight)
layers += [lin, act()]
net = nn.Sequential(*layers)
inp = torch.randn(32, width)
net(inp).sum().backward()
return net[0].weight.grad.norm().item()
print("first-layer grad, sigmoid:", grad_norm(nn.Sigmoid))
print("first-layer grad, relu :", grad_norm(nn.ReLU))The first block prints each function on the same inputs so you can compare shapes directly; note GELU and SiLU are slightly negative around z = −1 rather than zero. The softmax line shows logits [2, 1, 0.1] becoming probabilities of roughly 0.66, 0.24 and 0.10. The last part builds 20-layer networks and measures the gradient norm at the first layer; on a typical run the sigmoid network's gradient is many orders of magnitude smaller than the ReLU network's, a direct demonstration of vanishing gradients. Exact values depend on the seed and initialisation.
| Function | Range | Zero-centred | Saturates | Typical use |
|---|---|---|---|---|
| Sigmoid | (0, 1) | No | Both sides | Binary output, gates |
| Tanh | (−1, 1) | Yes | Both sides | RNN states, small nets |
| ReLU | [0, ∞) | No | Negative side (gradient 0) | Default for CNNs and MLPs |
| Leaky ReLU / PReLU | (−∞, ∞) | Roughly | No | When dead units are a problem |
| ELU | (−α, ∞) | Roughly | Negative side, softly | Alternative to ReLU |
| GELU / SiLU | about (−0.17, ∞) / (−0.28, ∞) | Roughly | No (smooth) | Transformers, modern LLMs |
| Softmax | (0, 1), sums to 1 | n/a | Yes, for large gaps | Multi-class output, attention |
Rule of thumb: ReLU for convolutional and plain feed-forward nets, GELU or SiLU-based gating for Transformers, sigmoid or softmax only at the output to match the loss function.
CrossEntropyLoss expects raw logits and applies log-softmax internally. Adding a softmax layer before it silently hurts training.BCEWithLogitsLoss, which is numerically stable for large logits.exp(z − max(z)); library functions already do this.