Backpropagation is how neural networks learn which weights to blame for an error. This page derives it from the chain rule, walks through every number in a small network, and checks the result against PyTorch.
Backpropagation is the algorithm that computes how much each weight in a neural network contributed to the error, so that an optimizer can adjust it. Formally, it computes the gradient of the loss with respect to every parameter, ∂L/∂w, in one backward sweep through the network, at a cost comparable to the forward pass. It is an application of the chain rule from calculus organised so that no intermediate derivative is computed twice. In the wider literature the same technique is called reverse-mode automatic differentiation.
Backpropagation only computes gradients. Using them to change weights is the optimizer's job, covered on the Optimizers page.
A modern network has millions or billions of weights. Estimating each gradient by nudging one weight and re-running the network (finite differences) would need one forward pass per weight, which is hopeless at that scale. Backpropagation gets every gradient for roughly the price of two forward passes. That efficiency, popularised for neural networks by Rumelhart, Hinton and Williams in 1986, is what made training deep networks practical, and every framework — PyTorch, TensorFlow, JAX — implements it as automatic differentiation.
Understanding it is not optional for practitioners. Vanishing and exploding gradients, the choice of activation functions, residual connections, normalisation layers, gradient clipping and the memory cost of training all follow directly from how gradients flow backward.
Any computation can be drawn as a graph of simple operations. Take f = (a + b) × c with a = 2, b = 3, c = 4. The forward pass computes values left to right: q = a + b = 5, then f = q × c = 20.
The backward pass runs right to left. Start with ∂f/∂f = 1. A multiply node passes each input the upstream gradient times the other input: ∂f/∂q = c = 4 and ∂f/∂c = q = 5. An add node passes the upstream gradient unchanged to both inputs: ∂f/∂a = ∂f/∂q × 1 = 4, and likewise ∂f/∂b = 4. Every node only needs its local derivative and the gradient arriving from above:
∂L/∂input = ∂L/∂output × ∂output/∂input
When a value feeds several downstream nodes, the gradients from each path are summed. That local rule, applied in reverse topological order, is the whole algorithm.
In a real network the same rule is applied to whole layers at once. For a dense layer z = W h + b, the backward pass computes ∂L/∂W = δ hT, ∂L/∂b = δ and ∂L/∂h = WT δ, where δ = ∂L/∂z is the error signal arriving at that layer. With a mini-batch, these become matrix multiplications summed over the batch, which is why training runs so efficiently on GPUs.
The network has inputs x = (1.0, 0.5), two hidden units with sigmoid activation σ(z) = 1 / (1 + e−z), and one linear output. Hidden biases are 0, the output bias is c = 0.1, and the target is y = 1. Weights are shown on the arrows.
One gradient-descent step with learning rate 0.5 (w ← w − 0.5 × gradient) moves every weight in the direction that reduces the error. Re-running the forward pass gives ŷ = 0.919 and a loss of about 0.0033, down from 0.337. Note how the sigmoid derivative, at most 0.25, shrinks the gradients reaching the first layer: stack many such layers and gradients vanish, which is why ReLU activations and residual connections became standard.
The script repeats the example by hand in NumPy, then lets PyTorch's autograd compute the same gradients. The printed numbers match the derivation above.
import numpy as np, torch
x = np.array([1.0, 0.5]); y = 1.0
W = np.array([[0.4, -0.2], [0.3, 0.6]]); v = np.array([0.7, -0.5]); c = 0.1
sig = lambda z: 1 / (1 + np.exp(-z))
# forward
z = W @ x; h = sig(z); y_hat = v @ h + c; L = 0.5 * (y_hat - y) ** 2
# backward
d_yhat = y_hat - y
d_v, d_c = d_yhat * h, d_yhat
d_h = d_yhat * v
d_z = d_h * h * (1 - h)
d_W = np.outer(d_z, x)
print("manual loss", round(L, 4), "dW", d_W.round(4).tolist(), "dv", d_v.round(4).tolist())
# autograd
Wt = torch.tensor(W, requires_grad=True); vt = torch.tensor(v, requires_grad=True)
ct = torch.tensor(c, requires_grad=True); xt = torch.tensor(x)
Lt = 0.5 * (vt @ torch.sigmoid(Wt @ xt) + ct - y) ** 2
Lt.backward()
print("autograd loss", round(Lt.item(), 4), "dW", Wt.grad.numpy().round(4).tolist(),
"dv", vt.grad.numpy().round(4).tolist())
# gradient check by finite differences on one weight
eps = 1e-6; W2 = W.copy(); W2[0, 0] += eps
L2 = 0.5 * (v @ sig(W2 @ x) + c - y) ** 2
print("numeric dW[0,0]", round((L2 - L) / eps, 4))The final line is a gradient check: perturb one weight slightly and measure the change in loss. It costs one forward pass per weight, so it is only for testing, but it is the standard way to verify a hand-written backward pass.
| Method | Cost for n parameters | Accuracy | Use |
|---|---|---|---|
| Finite differences | n + 1 forward passes | Approximate, rounding error | Gradient checks only |
| Symbolic differentiation | Expression can blow up | Exact | Computer algebra, small formulas |
| Forward-mode autodiff | One pass per input direction | Exact | Few inputs, many outputs |
| Reverse-mode autodiff (backprop) | About one forward + one backward pass | Exact | Many parameters, scalar loss: neural networks |
.grad; zero gradients each step.argmax, rounding or sampling block gradient flow; use relaxations or estimators..detach() mid-model silently stops gradients.