Gradient descent is how almost every modern model learns: compute which way the loss goes up, step the other way, repeat. This page builds the update rule from intuition, shows why the learning rate matters, compares batch, stochastic and mini-batch variants, and implements it in NumPy.
Most machine learning models are trained by choosing parameters that make a loss function small. The loss measures how wrong the model is on the training data: mean squared error for regression, cross-entropy for classification. Gradient descent is the simplest general method for finding parameters with low loss.
The idea is the one in the picture: if you stand on a hillside in fog, you can feel which way the ground slopes under your feet. Take a step downhill, feel again, repeat. The gradient is the multi-dimensional version of slope; it points in the direction the loss increases fastest, so we step the other way.
The yellow dots are successive parameter values. Because the slope gets smaller near the bottom, the steps naturally get shorter even though the learning rate stays the same.
Linear and logistic regression, neural networks of every size, matrix factorisation for recommenders, and gradient-boosted trees (in function space) all rely on gradient-based optimisation. Large language models are trained with variants of the same update rule described here. Understanding the learning rate, batch size and convergence behaviour explains most training failures you will meet: loss that explodes to NaN, loss that plateaus, or a model that fits training data but generalises poorly.
Write the parameters as a vector θ and the loss as L(θ). The gradient ∇L(θ) is the vector of partial derivatives ∂L/∂θj. One step of gradient descent is:
θt+1 = θt − η · ∇L(θt)
where η (eta) is the learning rate. For training data of n examples, the loss is usually an average, L(θ) = (1/n) Σi ℓ(θ; xi, yi), so the gradient is also an average of per-example gradients. In neural networks, those per-example gradients are computed with backpropagation (reverse-mode automatic differentiation).
The learning rate controls everything. Too small and training crawls. Too large and each step overshoots the valley, landing higher up the opposite side, and the loss grows without bound. For a simple quadratic loss with curvature c the update is stable only when η < 2/c; in many dimensions the steepest direction sets the limit, while the flattest direction sets how slowly you converge. The ratio between them (the condition number) is why feature scaling matters so much: standardising features makes the valley rounder and lets one learning rate suit every direction.
Batch gradient descent computes the exact gradient over all n examples each step. It is smooth but each step costs a full pass over the data. Stochastic gradient descent (SGD) uses one random example per step: the gradient is a noisy but unbiased estimate, and you take n steps per pass instead of one. Mini-batch gradient descent averages over a small random batch, reducing noise while using vectorised hardware efficiently. In practice "SGD" in libraries almost always means mini-batch.
The noise in SGD is not only a cost. It can help escape shallow local minima and saddle points, and there is evidence that it biases training toward solutions that generalise well. Because the noise never fully disappears, SGD with a fixed learning rate bounces around the minimum; a decaying learning rate schedule (step decay, cosine, or linear warmup then decay) lets it settle.
Popular refinements: momentum accumulates a running average of gradients so consistent directions speed up and oscillations cancel; Nesterov momentum looks ahead before computing the gradient; AdaGrad, RMSProp and Adam keep per-parameter scales from past squared gradients, which effectively gives each parameter its own learning rate. AdamW decouples weight decay from the adaptive step and is a common default for transformers.
Fit a line y = w x + b by mini-batch gradient descent in plain NumPy. The loss is mean squared error, whose gradients are ∂L/∂w = (2/m) Σ (ŷ − y) x and ∂L/∂b = (2/m) Σ (ŷ − y).
import numpy as np
rng = np.random.default_rng(0)
n = 1_000
X = rng.uniform(-3, 3, n)
y = 2.5 * X - 1.0 + rng.normal(0, 0.5, n) # true w=2.5, b=-1.0
w, b = 0.0, 0.0
lr, batch, epochs = 0.05, 32, 20
for epoch in range(epochs):
idx = rng.permutation(n) # shuffle each epoch
for start in range(0, n, batch):
j = idx[start:start + batch]
err = (w * X[j] + b) - y[j]
grad_w = 2 * np.mean(err * X[j])
grad_b = 2 * np.mean(err)
w -= lr * grad_w
b -= lr * grad_b
loss = np.mean((w * X + b - y) ** 2)
if epoch % 5 == 0 or epoch == epochs - 1:
print(f"epoch {epoch:2d} loss {loss:.4f} w {w:.3f} b {b:.3f}")The printed w and b approach 2.5 and −1.0 and the loss approaches the noise variance (about 0.25). Try lr = 0.5: with features spread over [−3, 3] the curvature in w is large enough that the updates diverge, a direct demonstration of the stability limit above. In PyTorch the same loop is loss.backward() followed by optimizer.step() with torch.optim.SGD or torch.optim.AdamW.
For convex, smooth losses (linear regression, logistic regression) gradient descent with a small enough fixed learning rate converges to the global minimum. Neural network losses are non-convex: there is no guarantee of a global minimum, but in high dimensions most critical points are saddle points rather than poor local minima, and momentum plus noise move through them. In practice you judge convergence empirically: plot training and validation loss per epoch, stop when validation loss stops improving (early stopping), and keep the best checkpoint.
| Method | Per-step cost | Tuning | Typical use |
|---|---|---|---|
| Batch GD | Full dataset | Learning rate | Small convex problems, teaching |
| SGD / mini-batch | One batch | Learning rate, schedule, batch size | Large data, CNNs (often with momentum) |
| SGD + momentum | One batch | Plus momentum (about 0.9) | Vision models, well-tuned pipelines |
| Adam / AdamW | One batch, extra memory for two moment estimates | Learning rate, betas, weight decay | Transformers, quick strong baselines |
| L-BFGS | Full or large batch, curvature history | Few knobs | Small to medium smooth problems, scikit-learn logistic regression |
| Newton's method | Hessian solve, expensive in high dimensions | Few knobs | Small models, GLMs |
NaN or inf loss early; a log of zero or a too-large learning rate is the usual cause.