Gradients say which way is downhill; optimizers decide how far and how to move. This page builds from plain gradient descent to Adam and AdamW, explains learning-rate schedules, and compares them in runnable PyTorch.
Training a neural network means finding weights w that make a loss L(w) small. Backpropagation computes the gradient g = ∇L(w), which points uphill. An optimizer is the rule that turns that gradient into a weight update. The simplest rule is gradient descent: step a small distance downhill, w ← w − η g, where η is the learning rate. Every optimizer on this page is a refinement of that one line.
The contour plot shows why refinements are needed. Real loss surfaces have narrow valleys that are steep in one direction and shallow in another. Plain gradient descent (red) bounces between the steep walls while crawling along the valley floor. Momentum (yellow) accumulates velocity in the consistent direction and cancels the side-to-side oscillation, reaching the minimum in fewer steps.
The optimizer and its learning rate decide whether training converges, how fast, and sometimes how well the model generalises. A learning rate that is too large makes the loss explode or oscillate; too small and training stalls. Modern networks have millions to billions of parameters with gradients of very different magnitudes, so a single global step size is a poor fit. Adaptive optimizers give each parameter its own effective step, which is why Adam and AdamW are the defaults for transformers, while well-tuned SGD with momentum remains common for convolutional image models.
Optimizer choice also interacts with everything else in the recipe: batch size, initialisation, normalisation layers and regularisation. A useful habit is to change one of these at a time and to tune the learning rate first, because it is almost always the single most influential hyperparameter.
Stochastic gradient descent (SGD). Compute g on a mini-batch rather than the whole dataset, then w ← w − η g. The noise from mini-batches is cheap and even helps escape poor regions.
Momentum. Keep a velocity v ← β v + g and update w ← w − η v, with β typically 0.9. Consistent gradient directions add up; alternating ones cancel. Nesterov momentum evaluates the gradient at the look-ahead point w − η β v, which corrects overshoot slightly earlier.
AdaGrad. Accumulate squared gradients per parameter, s ← s + g², and divide the step by √s + ε. Rarely updated parameters get larger steps, which suits sparse features, but s only grows, so the step size eventually shrinks toward zero.
RMSProp. Replace the sum with an exponential moving average, s ← ρ s + (1−ρ) g², so the denominator tracks recent gradient scale instead of all history.
Adam. Combine both ideas. Track a first moment m ← β1 m + (1−β1) g and a second moment v ← β2 v + (1−β2) g². Because both start at zero, correct the bias: m̂ = m / (1−β1t), v̂ = v / (1−β2t). Update w ← w − η m̂ / (√v̂ + ε). Defaults are β1 = 0.9, β2 = 0.999, ε = 10−8.
AdamW. L2 regularisation added to the loss interacts badly with Adam's per-parameter scaling: heavily scaled parameters receive less decay. AdamW decouples weight decay and applies it directly: w ← w − η (m̂ / (√v̂ + ε) + λ w). This generally generalises better and is the standard for training large language models.
The learning rate rarely stays constant. Warmup increases it linearly from near zero over the first few hundred or thousand steps; early in training Adam's second-moment estimates are unreliable and large updates can destabilise the network. After warmup, decay shrinks the rate so the optimizer can settle into a minimum. Common shapes are step decay (divide by 10 at fixed epochs), cosine decay (shown above), linear decay, and one-cycle (rise then fall within one run). ReduceLROnPlateau lowers the rate when validation loss stops improving.
The script fits the same small network on a synthetic regression problem with four optimizers and prints the final loss for each. It runs on CPU in a few seconds.
import torch, torch.nn as nn
torch.manual_seed(0)
X = torch.randn(2048, 10)
true_w = torch.randn(10, 1) * torch.logspace(-1, 1, 10).unsqueeze(1) # badly scaled
y = torch.tanh(X @ true_w) + 0.05 * torch.randn(2048, 1)
def make_model():
torch.manual_seed(1)
return nn.Sequential(nn.Linear(10, 64), nn.ReLU(), nn.Linear(64, 1))
configs = {
"SGD": lambda p: torch.optim.SGD(p, lr=0.05),
"SGD+momentum": lambda p: torch.optim.SGD(p, lr=0.05, momentum=0.9),
"RMSprop": lambda p: torch.optim.RMSprop(p, lr=1e-3),
"AdamW": lambda p: torch.optim.AdamW(p, lr=3e-3, weight_decay=0.01),
}
steps = 600
for name, make_opt in configs.items():
model = make_model()
opt = make_opt(model.parameters())
warmup = torch.optim.lr_scheduler.LinearLR(opt, start_factor=0.1, total_iters=50)
cosine = torch.optim.lr_scheduler.CosineAnnealingLR(opt, T_max=steps - 50)
sched = torch.optim.lr_scheduler.SequentialLR(opt, [warmup, cosine], milestones=[50])
for step in range(steps):
idx = torch.randint(0, len(X), (64,))
loss = nn.functional.mse_loss(model(X[idx]), y[idx])
opt.zero_grad()
loss.backward()
opt.step()
sched.step()
with torch.no_grad():
print(f"{name:13s} final MSE = {nn.functional.mse_loss(model(X), y).item():.4f}")On this problem the momentum and adaptive methods typically finish well below plain SGD at the same step budget; exact numbers depend on the seed and learning rates. The lesson is not that one optimizer always wins, but that each needs its own learning rate — comparing optimizers at a shared learning rate is a common unfair test.
| Optimizer | State per parameter | Strengths | Weaknesses | Typical use |
|---|---|---|---|---|
| SGD | None | Simple, low memory | Slow on ill-conditioned problems | Baselines, convex models |
| SGD + momentum | 1 (velocity) | Good generalisation, well understood | Learning rate sensitive | CNNs for vision |
| RMSProp | 1 (squared avg) | Adapts per parameter | No momentum by default | RNNs, reinforcement learning |
| Adam | 2 (m, v) | Fast, robust defaults | Coupled L2 decay, more memory | General deep learning |
| AdamW | 2 (m, v) | Correct weight decay | 2x parameter memory for state | Transformers, LLMs |
Memory matters at scale: with AdamW, optimizer state for a model is roughly twice the size of the weights themselves, which is why techniques such as 8-bit optimizers and sharded optimizer state (ZeRO) exist.
Adam(weight_decay=...) expecting AdamW. In PyTorch these differ; use AdamW.zero_grad(). Gradients accumulate across steps by default.