A multi-layer perceptron is the canonical feedforward neural network: an input layer, one or more fully connected hidden layers with a nonlinear activation, and an output layer. It generalizes the single Perceptron by stacking layers, which lets it represent non-linearly-separable functions such as XOR. MLPs are still the default head of almost every deep model — the feedforward block inside every Transformer layer is an MLP.
This page covers the forward pass, backpropagation, activation and initialization choices, a PyTorch training example, and the practical trade-offs versus tree models on tabular data. It sits under Deep Learning (DL) / Feedforward Neural Networks (FNN).
Every neuron in layer l is connected to every neuron in layer l+1, which is why MLP layers are also called dense or fully connected (nn.Linear in PyTorch, Dense in Keras). A layer with nin inputs and nout outputs holds nin×nout + nout parameters. The network above has 4·6+6 + 6·6+6 + 6·3+3 = 93 parameters.
The universal approximation theorem says a single hidden layer with enough units and a non-polynomial activation can approximate any continuous function on a compact set. In practice, depth is far more parameter-efficient than width: deeper networks compose features hierarchically, so most useful MLPs have 2–6 hidden layers.
For a batch X of shape (B, n0), layer l computes Z(l) = H(l-1) W(l)T + b(l) and H(l) = σ(Z(l)). The final layer usually has no hidden activation: for classification its raw outputs are logits that feed a softmax inside the cross-entropy loss; for regression it is a plain linear output trained with MSE or Huber loss.
Because everything is a matrix multiply, MLP inference is extremely fast on GPUs and CPUs alike, and batch size is the main throughput lever.
Backpropagation is reverse-mode automatic differentiation applied to the layer graph. The forward pass caches every pre-activation z and activation h. The backward pass starts from ∂L/∂z(L) (for softmax + cross-entropy this is simply ŷ − y) and applies the chain rule layer by layer. The cost of the backward pass is roughly twice that of the forward pass, and memory scales with the number of cached activations — which is why batch size and depth are bounded by GPU memory.
The parameter update is then any first-order optimizer: SGD with momentum, or Adam/AdamW, which adapt a per-parameter step size. AdamW with a learning rate around 1e-3 is a reasonable default for small MLPs.
Activations. Sigmoid and tanh saturate: for large |z| their derivative approaches zero, so gradients shrink multiplicatively through deep stacks (the vanishing gradient problem). ReLU has derivative 1 for positive inputs, which made deep MLPs trainable. Its failure mode is "dead" units stuck at zero; LeakyReLU, ELU and GELU (the Transformer default) address that with a small non-zero response for negative inputs. Use sigmoid only on a binary output, never as a hidden default.
Initialization. Weights must be random to break symmetry, with variance scaled to layer width so activations neither explode nor vanish. Use He (Kaiming) initialization, variance 2/nin, with ReLU-family activations, and Xavier/Glorot, variance 2/(nin+nout), with tanh. PyTorch's nn.Linear default is a Kaiming-uniform variant that works acceptably for most cases.
Normalization and regularization. BatchNorm or LayerNorm between layers stabilizes training and permits higher learning rates. Dropout (p = 0.1–0.5) and weight decay are the standard regularizers; early stopping on a validation set is often the most effective of all on small datasets.
A complete classifier on a non-linear toy dataset. Note the input standardization, logits output (no softmax in the model), AdamW, and evaluation under torch.no_grad().
import torch
import torch.nn as nn
from sklearn.datasets import make_moons
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
X, y = make_moons(n_samples=4000, noise=0.25, random_state=0)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=0)
scaler = StandardScaler().fit(X_tr) # fit on train only
X_tr, X_te = scaler.transform(X_tr), scaler.transform(X_te)
to_t = lambda a, dt=torch.float32: torch.tensor(a, dtype=dt)
X_tr, X_te = to_t(X_tr), to_t(X_te)
y_tr, y_te = to_t(y_tr, torch.long), to_t(y_te, torch.long)
class MLP(nn.Module):
def __init__(self, d_in=2, d_hidden=64, d_out=2, p_drop=0.1):
super().__init__()
self.net = nn.Sequential(
nn.Linear(d_in, d_hidden), nn.ReLU(), nn.Dropout(p_drop),
nn.Linear(d_hidden, d_hidden), nn.ReLU(), nn.Dropout(p_drop),
nn.Linear(d_hidden, d_out), # logits
)
def forward(self, x):
return self.net(x)
model = MLP()
opt = torch.optim.AdamW(model.parameters(), lr=1e-3, weight_decay=1e-4)
loss_fn = nn.CrossEntropyLoss() # applies log-softmax
loader = torch.utils.data.DataLoader(
torch.utils.data.TensorDataset(X_tr, y_tr), batch_size=64, shuffle=True)
for epoch in range(30):
model.train()
for xb, yb in loader:
opt.zero_grad()
loss = loss_fn(model(xb), yb)
loss.backward() # backprop
opt.step()
model.eval()
with torch.no_grad():
acc = (model(X_te).argmax(1) == y_te).float().mean().item()
print(f"test accuracy: {acc:.3f}")For quick baselines without PyTorch, sklearn.neural_network.MLPClassifier(hidden_layer_sizes=(64, 64), early_stopping=True) implements the same model with Adam and is fine for small tabular problems.
nn.CrossEntropyLoss expects logits; adding a softmax layer before it silently degrades gradients.model.eval(). Dropout and BatchNorm behave differently at inference; evaluation in train mode gives noisy, pessimistic metrics.