Every model is wrong in two ways: it can be too rigid to capture the pattern, or so flexible that it memorises noise. This page explains bias and variance from first principles, derives the error decomposition, measures both in a simulation, and shows which levers fix which problem.
A model's error on new data comes from three sources. Bias is error from wrong assumptions: a model too simple to capture the real pattern, like the straight line on the left, is systematically off no matter how much data it sees. Variance is error from sensitivity to the particular training sample: a very flexible model, like the degree-14 polynomial on the right, fits the noise in this sample and would look quite different if trained on another one. Irreducible noise is randomness in the target that no model can predict.
Making a model more flexible usually lowers bias and raises variance. The bias-variance trade-off is the search for the complexity where their sum is smallest.
The trade-off is the core reason we evaluate on held-out data. Training error keeps falling as a model grows, so it cannot tell you when you have gone too far. The dartboard view shows the vocabulary: imagine retraining the same algorithm on many different samples from the same population. Bias is how far the average dart lands from the bullseye; variance is how scattered the darts are. Diagnosing which one dominates tells you what to do next: high bias calls for a richer model or better features; high variance calls for more data, regularisation or a simpler model. Doing the wrong one wastes effort; adding data to a high-bias model barely helps.
For regression with squared error, suppose y = f(x) + ε with noise of mean 0 and variance σ². Let f̂(x) be the model learned from a random training set D. The expected error at a point x, averaged over training sets and noise, splits exactly into three terms:
E[(y − f̂(x))²] = (E[f̂(x)] − f(x))² + E[(f̂(x) − E[f̂(x)])²] + σ²
That is bias² + variance + noise. Bias² measures how far the average prediction is from the truth; variance measures how much predictions fluctuate around their average from one training set to another. The curve above is the classical picture: as complexity rises, training error falls steadily, while test error falls (bias shrinking) then rises (variance growing). The sweet spot is where test error bottoms out.
For classification with 0-1 loss the decomposition is not this clean, but the intuition carries over.
import numpy as np
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
rng = np.random.default_rng(0)
f = lambda x: np.sin(2 * np.pi * x)
x_test = np.linspace(0, 1, 200)
def bias_variance(degree, n_datasets=200, n=30, noise=0.3):
preds = np.empty((n_datasets, x_test.size))
for i in range(n_datasets): # many training sets from one process
x = rng.uniform(0, 1, n)
y = f(x) + rng.normal(0, noise, n)
model = make_pipeline(PolynomialFeatures(degree), LinearRegression())
model.fit(x[:, None], y)
preds[i] = model.predict(x_test[:, None])
mean_pred = preds.mean(axis=0)
bias2 = np.mean((mean_pred - f(x_test)) ** 2)
variance = np.mean(preds.var(axis=0))
return bias2, variance, noise ** 2
for d in [1, 3, 5, 9, 15]:
b, v, n2 = bias_variance(d)
print(f"degree {d:2d}: bias^2={b:.3f} variance={v:.3f} noise={n2:.3f} total={b+v+n2:.3f}")Because bias and variance are defined over many possible training sets, the code simulates exactly that: it draws 200 training sets of 30 points from the same noisy sine wave, fits a polynomial of a given degree to each, and measures how far the average prediction is from the truth (bias²) and how much the predictions spread (variance). Running it shows bias² large for degree 1 and close to zero by degree 5, while variance grows sharply for high degrees with only 30 points. Total error is lowest at a moderate degree. Exact numbers depend on the random seed; the shape of the result does not.
In real projects you only have one dataset, so you estimate the same picture with cross-validation and learning curves: plot training and validation error against training-set size. If both are high and close together, bias dominates. If training error is low and validation error much higher, with the gap shrinking as data grows, variance dominates.
| Lever | Effect on bias | Effect on variance | Examples |
|---|---|---|---|
| More model capacity | Down | Up | More layers, deeper trees, higher-degree features |
| Regularisation | Up (slightly) | Down | L1/L2 penalties, dropout, early stopping, pruning |
| More training data | Unchanged | Down | Collect data, augmentation |
| Better features | Down | Usually unchanged | Domain features, embeddings |
| Bagging | Unchanged | Down | Random forests |
| Boosting | Down | Can rise if over-trained | Gradient boosting, XGBoost |
| Fewer features | Up | Down | Feature selection |
Ensembles exploit the decomposition directly: averaging many high-variance, low-bias trees (bagging) cancels much of their variance, while boosting adds weak, high-bias learners sequentially to reduce bias.
A modern caveat. Very large neural networks and other heavily over-parameterised models can show double descent: test error rises near the point where the model can exactly fit the training set, then falls again as capacity grows further. The decomposition still holds, but "bigger always means more variance" does not, particularly with implicit regularisation from training methods such as SGD.
A practical diagnostic routine. Start with a simple baseline (a linear model or a shallow tree) and record its training and validation scores. If both are poor and close together, the problem is bias: add features, try a more flexible model family, or reduce regularisation. If training is good and validation much worse, the problem is variance: add regularisation, collect more data, simplify, or use bagging. Repeat with one change at a time, tracking each run, so the effect of every lever is visible. Also compare against the best achievable error where you can estimate it, for example human performance on a labelling task; the gap between that and training error is a rough estimate of avoidable bias, and the gap between training and validation error estimates variance.