Bagging is the simplest powerful idea in ensemble learning: train many noisy models on resampled data and average them. Random forests add per-split feature subsampling and turn the idea into one of the most dependable models for tabular data. This page covers the mechanics, the variance math that explains why it works, the knobs worth tuning, and where it falls short.
Bagging (bootstrap aggregating, Breiman 1996) trains B copies of the same high-variance model on B bootstrap resamples of the training data and averages their predictions. Each bootstrap sample draws n rows with replacement from the original n, so some rows repeat and roughly 1 - 1/e ≈ 36.8% are left out. Those left-out rows are the tree's out-of-bag (OOB) set, which gives a free validation estimate.
A Random Forest adds one more source of randomness: at every split, a tree only considers a random subset of max_features features. Bagging alone de-noises; feature subsampling de-correlates. The combination is one of the most reliable off-the-shelf models for tabular data. The site also has a hands-on Random Forest classifier walkthrough.
Suppose each tree's prediction has variance σ2 and any two trees have pairwise correlation ρ. The variance of their average is ρσ2 + (1 - ρ)σ2/B. Adding trees drives the second term to zero but never touches the first. That is the whole design rationale of a random forest: fully grown trees have low bias and high variance; averaging attacks the variance; feature subsampling lowers ρ so the floor drops too.
Bias is essentially unchanged by bagging — the average of many unbiased-but-noisy trees is still roughly unbiased. This is why bagging helps unstable learners (deep trees, high-degree models) and does almost nothing for stable ones such as linear regression or k-NN with a large k.
Two consequences follow. First, increasing n_estimators does not overfit — it only plateaus — so the right number is "enough that the OOB score stops moving," typically a few hundred. Second, the knob that actually trades bias against variance is max_features: smaller values de-correlate more but make each tree weaker.
A single unpruned decision tree carves axis-aligned rectangles until every leaf is pure, producing a staircase boundary that chases individual noisy points. Averaging hundreds of such trees, each built on a different resample, smooths the staircase into something close to the true boundary. The forest still uses axis-aligned splits internally, so the boundary is a fine-grained step function rather than a true curve, but at the scale of the data it behaves smoothly.
from sklearn.datasets import fetch_california_housing
from sklearn.ensemble import RandomForestRegressor, BaggingRegressor
from sklearn.tree import DecisionTreeRegressor
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_absolute_error
X, y = fetch_california_housing(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=0)
# Plain bagging of deep trees
bag = BaggingRegressor(DecisionTreeRegressor(), n_estimators=200,
oob_score=True, n_jobs=-1, random_state=0).fit(X_tr, y_tr)
# Random forest = bagging + per-split feature subsampling
rf = RandomForestRegressor(n_estimators=400, max_features=0.33,
min_samples_leaf=2, oob_score=True,
n_jobs=-1, random_state=0).fit(X_tr, y_tr)
for name, m in [("bagging", bag), ("forest", rf)]:
print(name, "OOB R^2=%.3f" % m.oob_score_,
"test MAE=%.3f" % mean_absolute_error(y_te, m.predict(X_te)))
# Impurity importances are fast but biased toward high-cardinality features;
# prefer permutation importance on held-out data.
from sklearn.inspection import permutation_importance
pi = permutation_importance(rf, X_te, y_te, n_repeats=5, random_state=0, n_jobs=-1)
print(pi.importances_mean.round(3))The OOB score is computed by predicting each training row using only the trees that did not see it. On reasonably sized datasets it tracks a held-out test score closely and saves you a separate validation split during tuning.
| Parameter | Effect | Typical starting point |
|---|---|---|
n_estimators | More trees = lower variance, diminishing returns, linear cost | 300–500; stop when OOB plateaus |
max_features | Main de-correlation knob | sqrt for classification, ~0.33 for regression |
min_samples_leaf | Smooths leaves, shrinks the model, helps noisy targets | 1–5 |
max_depth | Caps tree size; usually unnecessary but limits memory | None, or a cap for latency |
max_samples | Bootstrap size; smaller = faster and more diverse trees | None (=n), or 0.5–0.8 on big data |
class_weight | Imbalanced classification | balanced_subsample |
Random forests are famously forgiving: defaults are usually within a few percent of a tuned model, which is why they remain a strong baseline before reaching for gradient boosting.
max_depth or min_samples_leaf when model size matters.ExtraTreesClassifier) go further by also randomizing split thresholds; faster and sometimes better on noisy data.feature_importances_. They inflate continuous and high-cardinality features. Use permutation importance or SHAP on held-out data.GroupKFold or a time split instead.CalibratedClassifierCV if probabilities drive decisions.The curves are computed directly from the variance formula in section 2 with σ2 = 1. With ρ = 0.3, one tree has variance 1.0, ten trees about 0.37, and a hundred trees about 0.31, already close to the floor of 0.3. Reducing correlation to ρ = 0.1, which is what random feature subsets aim to do, brings a hundred trees down to about 0.11. Adding trees attacks the second term; de-correlating attacks the first, which is the one that limits large forests. In practice lowering max_features also makes each tree slightly worse (higher σ2 and bias), so the best value is found empirically, usually with the OOB score.
Two more numbers are worth knowing. A bootstrap sample of size n contains each original row with probability 1 − (1 − 1/n)n, which approaches 1 − 1/e ≈ 0.632. So each tree sees about 63% of distinct rows and the remaining 37% are out-of-bag for that tree; with 500 trees, each row is out-of-bag for roughly 180 of them, plenty for a stable OOB estimate. And for majority voting, three independent classifiers that are each right 60% of the time give a majority that is right with probability 3 · 0.62 · 0.4 + 0.63 = 0.648. The gain grows with more voters only if their errors stay weakly correlated, the same lesson in classification form.
A forest is not a black box if you use the right tools on held-out data:
Bagging is not limited to trees. Any unstable learner benefits, and the same idea appears as bagged neural network ensembles and as "pasting" (sampling without replacement) or random patches (sampling both rows and features) in scikit-learn's BaggingClassifier. Compared with gradient boosting, bagged trees are trained independently and parallelize trivially, whereas boosting trains sequentially, each tree correcting the residual errors of the ensemble so far.