An ensemble combines the predictions of many models into one, and is usually more accurate and more robust than any of its members. The two dominant strategies attack different parts of the error: bagging trains many high-variance models independently on resampled data and averages them, cutting variance; boosting trains weak, high-bias models one after another, each correcting the errors of the ensemble so far, cutting bias. Built on decision trees, these give random forests and gradient-boosted trees — the strongest general-purpose models for tabular data.
Bagging (bootstrap aggregating) draws B bootstrap samples, fits a deep tree to each in parallel, and averages predictions or takes a majority vote. Random forests add random feature subsets at every split to decorrelate the trees. The ~37% of rows left out of each bootstrap give a free out-of-bag error estimate. Boosting fits trees sequentially: each new shallow tree is fit to the gradient of the loss (the residuals, for squared error) of the current ensemble, and is added with a small learning rate η. AdaBoost reweights misclassified samples; gradient boosting (XGBoost, LightGBM, CatBoost) generalizes to any differentiable loss.
Bootstrap samples, parallel deep trees, feature subsampling and out-of-bag error — variance reduction with few knobs.
Sequential shallow trees fit to loss gradients; learning rate, early stopping and the libraries that dominate tabular ML.
A worked random forest classification model with scikit-learn.
If B models each have variance σ2 and pairwise correlation ρ, their average has variance ρσ2 + (1 − ρ)σ2/B. Adding models drives the second term to zero but leaves the first: averaging only helps as much as the members disagree. That is why random forests subsample features (lower ρ) and why diverse model types make better ensembles than copies of one model. Boosting works differently: it is a stagewise additive model, Fm(x) = Fm−1(x) + η fm(x), performing gradient descent in function space, so it can keep reducing bias — and will eventually overfit unless stopped early.
Beyond tree ensembles, heterogeneous models can be combined directly. Voting averages the predicted probabilities (soft voting) or labels (hard voting) of different models. Stacking trains a meta-model, often a regularized linear model, on the out-of-fold predictions of base models, learning how much to trust each. Stacking can add a little accuracy at the cost of complexity and latency; it is common in competitions and less common in production, where a single well-tuned gradient-boosted model often suffices.
| Bagging / Random Forest | Gradient Boosting | |
|---|---|---|
| Reduces | Variance | Bias (and variance with shrinkage) |
| Base trees | Deep, independent | Shallow, sequential |
| Training | Parallel | Sequential per tree (parallel within) |
| Tuning effort | Low | Moderate: learning rate, depth, rounds |
| Overfitting with more trees | No (plateaus) | Yes — use early stopping |
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import cross_val_score, StratifiedKFold
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import (RandomForestClassifier, HistGradientBoostingClassifier,
StackingClassifier)
X, y = load_breast_cancer(return_X_y=True)
cv = StratifiedKFold(5, shuffle=True, random_state=0)
tree = DecisionTreeClassifier(random_state=0)
forest = RandomForestClassifier(n_estimators=500, oob_score=True, random_state=0, n_jobs=-1)
gbm = HistGradientBoostingClassifier(learning_rate=0.05, max_iter=500,
early_stopping=True, random_state=0)
stack = StackingClassifier(
estimators=[("rf", forest), ("gbm", gbm)],
final_estimator=make_pipeline(StandardScaler(), LogisticRegression()),
cv=5,
)
for name, m in [("single tree", tree), ("random forest", forest), ("grad boosting", gbm), ("stacking", stack)]:
s = cross_val_score(m, X, y, cv=cv, scoring="roc_auc")
print(f"{name:<14} ROC-AUC {s.mean():.3f} +/- {s.std():.3f}")
print("OOB accuracy:", round(forest.fit(X, y).oob_score_, 3))
Plug numbers into the variance formula above. Suppose each tree has prediction variance σ2 = 1. With B = 100 fully independent trees (ρ = 0) the average has variance 1/100 = 0.01. Real bootstrap trees are correlated; with ρ = 0.5 the variance is 0.5 + 0.5/100 = 0.505, and adding a thousand more trees only reduces it to about 0.5. With ρ = 0.2 (as random feature subsets aim for) it falls to about 0.208. The lesson: past a few hundred trees, more trees barely help; decorrelating them is what moves the floor. That is why random forests restrict each split to a random subset of features (a common default is √p for classification).
For majority voting on classification, if five independent classifiers are each correct 70% of the time, the majority is correct whenever at least three are, which happens with probability about 0.84. That gain shrinks quickly as errors become correlated, and voting among models worse than chance makes things worse.
Stacking trains a meta-model on the predictions of base models. The meta-model must learn from predictions on data the base models did not see, otherwise it learns to trust whichever model overfit the most. The standard recipe produces out-of-fold predictions: split the training data into K folds, train each base model K times, each time predicting the held-out fold, and stack those predictions into a matrix. The meta-learner (usually a simple regularized linear model) trains on that matrix. At inference, base models refit on all data (or the K fold models averaged) feed the meta-learner. scikit-learn's StackingClassifier and StackingRegressor implement this with the cv argument.
Stacking pays off when base models are diverse (a tree ensemble, a linear model on different features, a neural network); stacking five variants of the same gradient-boosting model rarely adds much over tuning one.
| Random forest | Gradient boosting | Stacking / blending | |
|---|---|---|---|
| Main knobs | n_estimators, max_features, min_samples_leaf | learning_rate, n_estimators (early stopping), max_depth / num_leaves, subsampling | Choice of base models, meta-learner regularization |
| Tuning effort | Low | Moderate | High |
| Overfitting risk | Low | Higher without early stopping | Leakage if not out-of-fold |
| Training | Parallel | Sequential trees (parallel within a tree) | Multiplies cost by number of models and folds |
| Best for | Strong, quick baseline | Top accuracy on tabular data | Squeezing the last gains, competitions |
In production, every extra model is extra latency, memory and maintenance. A single well-tuned gradient-boosting model often captures most of the benefit. Ensembles also complicate explanation, though tree-specific SHAP values (Lundberg et al., 2020) make tree ensembles reasonably interpretable.
Further reading: Breiman, L. (1996), "Bagging predictors"; Breiman, L. (2001), "Random forests"; Freund, Y. & Schapire, R. (1997), AdaBoost; Friedman, J. (2001), "Greedy function approximation: a gradient boosting machine"; Wolpert, D. (1992), "Stacked generalization"; Chen, T. & Guestrin, C. (2016), "XGBoost".