Cross-validation is the standard way to estimate how a model will perform on data it has never seen. This page covers k-fold and its variants, how to tune without bias, and the leakage traps that make scores lie.
A model's score on the data it was trained on says little about how it will perform on new data. The honest test is to hold some data out. Cross-validation makes that test more reliable by holding out different portions in turn. In k-fold cross-validation the data is split into k equal folds; the model is trained k times, each time on k−1 folds and evaluated on the remaining one. Every row is used for validation exactly once.
The result is k scores. Their mean estimates generalisation performance; their standard deviation shows how sensitive the estimate is to which rows happened to land in the validation set. A single train/validation split gives one number with unknown variance; five or ten folds give both an estimate and a sense of its uncertainty.
Cross-validation is how you choose between models, features and hyperparameters without fooling yourself. With a small dataset, a single 80/20 split can make a weak model look strong purely by luck of the split; averaging over folds smooths that out. It also uses data efficiently: every row contributes to both training and evaluation, which matters when labelled data is expensive.
It is equally important to know what cross-validation does not do. Once you use CV scores to pick the best of many options, the winning score is optimistically biased, because you selected it for being high. A final test set, touched once at the very end, or nested cross-validation, is needed for an unbiased estimate of the chosen pipeline.
K-fold. Shuffle, split into k folds, rotate. Common choices are k = 5 or 10. Larger k means each model trains on more data (less pessimistic bias) but costs more fits. Leave-one-out (k = n) is nearly unbiased but expensive and high-variance; it is rarely worth it outside tiny datasets.
Stratified k-fold keeps the class proportions the same in every fold. With rare positives, plain folds may contain none at all, making metrics such as recall undefined or wildly variable. Stratification is the default in scikit-learn for classifiers when you pass an integer cv.
Group k-fold keeps all rows sharing a group identifier — the same patient, user, or device — in the same fold. Without it, a model can memorise individuals and score well on "new" rows from people it has already seen.
Time-series split respects order: each split trains on the past and tests on the period immediately after. Shuffling time-series data lets the model learn from the future to predict the past, a form of leakage. A gap between train and test is useful when labels are defined over a window (for example, "churned within 30 days"), so that training labels do not overlap the test period.
The standard error of the CV mean is roughly std / √k, but because the training sets overlap heavily the folds are not independent, so treat it as a rough guide. When two models differ by less than about one standard deviation of fold scores, the difference is usually not meaningful; repeated k-fold with different shuffles gives a more stable comparison.
The script compares plain, stratified and grouped CV on an imbalanced problem, then shows leak-free tuning with a pipeline inside nested CV.
import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import (KFold, StratifiedKFold, GroupKFold, TimeSeriesSplit,
GridSearchCV, cross_val_score)
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
X, y = make_classification(n_samples=600, n_features=20, n_informative=5,
weights=[0.95], random_state=0)
groups = np.repeat(np.arange(120), 5) # 120 "users", 5 rows each
model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))
for name, cv in [("KFold", KFold(5, shuffle=True, random_state=0)),
("StratifiedKFold", StratifiedKFold(5, shuffle=True, random_state=0)),
("GroupKFold", GroupKFold(5))]:
s = cross_val_score(model, X, y, cv=cv, groups=groups, scoring="average_precision")
print(f"{name:16s} mean={s.mean():.3f} std={s.std():.3f}")
# Time-series split: just show the index ranges
for i, (tr, te) in enumerate(TimeSeriesSplit(n_splits=4, gap=10).split(X)):
print(f"split {i+1}: train 0..{tr[-1]}, test {te[0]}..{te[-1]}")
# Nested CV: inner loop tunes C, outer loop scores the whole tuning procedure
inner = GridSearchCV(model, {"logisticregression__C": [0.01, 0.1, 1, 10]},
cv=StratifiedKFold(3, shuffle=True, random_state=1),
scoring="average_precision")
outer = cross_val_score(inner, X, y, cv=StratifiedKFold(5, shuffle=True, random_state=2),
scoring="average_precision")
print(f"nested CV estimate: {outer.mean():.3f} +/- {outer.std():.3f}")Because the scaler lives inside the pipeline, it is refit on each training fold, so no validation statistics leak into training. The nested estimate is the one to report: it measures the entire procedure "tune C by CV, then fit", not a single lucky configuration.
Suppose two models produce these five fold scores for ROC AUC:
| Model | Fold 1 | Fold 2 | Fold 3 | Fold 4 | Fold 5 | Mean | Std |
|---|---|---|---|---|---|---|---|
| A | 0.81 | 0.84 | 0.79 | 0.83 | 0.82 | 0.818 | 0.017 |
| B | 0.83 | 0.85 | 0.80 | 0.84 | 0.82 | 0.828 | 0.017 |
B's mean is 0.010 higher, smaller than either standard deviation, so on the means alone the gap looks like noise. But both models were scored on the same folds, so compare them fold by fold: B wins on four folds and ties on one, with differences of 0.02, 0.01, 0.01, 0.01 and 0. That paired view is much more convincing than the means suggest, because fold difficulty (fold 3 is hard for both) cancels out. Always evaluate competing models on identical splits by fixing the random_state, and compare paired differences. For a stronger conclusion, repeat 5-fold CV ten times with different shuffles and look at the distribution of the 50 paired differences.
| Strategy | Use when | Cost | Caveat |
|---|---|---|---|
| Hold-out split | Large datasets, expensive models | 1 fit | High variance on small data |
| K-fold | Default for i.i.d. regression | k fits | Ignores groups and time |
| Stratified k-fold | Classification, especially imbalanced | k fits | Still ignores groups |
| Group k-fold | Repeated entities (patients, users) | k fits | Folds may be uneven |
| Time-series split | Forecasting, any temporal drift | k fits | Early splits have little training data |
| Nested CV | Unbiased estimate after tuning | k × inner fits | Expensive |