Supervised Learning

Supervised learning is the branch of machine learning that learns a mapping from inputs x to known outputs y using labeled examples. Given training pairs (xi, yi), the algorithm searches for a function f that minimizes a loss on those pairs while still generalizing to new data. It covers most production ML: fraud scoring, demand forecasting, churn prediction, medical risk models, and the fine-tuning stage of language models.

1. The Workflow

Every supervised project follows the same loop: collect labeled data, split it so that some data is never seen during training, fit a model by minimizing a loss (squared error, log loss, hinge loss), select hyperparameters with cross-validation, and report performance on the untouched test set. The test set estimates how the model will behave in production; reusing it for tuning silently turns it into training data.

2. Classification vs Regression

The type of target decides the task. Classification predicts a discrete label (spam / not spam, one of 1,000 image classes) and is evaluated with accuracy, precision, recall, F1, ROC-AUC and log loss. Regression predicts a continuous quantity (price, demand, time to failure) and is evaluated with RMSE, MAE and R². Many algorithms — trees, ensembles, KNN, neural networks — come in both flavours.

3. Deep Dives

4. Core Concepts

• Bias–variance trade-off. Simple models underfit (high bias); flexible models overfit (high variance). Regularization, more data and ensembling move the balance.
• Generalization and validation. Use k-fold cross-validation for model selection; use time-based splits for temporal data and grouped splits when rows share an entity (patient, customer).
• Data leakage. Any information from the test set or from the future that reaches training — scaling on all data, target-derived features — inflates offline scores. Fit preprocessing inside a pipeline.
• Label quality. Noisy or biased labels cap achievable accuracy; auditing labels often beats tuning models.
• Choosing a baseline. Start with a linear model and a gradient-boosted tree; anything more complex has to beat both.

5. Minimal Example

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import HistGradientBoostingClassifier

X, y = load_breast_cancer(return_X_y=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, stratify=y, random_state=0)

candidates = {
    "logistic": make_pipeline(StandardScaler(), LogisticRegression(max_iter=2000)),
    "boosting": HistGradientBoostingClassifier(random_state=0),
}
for name, model in candidates.items():
    cv = cross_val_score(model, X_tr, y_tr, cv=5, scoring="roc_auc")
    print(f"{name:<9} CV ROC-AUC {cv.mean():.3f} +/- {cv.std():.3f}")

best = candidates["logistic"].fit(X_tr, y_tr)   # pick by CV, then touch the test set once
print("test accuracy:", round(best.score(X_te, y_te), 3))

6. Splits and Validation in Practice

The single most common reason a model disappoints in production is an evaluation that was too optimistic. Fit every preprocessing step (scaling, imputation, target encoding) on the training data only, which scikit-learn pipelines enforce automatically. Use the validation data, or cross-validation folds, for all decisions: algorithm, hyperparameters, decision thresholds and early stopping. Report the test score once.

The split must mirror deployment. If the model will score next month's customers, train on the past and test on the most recent period. If one patient contributes many rows, put all of that patient's rows on the same side with GroupKFold. For rare classes, stratify so each fold contains positives.

7. Picking an Algorithm

On tabular benchmarks, tree ensembles remain very strong and usually need less tuning than neural networks (Grinsztajn et al., 2022). Deep learning wins when the input is perceptual or sequential and large pretrained models exist.

8. Choosing the Right Metric

The metric should encode the cost of mistakes. With 1% fraud, a model that predicts "not fraud" for everyone is 99% accurate and useless; precision, recall and the precision-recall curve tell the real story. When a missed positive is far costlier than a false alarm (cancer screening), favor recall and pick the threshold accordingly. When outputs feed a decision that uses probabilities (pricing, expected loss), check calibration and use log loss or the Brier score. For regression, MAE is robust to outliers while RMSE punishes large errors more; MAPE breaks down near zero targets. Always compare against a trivial baseline (majority class, mean prediction, last value) so improvements are meaningful.

9. From Notebook to Production

• Version everything: data snapshot, code, features and model artifact, so a prediction can be reproduced.
• Feature parity: compute features the same way offline and online; a feature store or shared library avoids train/serve skew.
• Monitoring: track input distributions, prediction distributions and, once labels arrive, live accuracy. Labels often arrive late, so input drift is the early warning.
• Retraining: schedule it, and gate each new model behind the same test protocol plus a shadow or A/B comparison.

Further reading: Hastie, Tibshirani & Friedman, The Elements of Statistical Learning (2nd ed., 2009); Breiman, "Statistical Modeling: The Two Cultures" (2001); Sculley et al., "Hidden Technical Debt in Machine Learning Systems" (NeurIPS 2015).

10. Related Topics