Regression

Regression is the supervised learning task of predicting a continuous number — a price, a temperature, a demand forecast, a time to failure. A regression model learns a function f(x) that minimizes some measure of the gap between predictions and true values, most often squared error, which makes it estimate the conditional mean E[y | x]. This page introduces the linear family covered in the deep dives and the metrics used to compare any regressor.

1. The Linear Regression Family

All three deep-dive models share one engine: least squares on a design matrix. Linear regression uses the raw features. Polynomial regression expands features into powers and interactions to bend the fit while remaining linear in the coefficients. Ridge and lasso add a penalty on coefficient size to control variance when features are many or correlated, with lasso also selecting features. Together they cover a large share of practical regression problems where interpretability and stable extrapolation matter.

2. Deep Dives

3. Error Metrics

RMSE is in the target's units and weights large errors heavily, matching the squared-error loss most models optimize. MAE is more robust and corresponds to predicting the median. R² is unitless and compares against always predicting the mean (it can be negative on test data). MAPE is popular in forecasting but explodes near zero and penalizes over- and under-prediction asymmetrically. If the target spans orders of magnitude, model log(y) or use RMSLE. Always plot residuals against predictions; patterns reveal problems that single numbers hide.

4. Beyond Linear Models

Most classification algorithms have a regression twin: regression trees predict leaf means, KNN averages neighbors, SVR fits an epsilon-insensitive tube, and ensembles such as random forests and gradient boosting are usually the most accurate choice on tabular data. Quantile regression predicts percentiles instead of the mean, which is how prediction intervals are built. For sequences over time see Time-Series Forecasting.

5. Comparing Regressors in Code

import numpy as np
from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import cross_validate, KFold
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler, PolynomialFeatures
from sklearn.linear_model import LinearRegression, RidgeCV, LassoCV
from sklearn.ensemble import HistGradientBoostingRegressor

X, y = fetch_california_housing(return_X_y=True)
models = {
    "linear":     make_pipeline(StandardScaler(), LinearRegression()),
    "poly2+ridge": make_pipeline(StandardScaler(), PolynomialFeatures(2), StandardScaler(),
                                 RidgeCV(alphas=np.logspace(-2, 3, 20))),
    "lasso":      make_pipeline(StandardScaler(), LassoCV(cv=5, random_state=0)),
    "boosting":   HistGradientBoostingRegressor(random_state=0),
}
cv = KFold(5, shuffle=True, random_state=0)
for name, m in models.items():
    r = cross_validate(m, X, y, cv=cv,
                       scoring=["neg_root_mean_squared_error", "neg_mean_absolute_error", "r2"])
    print(f"{name:<12} RMSE {-r['test_neg_root_mean_squared_error'].mean():.3f}  "
          f"MAE {-r['test_neg_mean_absolute_error'].mean():.3f}  R2 {r['test_r2'].mean():.3f}")

6. A Practical Workflow

Start with a trivial baseline: predict the training mean (or, for time-ordered data, the last observed value). Its RMSE is the number every other model must beat, and R² is literally measured against it. Next fit a regularized linear model on standardized features. It trains in seconds, its coefficients explain the main effects, and its residual plots reveal what is missing: curvature suggests polynomial terms or splines, a funnel shape suggests a log transform of the target, and clusters of large residuals point to missing features. Only then move to tree ensembles, which capture interactions and nonlinearity automatically and are usually the strongest choice on tabular data.

7. A Worked Numeric Example

Suppose three houses have true prices of 200, 300 and 400 (thousand) and a model predicts 210, 280 and 430. The errors are +10, −20 and +30. MAE = (10 + 20 + 30) / 3 = 20. MSE = (100 + 400 + 900) / 3 ≈ 466.7, so RMSE ≈ 21.6; RMSE exceeds MAE because the 30-unit miss is weighted more heavily. The mean of the targets is 300, so the total sum of squares is 100² + 0 + 100² = 20,000, and the residual sum of squares is 1,400. R² = 1 − 1,400 / 20,000 = 0.93. If the model had instead predicted 300 for every house, R² would be exactly 0; predictions worse than that give a negative R².

8. Common Pitfalls

• Leakage: features computed with future information (a sale-price-derived column, a rolling mean that includes the target day) give excellent validation scores and fail in production.
• Random splits on time series: shuffled cross-validation lets the model see the future. Use forward-chaining splits, as on the forecasting page.
• Extrapolation: trees and forests predict a constant beyond the range of the training targets; linear models extrapolate a straight line that may be equally wrong. Know which behavior you want.
• Heteroscedastic and skewed targets: prices, durations and counts often have variance growing with the mean. Model log(y), or use a Poisson or Tweedie loss for counts and insurance claims.
• Multicollinearity: correlated features make individual linear coefficients unstable even when predictions are fine; ridge regression stabilizes them.
• Outliers: squared error lets a few extreme points pull the fit. Huber loss or MAE-based models are more robust.

9. Uncertainty and Production Notes

A point prediction rarely suffices for decisions. Quantile regression (supported by gradient boosting libraries through a quantile loss) produces, for example, 10th and 90th percentile predictions that bracket the likely outcome. Conformal prediction wraps any regressor and calibrates intervals on a held-out set so that they achieve a stated coverage under the assumption that future data is exchangeable with the calibration data. In production, monitor the input feature distributions and the residuals over time; rising error in one segment is often the earliest sign of drift. Retrain on a schedule, keep the baseline running for comparison, and version the feature pipeline with the model.

Further reading. Hastie, Tibshirani and Friedman (2009), The Elements of Statistical Learning, chapters 3 and 10; James, Witten, Hastie and Tibshirani (2013), An Introduction to Statistical Learning; Koenker and Bassett (1978), "Regression Quantiles", Econometrica; Friedman (2001), "Greedy Function Approximation: A Gradient Boosting Machine"; the scikit-learn user guide sections on linear models and model evaluation.

10. Related Topics