Ridge and lasso are linear regression with a penalty on coefficient size. Ordinary least squares chooses the coefficients that best fit the training data; when features are many, noisy or correlated, those coefficients become large and unstable and the model overfits. Adding a penalty λ·‖β‖ to the loss trades a little bias for a large reduction in variance. Ridge uses the squared L2 norm and shrinks all coefficients smoothly; lasso uses the L1 norm and sets some coefficients exactly to zero, performing feature selection. Elastic net blends the two.
Part of Regression within Supervised Learning; it extends Linear Regression.
minimize ‖y − Xβ‖2 + λ‖β‖22. Closed form: β̂ = (XTX + λI)−1XTy. Adding λI makes the matrix invertible even when features are collinear or p > n.minimize (1/2n)‖y − Xβ‖2 + λ‖β‖1. No closed form because |β| is not differentiable at zero; solved by coordinate descent with soft-thresholding, βj ← sign(z)·max(|z| − λ, 0), or by LARS.λ[ρ‖β‖1 + ½(1 − ρ)‖β‖22] with mixing ratio ρ (l1_ratio in scikit-learn).The intercept is not penalized. Penalties are equivalent to Bayesian priors: ridge is the MAP estimate under a Gaussian prior on β, lasso under a Laplace prior. In scikit-learn the strength parameter is called alpha; in logistic regression and SVMs the inverse C = 1/λ is used instead.
Each penalized problem is equivalent to minimizing the squared error subject to a budget ‖β‖ ≤ t. The squared-error contours are ellipses centred on the OLS solution; the penalized solution is where the smallest ellipse first touches the constraint region. The L2 region is a disk (a sphere in higher dimensions) with no corners, so the touching point generally has every coordinate non-zero. The L1 region is a diamond (a cross-polytope) whose corners lie on the axes, and an ellipse is likely to touch a corner first — giving coefficients that are exactly zero. In high dimensions the diamond has many corners and edges, so lasso solutions become sparse.
Plotting each coefficient against log λ shows the difference directly. At λ = 0 both equal OLS. As λ grows, ridge coefficients shrink proportionally and approach zero only in the limit; lasso coefficients shrink linearly and drop to exactly zero one at a time, so the path doubles as a ranking of feature relevance. Choose λ by cross-validation over a log-spaced grid (RidgeCV, LassoCV, ElasticNetCV compute whole paths efficiently with warm starts). For a sparser, more stable model, pick the largest λ whose CV error is within one standard error of the minimum.
Ridge is the safer default for prediction when many features each carry a little signal, and it handles groups of correlated features by sharing weight among them. Lasso suits problems where you believe only a few features matter and you want a compact, interpretable model; with correlated features it tends to pick one arbitrarily and drop the rest, and it can select at most n features when p > n. Elastic net keeps lasso's sparsity while selecting correlated features together, and is the usual choice for wide data such as genomics or high-dimensional text.
import numpy as np
from sklearn.datasets import make_regression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LinearRegression, RidgeCV, LassoCV, ElasticNetCV
from sklearn.metrics import mean_squared_error
# 200 samples, 150 features, only 10 informative -> OLS overfits badly
X, y, true_coef = make_regression(n_samples=200, n_features=150, n_informative=10,
noise=10.0, coef=True, random_state=0)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, random_state=0)
models = {
"OLS": LinearRegression(),
"Ridge": RidgeCV(alphas=np.logspace(-3, 3, 50)),
"Lasso": LassoCV(cv=5, random_state=0, max_iter=20000),
"ElasticNet": ElasticNetCV(l1_ratio=[0.2, 0.5, 0.8, 1.0], cv=5, random_state=0, max_iter=20000),
}
for name, est in models.items():
pipe = make_pipeline(StandardScaler(), est).fit(X_tr, y_tr)
rmse = mean_squared_error(y_te, pipe.predict(X_te)) ** 0.5
coef = pipe[-1].coef_
print(f"{name:<10} test RMSE {rmse:7.2f} non-zero coefs {np.sum(np.abs(coef) > 1e-8):3d}")
lasso = make_pipeline(StandardScaler(), models["Lasso"]).fit(X_tr, y_tr)[-1]
print("lasso alpha:", round(lasso.alpha_, 4))
print("true informative:", np.flatnonzero(true_coef)[:10])
print("lasso selected :", np.flatnonzero(lasso.coef_)[:15])
With 105 training rows and 150 features, OLS interpolates the training data and generalizes poorly; ridge and lasso both recover, and lasso keeps a much smaller set of non-zero coefficients that overlaps heavily with the true informative features.
| Ridge | Lasso | Elastic Net | |
|---|---|---|---|
| Sparsity | No | Yes | Yes |
| Correlated features | Shares weight | Picks one, unstable | Selects groups |
| p > n | Works | At most n features | Works |
| Solver | Closed form / fast | Coordinate descent | Coordinate descent |
| Hyperparameters | λ | λ | λ, ρ |
1/2n while Ridge does not, so alpha values are not comparable across the two.max_iter or loosen tol, and scale features.When the features are orthonormal (XTX = I), both estimators have simple closed forms in terms of the OLS coefficients, which makes their behaviour easy to see. Ridge with objective ‖y − Xβ‖2 + λ‖β‖22 divides every coefficient by 1 + λ. Lasso with objective ½‖y − Xβ‖2 + λ‖β‖1 applies soft-thresholding: it subtracts λ from each magnitude and clips at zero.
Suppose OLS gives (3.0, 1.2, 0.4, −0.2) and λ = 0.5. Ridge returns (2.0, 0.8, 0.27, −0.13): everything shrinks by the same factor and nothing reaches zero. Lasso returns (2.5, 0.7, 0, 0): large coefficients lose a fixed 0.5, and the two small ones are removed from the model. That constant shift is also lasso's known weakness: big, genuine effects are biased toward zero by the full λ. The relaxed lasso (refit OLS or a lighter penalty on the selected set) and the adaptive lasso (smaller penalties on large initial coefficients) reduce that bias. With correlated features the closed forms no longer hold, but the qualitative picture is the same.
Ridge has a direct solution and is usually solved with a Cholesky or SVD factorization; the SVD form shows that ridge shrinks the component along each singular direction by dj2 / (dj2 + λ), so directions with little variance in the data are shrunk most. That is why ridge stabilizes nearly collinear designs. Lasso and elastic net are usually solved by coordinate descent: cycle through the coefficients, and update each one by soft-thresholding its partial residual correlation while holding the others fixed. Computing a whole path of λ values from large to small with warm starts is often barely slower than a single fit, which is why LassoCV and glmnet fit full paths by default. LARS computes the exact lasso path in a different way and is efficient when the number of features is modest.
The same penalties appear well beyond linear regression. L2-penalized logistic regression is scikit-learn's default (C is the inverse of the penalty strength), L1 logistic regression is a standard sparse classifier, and "weight decay" in neural network training is an L2 penalty (or, in AdamW, a decoupled shrinkage step closely related to it).
| Situation | Suggested start |
|---|---|
| Many weak, correlated predictors; prediction is the goal | RidgeCV |
| Expect a few strong predictors; want a short model | LassoCV, then check selection stability |
| Groups of correlated predictors; want sparsity | ElasticNetCV with l1_ratio in 0.1–0.9 |
| p much larger than n (genomics, text) | Elastic net or lasso with a cross-validated path |