Logistic regression is the default baseline for binary classification: a linear model whose output is squashed through the sigmoid function into a probability. Despite the name it is a classifier, not a regressor — "regression" refers to the fact that it regresses the log-odds of the positive class on the features. It is fast, convex, well calibrated out of the box, and its coefficients are directly interpretable, which is why it still anchors credit scoring, medical risk models, click-through prediction and nearly every first pass at a new tabular problem.
It sits under Classification in the Supervised Learning branch of Machine Learning.
Given a feature vector x, the model computes a linear score z = w·x + b and passes it through the logistic (sigmoid) function σ(z) = 1 / (1 + e−z). The result is interpreted as P(y = 1 | x). Rearranging gives the key identity: ln(p / (1 − p)) = w·x + b. The model is linear in the log-odds.
That identity is what makes the coefficients interpretable. A one-unit increase in feature xj, holding the others fixed, adds wj to the log-odds, i.e. multiplies the odds by ewj. A coefficient of 0.69 means "doubles the odds". This only holds cleanly when features are not strongly collinear and when you remember the scale of each feature — standardize first if you want to compare coefficient magnitudes.
Logistic regression is fit by maximum likelihood. Each label is treated as a Bernoulli draw with probability pi = σ(w·xi + b); maximizing the likelihood is equivalent to minimizing the average binary cross-entropy (log loss):
L(w, b) = −(1/n) Σ [ yi ln pi + (1 − yi) ln(1 − pi) ]
The loss is convex in (w, b), so there is a single global optimum and no initialization lottery. Its gradient has a remarkably clean form: ∂L/∂w = (1/n) Σ (pi − yi) xi — the prediction error times the input. There is no closed-form solution (unlike linear regression), so solvers iterate: Newton / IRLS for small problems, L-BFGS as a strong general default, and SAG/SAGA or plain SGD for large or sparse data.
Why not mean-squared error? Squared error on a sigmoid output is non-convex and gives vanishing gradients when the model is confidently wrong. Log loss penalizes confident mistakes heavily (the loss goes to infinity as p → 0 for a true positive), which is exactly the signal you want.
The classifier predicts class 1 when p ≥ t for some threshold t. With the default t = 0.5 that is the same as w·x + b ≥ 0: a hyperplane. Moving the threshold shifts the boundary parallel to itself and trades precision for recall — pick t from a precision-recall or cost curve on validation data rather than accepting 0.5 by default, especially for imbalanced classes.
Because the boundary is linear in whatever features you give it, non-linear structure must be added by hand: interaction terms, polynomial features (see Polynomial Regression), splines, or bucketized numerics. If you find yourself engineering many of these, a tree ensemble such as gradient boosting will usually find them for you.
With many features or perfectly separable data, unregularized maximum likelihood drives weights toward infinity (the sigmoid becomes a step function). Regularization adds a penalty to the loss:
λ‖w‖22 — shrinks all weights smoothly; the scikit-learn default.λ‖w‖1 — drives some weights exactly to zero, giving built-in feature selection. Needs the liblinear or saga solver.In scikit-learn the strength is set by C = 1/λ, so smaller C means stronger regularization. The same penalties are discussed for regression in Ridge / Lasso Regression.
For K > 2 classes there are two approaches. One-vs-rest trains K independent binary models. Multinomial (softmax) regression trains a single model with one weight vector per class and P(y = k | x) = ezk / Σj ezj, optimizing categorical cross-entropy. Softmax regression is exactly the final layer of most neural-network classifiers.
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegressionCV
from sklearn.metrics import classification_report, roc_auc_score, log_loss
X, y = load_breast_cancer(return_X_y=True, as_frame=True)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, stratify=y, random_state=0)
# Scaling matters: the penalty treats every coefficient equally.
model = make_pipeline(
StandardScaler(),
LogisticRegressionCV(Cs=20, cv=5, penalty="l2", scoring="neg_log_loss", max_iter=5000),
)
model.fit(X_tr, y_tr)
proba = model.predict_proba(X_te)[:, 1]
print("ROC AUC :", round(roc_auc_score(y_te, proba), 4))
print("log loss:", round(log_loss(y_te, proba), 4))
print(classification_report(y_te, (proba >= 0.5).astype(int)))
# Interpret: odds ratio per one standard deviation of each feature
clf = model[-1]
odds = np.exp(clf.coef_[0])
top = np.argsort(np.abs(clf.coef_[0]))[::-1][:5]
for i in top:
print(f"{X.columns[i]:<25} coef={clf.coef_[0][i]:+.3f} odds x{odds[i]:.2f}")
LogisticRegressionCV picks C by cross-validation. Because inputs are standardized, each odds ratio reads as "the odds multiply by this much per one standard deviation of the feature".
| Strength | Limitation |
|---|---|
| Convex training, fast, scales to millions of rows and sparse features (text, one-hot). | Linear boundary; misses interactions unless you engineer them. |
| Probabilities are usually well calibrated. | Sensitive to outliers in feature space and to unscaled inputs. |
| Coefficients are odds ratios — easy to explain to regulators. | Coefficients become unstable with strong multicollinearity. |
| Strong baseline; any fancier model must beat it. | Usually outperformed by boosted trees on complex tabular data. |
Reach for it when you need a transparent, calibrated score, when data are high-dimensional and sparse (bag-of-words text classification is a classic fit, alongside Naive Bayes), or as the benchmark every other model must justify itself against.
class_weight="balanced" and then trusting probabilities. Reweighting shifts the intercept, so the outputs are no longer calibrated to the true base rate.max_iter, scale the features, or switch solver; do not ignore them.