Classification is the supervised learning task of assigning each input to one of a fixed set of categories: fraud or legitimate, which of ten digits, which product category, which diagnosis. A classifier learns a decision boundary in feature space from labeled examples and usually outputs a probability or score per class, which is then thresholded into a label. This page compares the main algorithm families and the metrics used to judge them.
Each family makes a different assumption about the shape of the boundary. Linear models draw a hyperplane; trees cut feature space into axis-aligned boxes; kernel SVMs draw smooth curved margins; nearest-neighbor methods follow the local data, as jagged as k allows. Probabilistic generative models such as Naive Bayes instead model how each class generates features. Ensembles of trees combine many boxy boundaries into a flexible, accurate one. Setups also vary: binary, multiclass (one label from K, via softmax or one-vs-rest) and multilabel (several labels at once).
Sigmoid of a linear score; convex log-loss training, odds-ratio coefficients, calibrated probabilities. The default baseline.
Greedy Gini/entropy splits into axis-aligned rules; interpretable when shallow, the base learner for ensembles.
Bagged, feature-subsampled decision trees whose votes are averaged; strong out-of-the-box accuracy.
Maximum-margin hyperplanes, hinge loss and the kernel trick for curved boundaries on small-to-medium data.
Bayes rule with conditionally independent features; one-pass training and a fast, strong text baseline.
Majority vote of the k closest training points; no training, sensitive to scaling and dimensionality.
Accuracy hides failure on rare classes: a fraud model that predicts "legitimate" for everything is 99.9% accurate and useless. Pick the metric that matches the cost of errors: recall when misses are expensive (screening), precision when false alarms are (alert fatigue), F1 for a single balance. ROC-AUC measures ranking quality independent of threshold; PR-AUC is more informative under heavy imbalance. Log loss and the Brier score evaluate the probabilities themselves, which matters whenever scores feed a downstream decision. For multiclass, report macro averages (each class equal) alongside micro or weighted averages.
| Situation | Good first choice |
|---|---|
| Tabular data, mixed feature types | Gradient boosting or random forest |
| Need transparent, calibrated scores | Logistic regression, shallow tree |
| Sparse text features, small labeled set | Naive Bayes, linear SVM, logistic regression |
| Few samples, many features, clear margin | SVM (linear or RBF) |
| Good embeddings, similarity matters | KNN on the embeddings |
| Images, audio, raw text at scale | Deep learning |
Whatever the model, handle class imbalance deliberately (class weights, resampling, threshold tuning), use stratified splits, and calibrate probabilities (CalibratedClassifierCV) if they will be read as risks.
from sklearn.datasets import make_classification
from sklearn.model_selection import cross_validate, StratifiedKFold
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.svm import SVC
from sklearn.naive_bayes import GaussianNB
from sklearn.neighbors import KNeighborsClassifier
X, y = make_classification(n_samples=3000, n_features=20, n_informative=8,
weights=[0.85, 0.15], random_state=0)
models = {
"logreg": make_pipeline(StandardScaler(), LogisticRegression(max_iter=2000)),
"tree": DecisionTreeClassifier(max_depth=6, random_state=0),
"forest": RandomForestClassifier(n_estimators=300, random_state=0, n_jobs=-1),
"svm": make_pipeline(StandardScaler(), SVC(C=1.0, gamma="scale")),
"nb": GaussianNB(),
"knn": make_pipeline(StandardScaler(), KNeighborsClassifier(n_neighbors=15)),
}
cv = StratifiedKFold(5, shuffle=True, random_state=0)
for name, m in models.items():
r = cross_validate(m, X, y, cv=cv, scoring=["roc_auc", "average_precision", "f1"])
print(f"{name:<7} ROC-AUC {r['test_roc_auc'].mean():.3f} "
f"PR-AUC {r['test_average_precision'].mean():.3f} F1 {r['test_f1'].mean():.3f}")
Most classifiers produce a score, and the label comes from comparing it to a threshold. The default of 0.5 is only right when classes are balanced and both kinds of error cost the same. If a missed fraud case costs 50 times more than a false alarm, the cost-minimizing threshold on a calibrated probability is about 1 / (1 + 50) ≈ 0.02. Calibration means that among cases scored 0.2, about 20 percent are truly positive; logistic regression is usually well calibrated, while boosted trees, SVMs and Naive Bayes often are not. Fix it with Platt scaling or isotonic regression fitted on held-out data (CalibratedClassifierCV in scikit-learn) and check it with a reliability diagram.
A screening model is tested on 1,000 cases, 50 of them positive. It flags 60 cases, of which 40 are truly positive. So TP = 40, FP = 20, FN = 10, TN = 930. Accuracy = 970 / 1,000 = 97 percent, which sounds strong, but a model that flags nothing scores 95 percent. Precision = 40 / 60 ≈ 0.67, recall = 40 / 50 = 0.80, and F1 = 2 · 0.67 · 0.80 / (0.67 + 0.80) ≈ 0.73. Lowering the threshold to catch the remaining 10 positives will raise FP; the precision-recall curve shows the whole trade-off, and the business decides where to sit on it.
class_weight="balanced") or threshold tuning before resampling; random oversampling and SMOTE can help but are easy to leak if applied before splitting.Beyond tabular data, classification of text, images and audio is now usually done by fine-tuning a pretrained neural network; see Text Classification and Image Classification. For tabular data, gradient boosting remains a strong default.
Further reading. Hastie, Tibshirani and Friedman (2009), The Elements of Statistical Learning; Niculescu-Mizil and Caruana (2005), "Predicting Good Probabilities with Supervised Learning", ICML; Saito and Rehmsmeier (2015), "The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets", PLOS ONE; Elkan (2001), "The Foundations of Cost-Sensitive Learning".