A support vector machine (SVM) finds the hyperplane that separates two classes with the widest possible margin. Only the training points closest to the boundary — the support vectors — determine where it goes. Combined with the kernel trick, the same convex optimization produces highly non-linear boundaries without ever building the high-dimensional features explicitly. SVMs were the dominant general-purpose classifier before deep learning and gradient boosting, and they remain a strong choice for small-to-medium, high-dimensional datasets such as text, genomics, and engineered signal features.
Part of Classification within Supervised Learning.
With labels yi ∈ {−1, +1}, a linear classifier predicts sign(w·x + b). Many hyperplanes may separate the data; the SVM chooses the one that maximizes the distance to the nearest point on either side. Scaling w so that the closest points satisfy yi(w·xi + b) = 1, the margin width becomes 2/‖w‖, and the hard-margin problem is:
minimize ½‖w‖2 subject to yi(w·xi + b) ≥ 1 for all i
This is a convex quadratic program with a unique solution. In its dual form the solution is w = Σ αi yi xi, where αi > 0 only for points on (or inside) the margin. Those are the support vectors; the model is literally a weighted sum over them. A wide margin is a form of regularization: it is the geometric reason SVMs generalize well with few samples.
Real data overlap, so the soft-margin SVM introduces slack variables ξi ≥ 0 that let points violate the margin at a cost:
minimize ½‖w‖2 + C Σ ξi subject to yi(w·xi + b) ≥ 1 − ξi
Eliminating the slacks gives an equivalent unconstrained form: L2-regularized hinge loss, Σ max(0, 1 − yi f(xi)) + (1/2C)‖w‖2. Compare with logistic regression, which uses log loss: hinge loss is exactly zero for points correctly classified beyond the margin, which is why only support vectors matter, and why SVMs do not natively output probabilities. C trades margin width against violations: large C means "fit the training points", small C means "keep the margin wide".
The dual problem and the prediction f(x) = Σ αi yi (xi·x) + b touch the data only through dot products. Replacing xi·x with a kernel K(xi, x) = φ(xi)·φ(x) fits a linear SVM in the space of φ without computing φ. Common kernels:
x·z — best for very high-dimensional sparse data (text); use LinearSVC, which scales to millions of rows.exp(−γ‖x − z‖2) — an infinite-dimensional feature space; the default non-linear choice.(γ x·z + r)d — explicit degree-d interactions.The cost: a kernel SVM needs the n × n kernel matrix (or large parts of it), so training time grows roughly between O(n2) and O(n3). Beyond a few tens of thousands of samples, use a linear SVM, an approximate feature map (Nystroem, RBFSampler) followed by a linear model, or a different algorithm.
For the RBF kernel, γ sets the reach of each support vector: small γ means each point influences a wide region (smooth boundary), large γ means very local influence (island-like boundaries that memorize). C controls how hard the optimizer tries to classify every training point. The two interact, so search them jointly on a log grid (e.g. C and γ each over 10−3 … 103) with cross-validation. scikit-learn's default gamma="scale" = 1 / (n_features · Var(X)) is a reasonable starting point.
from sklearn.datasets import make_moons
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
X, y = make_moons(n_samples=1000, noise=0.25, random_state=0)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.25, random_state=0)
pipe = make_pipeline(StandardScaler(), SVC(kernel="rbf"))
grid = GridSearchCV(
pipe,
{"svc__C": [0.1, 1, 10, 100], "svc__gamma": [0.01, 0.1, 1, 10]},
cv=5, n_jobs=-1,
).fit(X_tr, y_tr)
best = grid.best_estimator_
svc = best[-1]
print("best params :", grid.best_params_)
print("test accuracy :", round(best.score(X_te, y_te), 3))
print("support vecs :", svc.n_support_.sum(), "of", len(X_tr))
# Need probabilities? Calibrate instead of trusting raw margins.
from sklearn.calibration import CalibratedClassifierCV
cal = CalibratedClassifierCV(best, method="sigmoid", cv=5).fit(X_tr, y_tr)
print(cal.predict_proba(X_te[:3]).round(3))
Scaling inside the pipeline is essential: RBF distances are dominated by whichever feature has the largest numeric range. The fraction of training points that end up as support vectors is a useful diagnostic — when it approaches 100% the model is effectively memorizing.
| Strength | Limitation |
|---|---|
| Convex: one global optimum, reproducible. | Kernel SVM training scales poorly past ~105 samples. |
| Effective when features outnumber samples. | No native probabilities; needs calibration. |
| Kernels give flexible boundaries with few hyperparameters. | Sensitive to feature scaling and to C/γ. |
| Linear SVM is excellent for sparse text features. | Hard to interpret with non-linear kernels. |
Good fits: text classification with TF-IDF features, small biomedical datasets, image features from a frozen embedding model, and any problem with a modest number of samples and a clear margin. For large tabular datasets, gradient-boosted trees usually win; for raw images, audio and text, neural networks do.
SVC(kernel="linear") on big data instead of LinearSVC or SGDClassifier(loss="hinge").decision_function as a probability, or relying on probability=True without understanding it runs an internal cross-validated Platt scaling (slow, and can disagree with predict).γ matters at least as much.class_weight="balanced" and threshold the decision function.SVC, LinearSVC, SVR.