Semi-supervised learning trains a model for a labeled task when only a small fraction of the data has labels and the rest is unlabeled. Self-supervised learning goes further: it manufactures its own targets from raw data (mask a word and predict it, crop an image twice and match the crops), learning general representations with zero human labels. Both exist because labels are expensive and raw data is cheap.
Self-supervised pretraining is how every modern foundation model is built — BERT's masked language modeling and GPT's next-token prediction are self-supervised objectives. Semi-supervised methods such as pseudo-labeling and consistency regularization remain practical tools when you have, say, 2,000 labeled tickets and 2 million unlabeled ones.
The distinction is about where the training signal comes from. In supervised learning a human provides y for every x. In unsupervised learning there is no y; you model density or structure (clustering, PCA). Semi-supervised sits between: the goal is still to predict a human-defined y, but the unlabeled data shapes the decision boundary. Self-supervised learning creates a pretext task whose labels are free (the hidden token, the rotation angle, which two crops came from the same image), solves it at scale, then transfers the learned encoder to a downstream task.
Both rely on assumptions about the data. The cluster assumption says points in the same dense cluster share a label, so decision boundaries should pass through low-density regions. The manifold assumption says high-dimensional data lies near a lower-dimensional manifold, and labels vary smoothly along it. When these assumptions fail — classes heavily overlap, or the unlabeled data comes from a different distribution — unlabeled data can actively hurt.
Self-training / pseudo-labeling. Train on the labeled set, predict the unlabeled pool, keep predictions whose confidence exceeds a threshold τ, add them as labels, retrain. It is simple and works with any classifier, including gradient-boosted trees. scikit-learn ships it as SelfTrainingClassifier.
Consistency regularization. A model's prediction on an unlabeled example should not change under small perturbations. The loss adds a term ‖f(aug1(x)) − f(aug2(x))‖ on unlabeled data. Mean Teacher, Pi-model, and UDA follow this idea. FixMatch combines both: it pseudo-labels a weakly augmented view (only if confidence ≥ τ, typically 0.95) and trains the model to predict that label on a strongly augmented view of the same image.
Graph-based label propagation. Build a k-NN similarity graph over all points; labels diffuse along edges until convergence. LabelPropagation and LabelSpreading in scikit-learn implement this. It directly encodes the cluster assumption but scales as O(n2) in memory for dense graphs, so it is best for tens of thousands of points, not millions.
Entropy minimization pushes the model toward confident predictions on unlabeled data, which moves boundaries out of dense regions. It is usually a small auxiliary term rather than a standalone method.
Self-supervised objectives fall into three families:
zi with positive zj is −log( exp(sim(zi,zj)/τ) / Σk≠i exp(sim(zi,zk)/τ) ), where sim is cosine similarity and τ a temperature. SimCLR, MoCo, and CLIP (image-text pairs) use it.The augmentation policy is the inductive bias: it defines what the representation should be invariant to. Heavy color jitter teaches color invariance — good for object recognition, bad if color is the label (e.g. ripe vs unripe fruit).
The example hides 95% of the digit labels and compares a baseline trained only on the labeled 5% against SelfTrainingClassifier, which uses the unlabeled 95% too. Unlabeled samples are marked with -1.
import numpy as np
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
from sklearn.svm import SVC
from sklearn.semi_supervised import SelfTrainingClassifier, LabelSpreading
X, y = load_digits(return_X_y=True)
X = X / 16.0
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, random_state=0, stratify=y)
rng = np.random.default_rng(0)
y_partial = y_tr.copy()
mask_unlabeled = rng.random(len(y_tr)) < 0.95
y_partial[mask_unlabeled] = -1 # -1 means "unlabeled"
print("labeled examples:", (y_partial != -1).sum())
# Baseline: labeled data only
base = SVC(probability=True, gamma="scale").fit(X_tr[~mask_unlabeled], y_tr[~mask_unlabeled])
print("supervised-only acc:", base.score(X_te, y_te))
# Self-training: iteratively adds confident pseudo-labels
st = SelfTrainingClassifier(SVC(probability=True, gamma="scale"), threshold=0.9, max_iter=10)
st.fit(X_tr, y_partial)
print("self-training acc: ", st.score(X_te, y_te))
print("pseudo-labeled added:", (st.labeled_iter_ > 0).sum())
# Graph-based propagation over a kNN graph
ls = LabelSpreading(kernel="knn", n_neighbors=7, alpha=0.2).fit(X_tr, y_partial)
print("label spreading acc:", ls.score(X_te, y_te))
On digits, which satisfies the cluster assumption well, both semi-supervised variants typically beat the labeled-only baseline. Always run the baseline: on data where classes overlap, the gain can vanish or reverse.
For self-supervised pretraining in practice, you rarely train from scratch — you download a pretrained encoder (a sentence-transformer, a DINOv2 or CLIP image encoder from Hugging Face), freeze it, and train a linear probe or small head on your labels. That pattern, "self-supervised backbone + few labels," is the dominant way semi-supervised problems are solved today.
| Situation | Reasonable approach |
|---|---|
| Images or text, a few hundred labels, a domain a public model covers | Pretrained self-supervised encoder + linear probe or light fine-tune |
| Tabular data, small labeled set, large unlabeled set from the same source | Pseudo-labeling with gradient boosting; validate on held-out labels |
| Strong cluster structure, moderate n | Label propagation / spreading |
| Large proprietary corpus unlike public data (medical scans, logs, sensor streams) | Domain-specific self-supervised pretraining (MAE, contrastive, masked LM), then fine-tune |
| Labels plentiful | Plain supervised learning; semi-supervised adds complexity for little gain |
τ on it. Keep a labeled validation split untouched.