Dimensionality reduction turns hundreds of features into a handful while keeping the structure that matters. This landing page explains the goals, contrasts linear and nonlinear methods, and links to the PCA, t-SNE and UMAP pages.
Dimensionality reduction maps data with many features to a lower-dimensional representation that keeps the structure you care about. Reasons to do it: visualize high-dimensional data in 2-D; remove noise and collinearity; speed up downstream models and nearest-neighbor search; and counter the curse of dimensionality, where distances between points become nearly equal and density estimates break down. Feature selection keeps a subset of original columns; feature extraction, the topic here, builds new combined axes.
Linear projection onto orthogonal directions of maximum variance, computed with the SVD. Fast, invertible, interpretable loadings.
Nonlinear embedding that preserves local neighborhoods using a heavy-tailed Student-t kernel. Best for 2-D cluster visualization.
Graph-based manifold learning. Faster than t-SNE, keeps more global layout, and has a transform for new data.
| Need | Use |
|---|---|
| Preprocessing for a model, compress features, denoise | PCA (or TruncatedSVD for sparse data) |
| 2-D plot of clusters in embeddings | UMAP or t-SNE, usually after PCA to ~50 dims |
| Reduce before density clustering | UMAP to 5–20 dims, then HDBSCAN |
| Nonlinear and invertible / generative | Autoencoders |
| Supervised projection that separates classes | LDA (linear discriminant analysis) |
from sklearn.datasets import load_digits
from sklearn.decomposition import PCA
from sklearn.manifold import TSNE
from sklearn.preprocessing import StandardScaler
X, y = load_digits(return_X_y=True) # 1797 x 64
Xs = StandardScaler().fit_transform(X)
X50 = PCA(n_components=30, random_state=0).fit_transform(Xs)
Z = TSNE(n_components=2, perplexity=30, init="pca", random_state=0).fit_transform(X50)
# import umap; Z = umap.UMAP(n_neighbors=15, min_dist=0.1).fit_transform(X50)
print(Z.shape)Most reducers are sensitive to feature scale. PCA maximizes variance, so an unscaled column measured in dollars will dominate one measured in fractions. Standardize first unless every feature already shares a unit (pixel intensities, word counts after TF-IDF). Then fit the reducer on the training split only and apply the learned transform to validation and test data; fitting PCA on the full dataset is a quiet form of leakage. In scikit-learn the clean way is a Pipeline of scaler, reducer and model, which cross-validation then refits inside each fold.
How many dimensions to keep is a hyperparameter, not a fact about the data. For PCA, the explained-variance curve is the usual guide: plot cumulative variance against the number of components and look for the point where extra components add little, or pick a threshold such as 90 or 95 percent. When the reduced data feeds a model, the better answer is to tune n_components by cross-validated score, because variance kept and predictive signal kept are not the same thing; a low-variance direction can carry the label.
Linear projection (PCA). Center the data matrix X, then find orthonormal directions w that maximize the variance of Xw. The answer is the top eigenvectors of the covariance matrix, obtained in practice from the singular value decomposition X = UΣVT. The first k columns of V are the components; the squared singular values give each component's share of variance. Projection is a matrix multiply, so it is fast, deterministic and invertible up to the discarded variance.
Neighbor-graph embeddings (t-SNE, UMAP). These methods first build a notion of who is near whom in the original space, then search for low-dimensional coordinates whose neighbor structure matches. t-SNE converts distances into probabilities with a Gaussian kernel whose width is set by perplexity (roughly the effective number of neighbors), and matches them using a heavy-tailed Student-t kernel in 2-D by minimizing KL divergence. UMAP builds a weighted k-nearest-neighbor graph (n_neighbors) and optimizes a cross-entropy between that graph and one built on the embedding, with min_dist controlling how tightly points pack. Both are iterative and stochastic, so different seeds give different pictures.
Learned encoders. An autoencoder trains a network to compress and reconstruct its input. With linear layers and squared error it recovers the PCA subspace; with nonlinear layers it can follow curved manifolds. The price is training time, tuning, and less interpretable axes.
| Question | Measure |
|---|---|
| How much information did a linear projection keep? | Explained variance ratio; reconstruction error on held-out data |
| Are neighbors in the embedding real neighbors? | Trustworthiness and continuity (sklearn.manifold.trustworthiness); k-NN recall between spaces |
| Does the reduction help the task? | Cross-validated model score with and without the reducer |
| Is the picture stable? | Rerun with several seeds and hyperparameter values; compare cluster membership |
TruncatedSVD (latent semantic analysis for text) instead.Use cases. Compressing sentence or image embeddings before indexing in a vector store; visualizing single-cell gene expression; denoising sensor arrays; removing collinearity before linear regression; and preparing features for K-Means or anomaly detection.
Further reading. Pearson (1901), "On Lines and Planes of Closest Fit"; Hotelling (1933), principal components; van der Maaten and Hinton (2008), "Visualizing Data using t-SNE", JMLR; McInnes, Healy and Melville (2018), "UMAP", arXiv:1802.03426; Wattenberg, Viégas and Johnson (2016), "How to Use t-SNE Effectively", Distill.