Every machine learning algorithm, from linear regression to transformers, rests on the same small set of ideas. This section covers those ideas once, carefully, so the algorithm pages can build on them: how a loss is derived from probability, how features are prepared, how gradient descent minimises the loss, and how evaluation, cross-validation and regularisation tell you whether the model will work on data it has never seen.
Supervised learning has one goal: learn a function from examples that makes good predictions on new examples. Three questions follow. What should "good" mean? That is the loss function, which for most models comes from maximum likelihood, so probability and statistics come first. How do we find the parameters that minimise the loss? For almost all modern models the answer is some variant of gradient descent. And how do we know the result will generalise? That is the job of evaluation metrics, held-out data, cross-validation, and the bias-variance trade-off with its practical tool, regularisation.
Features sit in the middle. A model can only use the information you give it in a form it can exploit, and many real-world gains come from better features rather than a fancier algorithm. Feature engineering is also where the most damaging mistake in ML, data leakage, usually happens.
If you are new to the field, read in this order: probability and statistics, then gradient descent, then the bias-variance trade-off and overfitting with regularisation, then evaluation metrics and cross-validation, and finally feature engineering. If you already train models and want to avoid common mistakes, jump straight to cross-validation (leakage and split strategy), evaluation metrics (choosing the right one for imbalanced data), and feature engineering (pipelines that fit transforms only on training folds).
| If you are asking... | Read |
|---|---|
| Why is my loss function what it is? Why does L2 regularisation work? | Probability and Statistics; Overfitting and Regularization |
| Training diverges or stalls; what learning rate or optimizer? | Gradient Descent |
| Great training score, poor test score | Bias-Variance Trade-off; Overfitting and Regularization |
| Accuracy is 99% but the model is useless | Evaluation Metrics |
| How should I split data, and how do I tune without cheating? | Cross-Validation |
| Is model B really better than model A? | Probability and Statistics; Cross-Validation |
| How do I encode categories, scale, or create new inputs? | Feature Engineering |
With these foundations, the algorithm families make more sense. Supervised learning covers regression, classification and ensembles; clustering covers learning without labels; and MLOps covers what happens after a model is trained: tracking, registering, serving and monitoring it. For tooling, see scikit-learn and PyTorch.