Almost every machine learning method is probability and statistics in disguise. Loss functions are negative log-likelihoods, regularisation is a prior, and comparing two models is a hypothesis test. This page builds the essentials from first principles: random variables and distributions, Bayes' rule, maximum likelihood and MAP estimation, and the tests and intervals you need to decide whether one model is really better than another.
A random variable is a quantity whose value depends on chance: whether a customer churns, how many requests arrive per second, the error of a sensor. Its distribution says how likely each value is. Discrete variables have a probability mass function P(X = x); continuous ones have a density p(x) whose integral over a range gives a probability.
Four distributions cover a surprising share of ML. Bernoulli(p) models one binary outcome and underlies logistic regression and binary cross-entropy. Binomial(n, p) counts successes in n trials, for example correct predictions on a test set. Normal(μ, σ²) appears whenever many small effects add up (the central limit theorem) and underlies squared-error loss. Poisson(λ) models counts of events per interval, such as clicks per hour. Also worth knowing: the categorical distribution (softmax outputs), the uniform, and the Beta distribution, which describes uncertainty about a probability.
Two summaries matter most. The expectation E[X] is the long-run average. The variance Var(X) = E[(X − E[X])²] measures spread. Bias and variance of an estimator, and the bias-variance trade-off in model complexity, are built directly on these two definitions.
Probability gives ML its vocabulary. A classifier that outputs 0.9 is claiming a conditional probability P(y = 1 | x); if it is poorly calibrated, decisions based on thresholds will be wrong. Training by minimising cross-entropy is exactly maximum likelihood under a Bernoulli or categorical model. L2 regularisation is a Gaussian prior on weights. And every claim of the form "the new model is better" is a statistical inference from a finite sample, which can be wrong by chance.
Conditional probability P(A | B) is the probability of A given that B happened: P(A | B) = P(A and B) / P(B). Rearranging gives Bayes' rule: P(H | D) = P(D | H) · P(H) / P(D). It tells you how to update a belief about a hypothesis H after seeing data D.
The diagnostic-test example in the figure is the classic lesson: even a test with 99% sensitivity gives only about a 17% chance of disease after a positive result if the disease affects 1% of people and the false positive rate is 5%. The prior matters. The same effect shows up in fraud and anomaly detection, where rare positives mean that most alerts are false alarms even for a good model. Naive Bayes classifiers apply the rule directly, with the "naive" assumption that features are conditionally independent given the class.
Maximum likelihood estimation (MLE) picks parameters θ that make the observed data most probable: θMLE = argmax ∏ p(xi | θ). In practice we maximise the log-likelihood, because sums are easier than products, or equivalently minimise the negative log-likelihood, which is what a loss function is. For a normal model the MLE of the mean is the sample mean; minimising squared error is MLE with Gaussian noise; minimising cross-entropy is MLE for a Bernoulli or categorical output.
Maximum a posteriori (MAP) adds a prior: θMAP = argmax p(D | θ) · p(θ). After three heads in three flips the MLE says the coin always lands heads (p = 1.0), which is overconfident. A Beta(2, 2) prior, gently favouring fair coins, gives MAP = (3 + 1) / (3 + 2) = 0.8. With more data the prior's influence fades. A Gaussian prior on weights turns MAP into L2 (ridge) regularisation and a Laplace prior gives L1 (lasso), which is why regularisation is often explained as "a prior on the parameters".
A hypothesis test asks whether an observed difference could plausibly be due to chance. You state a null hypothesis (the models are equally accurate), compute a test statistic, and get a p-value: the probability of a result at least this extreme if the null were true. A small p-value is evidence against the null; it is not the probability that the null is true, and it says nothing about the size of the effect.
A confidence interval is usually more informative: a range that, under repeated sampling, would contain the true value a stated fraction of the time (for example 95%). The bootstrap computes intervals for almost any metric by resampling the test set with replacement many times and recomputing the metric each time.
| Question | Test | Note |
|---|---|---|
| Two models, same test examples, accuracy | McNemar's test or paired t-test | Pairing removes per-example difficulty |
| Two A/B variants, conversion rates | Two-proportion z-test or chi-squared | Fix sample size before looking |
| Means of two independent groups | Welch's t-test | Does not assume equal variances |
| Any metric (F1, AUC, BLEU) | Bootstrap confidence interval | Resample examples, not predictions |
| Did the feature distribution shift? | Kolmogorov-Smirnov test | Large samples flag tiny shifts; check effect size |
import numpy as np
from scipy import stats
rng = np.random.default_rng(0)
# 1) MLE for a normal distribution = sample mean and (biased) std
data = rng.normal(loc=5.0, scale=2.0, size=500)
mu_hat, sigma_hat = data.mean(), data.std(ddof=0)
print(f"MLE mu={mu_hat:.2f} sigma={sigma_hat:.2f}")
# 2) MLE vs MAP for a coin: 3 heads, 0 tails, Beta(2,2) prior
heads, tails, a, b = 3, 0, 2, 2
p_mle = heads / (heads + tails)
p_map = (heads + a - 1) / (heads + tails + a + b - 2)
print(f"MLE p={p_mle:.2f} MAP p={p_map:.2f}")
# 3) Bayes rule for a diagnostic test
prior, sens, fpr = 0.01, 0.99, 0.05
post = sens * prior / (sens * prior + fpr * (1 - prior))
print(f"P(sick | positive) = {post:.3f}")
# 4) Is model B really better than model A? (per-example correctness)
a_correct = rng.binomial(1, 0.80, size=1000)
b_correct = rng.binomial(1, 0.83, size=1000)
t, p = stats.ttest_rel(b_correct, a_correct) # paired: same test examples
print(f"paired t-test: t={t:.2f} p={p:.4f}")
# 5) Bootstrap 95% CI for the accuracy difference
diffs = [rng.choice(b_correct - a_correct, size=1000).mean() for _ in range(2000)]
lo, hi = np.percentile(diffs, [2.5, 97.5])
print(f"accuracy gain 95% CI: [{lo:.3f}, {hi:.3f}]")The script estimates a normal distribution by MLE, contrasts MLE and MAP on the three-heads coin, reproduces the 17% diagnostic result, and compares two classifiers evaluated on the same 1,000 examples using both a paired test and a bootstrap interval for the accuracy gain. If the interval excludes zero, the improvement is unlikely to be noise. Requires numpy and scipy.