Deployed models degrade quietly as the world changes. This page explains the kinds of drift, how to measure them with PSI, the KS test and other methods, and how to build a monitoring loop that catches problems before the business does.
A model is trained on a snapshot of the world and then used in a world that keeps changing. Model monitoring is the practice of continuously checking that a deployed model still receives the kind of data it was trained on and still makes good predictions. Drift is the general name for the ways the data or the relationship being modelled changes over time.
In probability terms, a model learns P(y | x) from examples drawn from P(x, y). Data drift means the input distribution P(x) moved; concept drift means the mapping P(y | x) itself moved; label shift means the base rate P(y) moved. Each needs a different response.
Models fail silently. A web service with a bug returns errors; a model with stale assumptions still returns confident numbers. Typical causes are mundane: an upstream team changes a unit from dollars to cents, a new product launches, a mobile app update stops sending a field, a pandemic changes buying habits. Without monitoring these show up weeks later as a business metric that went wrong for reasons nobody can trace.
The Population Stability Index (PSI), from credit scoring, compares a reference distribution (usually training or a validation window) with a current window. Bin the feature using the reference quantiles, compute the fraction of rows in each bin for reference ri and current ci, and sum (ci − ri) · ln(ci / ri). It is symmetric, zero when the distributions match, and works for categorical features with one bin per category. Add a small epsilon to empty bins to avoid dividing by zero. The thresholds 0.1 and 0.25 are industry conventions, not statistical tests.
The two-sample Kolmogorov–Smirnov (KS) test compares two samples of a continuous variable. Its statistic D is the largest vertical gap between the two empirical cumulative distribution functions, and it yields a p-value. Beware: with hundreds of thousands of rows, KS flags tiny, irrelevant differences as significant. Use the statistic D as an effect size and set thresholds on it, or test on samples.
Other common measures: chi-squared tests for categorical features, Jensen–Shannon divergence and Wasserstein distance for distributions, and the share of missing values or unseen categories. For multivariate drift, a useful trick is a domain classifier: train a model to distinguish reference rows from current rows; an AUC well above 0.5 means the distributions differ, and its feature importances say where.
Compute PSI and KS for a feature whose mean has shifted, using NumPy and SciPy.
import numpy as np
from scipy.stats import ks_2samp
def psi(reference, current, bins=10, eps=1e-6):
edges = np.quantile(reference, np.linspace(0, 1, bins + 1))
edges[0], edges[-1] = -np.inf, np.inf
r = np.histogram(reference, edges)[0] / len(reference)
c = np.histogram(current, edges)[0] / len(current)
r, c = np.clip(r, eps, None), np.clip(c, eps, None)
return float(np.sum((c - r) * np.log(c / r)))
rng = np.random.default_rng(42)
train = rng.normal(50, 10, 20_000) # e.g. customer age at training time
stable = rng.normal(50, 10, 5_000)
shifted = rng.normal(56, 12, 5_000) # younger app launched, older users join
for name, cur in [("stable", stable), ("shifted", shifted)]:
d, p = ks_2samp(train, cur)
print(f"{name:8s} PSI={psi(train, cur):.3f} KS D={d:.3f} p={p:.2g}")The stable window gives a PSI near zero and a small D; the shifted one gives a PSI in the moderate-to-major range and a tiny p-value. In production you would run this per feature on a schedule (hourly or daily windows), store the results, and alert on sustained breaches rather than single spikes. Libraries such as Evidently, NannyML, whylogs and the monitoring features of SageMaker, Vertex AI and Databricks package these checks.
Monitor four layers. Service: latency, errors, throughput. Inputs: schema, nulls, ranges, drift per feature. Outputs: distribution of predictions and scores, which drift even when labels are not yet available. Outcomes: accuracy, calibration and business KPIs once ground truth arrives — minutes later for clicks, months later for loan defaults. When labels are slow, estimate performance (for example, confidence-based estimators) and rely more heavily on input and output drift.
Respond in proportion: a data-quality bug is fixed upstream, not by retraining; real data drift may need retraining on recent data; concept drift may need new features or a new model. Scheduled retraining is a reasonable default, but always validate the new model against the current one before promotion.
Monitoring is only as good as what was recorded at prediction time. For every request, log a request id, timestamp, model name and version, the exact feature values the model saw (after feature engineering, not just raw inputs), the prediction or score, and any decision taken on it. Store these in a table partitioned by date so you can join outcomes later by request id. Keep the reference dataset and its summary statistics versioned alongside the model artefact, so every model version has a matching baseline.
For unstructured inputs such as text and images, per-feature tests do not apply directly. A common approach is to monitor embedding distributions (for example, distance of new embeddings to the training centroid, or a domain classifier on embeddings) plus simple proxies like input length, language and the share of out-of-vocabulary tokens.
| Method | Data type | Output | Notes |
|---|---|---|---|
| PSI | Numeric (binned) or categorical | Index, conventional thresholds | Easy to explain; depends on binning |
| KS test | Continuous | Statistic D and p-value | Over-sensitive on large samples |
| Chi-squared | Categorical | Statistic and p-value | Needs enough counts per category |
| Jensen–Shannon | Any (binned) | Bounded 0 to 1 (base-2 log) | Symmetric and smooth |
| Wasserstein | Numeric | Distance in feature units | Interpretable magnitude |
| Domain classifier | Multivariate | AUC + importances | Catches joint shifts univariate tests miss |