A model is only as good as the way it is measured. This page explains the main metrics for classification, regression and ranking, what each one rewards, and how to choose the one that matches the real cost of mistakes.
An evaluation metric turns a model's predictions on held-out data into a number that says how good it is. The metric you optimise and report decides which model you ship, so choosing it is a design decision, not a formality. Metrics differ by task: classification compares predicted labels or scores to true classes, regression measures numeric error, and ranking measures whether the most relevant items appear near the top.
Every metric should be computed on data the model has not seen during training or tuning: a held-out test set, cross-validation folds, or a time-based split for anything temporal.
A single headline number hides trade-offs. A fraud model with 99.9% accuracy may catch no fraud at all when only 0.1% of transactions are fraudulent. A forecast with low average error may be badly wrong on the high-value days that matter. Good metric choice aligns the model with the real cost of errors, makes comparisons between models fair, and gives monitoring something meaningful to track after deployment.
All threshold-based classification metrics come from the four cells of the confusion matrix. Accuracy = (TP + TN) / total. Precision = TP / (TP + FP): of the items flagged, how many were right. Recall (sensitivity, true positive rate) = TP / (TP + FN): of the real positives, how many were found. Specificity = TN / (TN + FP). F1 is the harmonic mean 2PR / (P + R), which is low unless both precision and recall are reasonable; Fβ weights recall β times as heavily as precision.
Most classifiers output a score, and a threshold converts it to a label. Raising the threshold usually increases precision and decreases recall. Threshold-free metrics summarise all thresholds at once:
ROC AUC plots true positive rate against false positive rate; its area equals the probability that a randomly chosen positive is scored above a randomly chosen negative. It is insensitive to class balance, which makes it look optimistic on rare-event problems. PR AUC (average precision) plots precision against recall; its baseline is the positive rate, so it is more informative when positives are rare. Log loss (cross-entropy) evaluates the probabilities themselves and punishes confident mistakes heavily; pair it with a calibration plot when probabilities drive decisions. For multi-class problems, compute per-class metrics and average them: macro averaging treats classes equally, micro pools all decisions, weighted weights by class frequency.
Choosing the operating threshold is a separate step from choosing the model. A principled approach assigns a cost to each kind of error (for example, a missed fraud costs the transaction amount, a false alert costs an analyst's time) and picks the threshold that minimises expected cost on validation data. Alternatively, fix a constraint such as "recall at least 0.9" and maximise precision subject to it. Revisit the threshold when the class balance changes.
With errors ei = yi − ŷi: MAE = mean |ei| is in the target's units and robust to outliers. RMSE = √(mean ei2) penalises large errors more and is what least-squares models minimise. MAPE = mean |ei / yi| is scale-free but undefined at zero and asymmetric; sMAPE and weighted MAPE (sum of absolute errors over sum of actuals) are common fixes. R2 = 1 − SSres/SStot compares the model with predicting the mean; it can be negative on test data. For forecasting, also compare against a naive baseline such as "same as last week" (MASE does this formally).
Search, recommendation and retrieval for RAG are judged on order. Precision@k is the share of the top k results that are relevant; recall@k is the share of all relevant items found in the top k. MRR (mean reciprocal rank) averages 1/rank of the first relevant result, rewarding getting one good answer to the top. NDCG@k handles graded relevance: DCG = Σ (2reli − 1) / log2(i + 1), divided by the DCG of the ideal ordering so scores fall between 0 and 1. MAP averages precision at each relevant position.
The script computes the main metrics of each family with scikit-learn and reproduces the confusion matrix above (pip install scikit-learn numpy).
import numpy as np
from sklearn.metrics import (confusion_matrix, precision_score, recall_score, f1_score,
accuracy_score, roc_auc_score, average_precision_score, log_loss,
mean_absolute_error, mean_squared_error, r2_score, ndcg_score)
# Classification: 100 positives, 900 negatives
rng = np.random.default_rng(0)
y = np.r_[np.ones(100), np.zeros(900)].astype(int)
pred = np.r_[np.ones(80), np.zeros(20), np.ones(40), np.zeros(860)].astype(int)
score = np.clip(pred * 0.6 + rng.uniform(0, 0.4, 1000), 0.001, 0.999)
print(confusion_matrix(y, pred, labels=[1, 0])) # [[80 20] [40 860]]
print("acc %.3f P %.3f R %.3f F1 %.3f" % (accuracy_score(y, pred),
precision_score(y, pred), recall_score(y, pred), f1_score(y, pred)))
print("ROC AUC %.3f PR AUC %.3f log loss %.3f" % (roc_auc_score(y, score),
average_precision_score(y, score), log_loss(y, score)))
# Regression
y_true = np.array([100, 150, 200, 250, 300.0])
y_hat = np.array([110, 140, 210, 230, 330.0])
print("MAE %.1f RMSE %.1f R2 %.3f" % (mean_absolute_error(y_true, y_hat),
np.sqrt(mean_squared_error(y_true, y_hat)), r2_score(y_true, y_hat)))
# Ranking: graded relevance of 5 results for one query, and the model's scores
relevance = np.array([[3, 2, 0, 0, 1]])
scores = np.array([[0.9, 0.3, 0.8, 0.1, 0.5]])
print("NDCG@5 %.3f" % ndcg_score(relevance, scores, k=5))Note that accuracy (0.94) looks excellent while precision is only 0.67: one in three alerts is false. Whether that is acceptable depends on what an alert costs compared with a missed case, which is a business question, not a statistical one.
| Metric | Task | Range | Use when | Avoid when |
|---|---|---|---|---|
| Accuracy | Classification | 0–1 | Classes balanced, errors cost the same | Rare positives |
| Precision / Recall / F1 | Classification | 0–1 | Positive class matters; costs asymmetric | You need threshold-free comparison |
| ROC AUC | Classification | 0.5–1 (useful range) | Ranking quality, balanced-ish data | Heavy imbalance |
| PR AUC | Classification | baseline–1 | Rare-event detection | Negatives matter as much |
| Log loss | Classification | 0–∞ | Probabilities drive decisions | Only labels matter |
| MAE / RMSE | Regression | 0–∞ (units) | Typical error; RMSE when big errors hurt | Comparing across scales |
| MAPE | Regression | 0–∞ (%) | Business-friendly percentages | Targets near zero |
| NDCG@k / MRR | Ranking | 0–1 | Order of results matters | Unordered set outputs |