How unfairness gets into machine-learning systems, how to measure it with group fairness metrics, why the metrics conflict, and how to mitigate it before, during and after training, with a runnable Fairlearn example.
A machine-learning system is unfair when its errors or decisions systematically disadvantage people based on characteristics such as race, sex, age or disability, often called protected or sensitive attributes. Bias in this sense is not the statistical bias of an estimator; it is a systematic skew, introduced somewhere between the world and the decision, that falls unevenly on groups.
Bias can enter at every stage: historical bias in the world the data records, unrepresentative sampling, labels that measure the wrong thing (arrests instead of offending, past hiring decisions instead of job performance), models that learn proxies for protected attributes, and deployment loops where decisions shape the next round of training data. Simply removing the sensitive column ("fairness through unawareness") rarely helps, because zip code, name, school or purchase history can encode it.
Automated decisions in hiring, lending, insurance, healthcare and policing affect people at scale, and errors that look small on average can concentrate on one group. Well-documented cases include commercial face-analysis systems with much higher error rates for darker-skinned women (Buolamwini and Gebru, 2018) and a healthcare risk algorithm that used past cost as a proxy for need and so under-referred Black patients (Obermeyer et al., 2019). Anti-discrimination law applies to automated decisions, and AI-specific rules such as the EU AI Act require data governance and bias examination for high-risk systems. Beyond compliance, a model that fails on a subgroup is simply a worse model for those users.
Fairness metrics compare a model's behaviour across groups. Let Ŷ be the prediction, Y the true outcome and A the group.
These criteria conflict. When base rates differ between groups, a classifier that is not perfect cannot satisfy calibration and equal error rates at the same time (Kleinberg, Mullainathan and Raghavan, 2016; Chouldechova, 2017). Choosing a metric is therefore a value judgement about which error matters most in the context, and should be made with domain experts and affected stakeholders, then documented.
Pre-processing changes the data: reweighing examples so group and label are independent, resampling, or repairing labels. In-processing adds a fairness constraint or penalty to training (for example the reductions approach of Agarwal et al., 2018, implemented in Fairlearn's ExponentiatedGradient). Post-processing adjusts decisions after training, such as group-specific thresholds chosen to equalise error rates (Fairlearn's ThresholdOptimizer). Post-processing is simple but needs the sensitive attribute at decision time, which may be legally restricted.
This example uses Fairlearn on synthetic data in which a proxy feature leaks the group. It measures per-group metrics, then applies a post-processing mitigation. Install with pip install fairlearn scikit-learn.
import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from fairlearn.metrics import (MetricFrame, selection_rate, true_positive_rate,
false_positive_rate, demographic_parity_ratio,
equalized_odds_difference)
from fairlearn.postprocessing import ThresholdOptimizer
rng = np.random.default_rng(0)
n = 20_000
group = rng.choice(["A", "B"], n, p=[0.6, 0.4])
skill = rng.normal(0, 1, n)
proxy = (group == "A").astype(float) + rng.normal(0, 0.5, n) # e.g. zip code signal
# historical labels favoured group A
y = (skill + 0.6 * (group == "A") + rng.normal(0, 0.5, n) > 0.5).astype(int)
X = np.column_stack([skill, proxy])
Xtr, Xte, ytr, yte, gtr, gte = train_test_split(X, y, group, test_size=0.3, random_state=0)
clf = LogisticRegression().fit(Xtr, ytr)
pred = clf.predict(Xte)
mf = MetricFrame(metrics={"selection": selection_rate, "TPR": true_positive_rate,
"FPR": false_positive_rate},
y_true=yte, y_pred=pred, sensitive_features=gte)
print(mf.by_group.round(3))
print("DP ratio", round(demographic_parity_ratio(yte, pred, sensitive_features=gte), 3))
print("EO diff ", round(equalized_odds_difference(yte, pred, sensitive_features=gte), 3))
post = ThresholdOptimizer(estimator=clf, constraints="equalized_odds",
prefit=True, predict_method="predict_proba")
post.fit(Xtr, ytr, sensitive_features=gtr)
fair = post.predict(Xte, sensitive_features=gte, random_state=0)
print("after: EO diff", round(equalized_odds_difference(yte, fair, sensitive_features=gte), 3),
" accuracy", round((fair == yte).mean(), 3), "vs", round((pred == yte).mean(), 3))Read the output as a trade-off report: the equalized-odds gap shrinks, usually at some cost in overall accuracy against the historical labels. Note that those labels themselves encode the historical preference for group A, so "accuracy" here partly measures agreement with past bias. That is why label quality and metric choice deserve as much attention as the algorithm.
| Criterion | Requires equal | Good when | Caveat |
|---|---|---|---|
| Demographic parity | Selection rate | Outcome should not depend on group (e.g. ad delivery) | Ignores real differences in qualification; can reduce accuracy |
| Equal opportunity | True positive rate | Missing a qualified person is the main harm (hiring, loans) | Allows different false positive rates |
| Equalized odds | TPR and FPR | Both kinds of error harm people (risk assessment) | Often costly in accuracy; may need randomised thresholds |
| Calibration | Outcome rate at each score | Scores are read as probabilities by humans | Incompatible with equal error rates when base rates differ |
| Individual fairness | Similar treatment for similar people | A meaningful similarity metric exists | Defining similarity is hard |