When a model makes a decision, people want to know why. This page explains the two most widely used model-agnostic explanation methods, SHAP and LIME, from the underlying ideas to working code, and where they mislead.
Many accurate models — gradient-boosted trees, random forests, neural networks — are hard to read. Explainability methods answer questions such as "why did the model reject this loan?" or "which inputs matter most overall?" without changing the model. SHAP and LIME are the two best-known post-hoc, model-agnostic methods: they treat the model as a function that can be queried and attribute its output to input features.
The SHAP waterfall above explains one prediction: starting from the model's average output on background data (0.20), each feature pushes the score up or down until it reaches this applicant's output (0.38).
Explanations help in four ways. Debugging: a model that leans on a patient ID or a timestamp has learned a leak. Trust: domain experts adopt models whose reasoning they can check. Compliance: credit decisions in many jurisdictions must come with reasons (for example adverse-action notices in US lending), and regulations such as the EU AI Act emphasise transparency for high-risk systems. Fairness review: attributions can reveal reliance on proxies for protected attributes.
SHAP (SHapley Additive exPlanations, Lundberg and Lee, 2017) borrows the Shapley value from cooperative game theory. Treat features as players and the prediction as the payout. A feature's Shapley value is its marginal contribution averaged over every possible order in which features could be added:
φi = ΣS ⊆ F \ {i} [ |S|! (|F| − |S| − 1)! / |F|! ] · ( v(S ∪ {i}) − v(S) )
where v(S) is the expected model output when only features in S are known (the others are averaged over background data). Shapley values are the unique attribution satisfying a set of fairness axioms; the most useful in practice is local accuracy: the base value plus all φi equals the prediction exactly.
Exact computation is exponential in the number of features, so SHAP provides estimators. TreeSHAP computes exact values for tree ensembles in polynomial time and is fast enough for production. KernelSHAP is model-agnostic but slow, using weighted sampling of coalitions. DeepSHAP and gradient-based variants approximate values for neural networks. A subtle choice is how "missing" features are filled: interventional (independent background) versus conditional (respecting correlations) expectations can give different answers when features are correlated.
LIME (Local Interpretable Model-agnostic Explanations, Ribeiro, Singh and Guestrin, 2016) explains one prediction by fitting a simple model in its neighbourhood. It perturbs the instance (for tabular data, sampling feature values; for text, removing words; for images, hiding superpixels), queries the black box on each perturbed sample, weights samples by proximity, and fits a sparse linear model. The linear coefficients are the explanation. LIME is intuitive and works on any model, but results depend on the kernel width and sampling, and can change between runs.
Train a gradient-boosted classifier on a built-in dataset, then explain it with SHAP and LIME (pip install scikit-learn shap lime).
import numpy as np, shap
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.model_selection import train_test_split
from lime.lime_tabular import LimeTabularExplainer
data = load_breast_cancer()
X_tr, X_te, y_tr, y_te = train_test_split(data.data, data.target, random_state=0)
model = GradientBoostingClassifier(random_state=0).fit(X_tr, y_tr)
# SHAP: exact values for trees (in log-odds units)
explainer = shap.TreeExplainer(model)
sv = explainer.shap_values(X_te) # shape (n_samples, n_features)
top = np.argsort(np.abs(sv).mean(axis=0))[::-1][:5]
print("Global top features:", [data.feature_names[i] for i in top])
i = 0
print("base + sum(phi) =", explainer.expected_value + sv[i].sum(),
" model margin =", model.decision_function(X_te[i:i+1])[0])
# LIME: local surrogate for the same instance
lime_exp = LimeTabularExplainer(X_tr, feature_names=list(data.feature_names),
class_names=list(data.target_names),
mode="classification", random_state=0)
exp = lime_exp.explain_instance(X_te[i], model.predict_proba, num_features=5)
for feat, weight in exp.as_list():
print(f"{weight:+.3f} {feat}")The SHAP check prints two equal numbers, demonstrating local accuracy. The global top five comes from averaging absolute SHAP values over the test set, as in the next diagram. LIME prints its five strongest local weights; they usually overlap with SHAP's top contributors for this instance but need not match exactly, because the methods answer slightly different questions.
A single explanation answers one customer's question; aggregated explanations describe the model. SHAP summary (beeswarm) plots show every instance's value per feature; dependence plots show how a feature's contribution changes with its value, revealing nonlinearity and interactions.
| Method | Scope | Guarantees | Cost | Best for |
|---|---|---|---|---|
| SHAP (TreeSHAP) | Local + global | Additive, consistent | Fast for trees | Tabular tree models |
| SHAP (Kernel) | Local + global | Same, approximate | Slow | Any model, small feature sets |
| LIME | Local | None formally; unstable | Moderate | Quick intuition, text and images |
| Permutation importance | Global | Measures performance drop | Moderate | Model-level feature ranking |
| Partial dependence / ICE | Global / per-instance curves | Visual | Moderate | Feature effect shapes |
| Integrated Gradients | Local | Completeness for differentiable models | Gradient passes | Deep networks |
| Inherently interpretable models | Global | Exact | Cheap | High-stakes decisions |
Pick the method from the question and the model. For tree ensembles on tabular data, TreeSHAP is the default: exact, fast and consistent between local and global views. For a deep network on images or text, gradient-based attributions (Integrated Gradients, DeepSHAP) scale better than sampling methods. LIME is useful as a quick sanity check or when you only have prediction access through an API.
When showing explanations to end users, translate them. A customer does not need log-odds; they need two or three plain reasons ("high debt-to-income ratio", "short credit history") and ideally what would change the outcome, which is the job of counterfactual explanations. Internally, store explanations with each prediction for high-stakes decisions, so an audit months later can reproduce the reasons with the exact model version and background data that were used.
Finally, validate explanations like any other output: check that attributions are stable across seeds and similar inputs, and that removing the top-attributed features actually changes the prediction.