Interpretability & Explainability

When a model makes a consequential decision, denying a loan, flagging a medical scan, rejecting a resume, being able to explain why matters, sometimes as a matter of trust, sometimes as a genuine legal requirement. Not every model makes this equally easy.

Some models are inherently interpretable. You can look directly at how they arrived at a specific prediction. A single decision tree, from the Classical ML Toolbox module, is a clean example. You can trace the exact sequence of questions that led to a given output. Linear regression is another. Each input's weight directly tells you how much that input pushed the prediction up or down.

Other models, large neural networks especially, are much harder to interpret directly. A prediction emerges from the combined effect of millions or billions of individual weighted-sum computations, and there is no single clean explanation like "this one feature crossed this one threshold." These are often called black box models. You can observe the input and the output clearly, but the internal reasoning is not directly human-readable.

For black-box models, a family of explainability techniques has emerged that approximate an explanation after the fact. Two of them come up constantly enough to know by name.

LIME, short for Local Interpretable Model-agnostic Explanations, explains one single prediction at a time. It slightly perturbs the input in many different ways, observes how the black box's output changes each time, and fits a simple, interpretable model, usually just a linear one, to those perturbed points. The coefficients of that simple local model become the explanation for this one prediction, even though the underlying black box is nothing like linear globally.

SHAP, short for SHapley Additive exPlanations, takes a different, more mathematically grounded approach, borrowed from cooperative game theory. It treats each feature as a "player" contributing to the final prediction and computes a Shapley value for every feature: a fair share of credit or blame for that prediction, calculated by averaging that feature's effect across every possible combination of other features being present or absent. The math is heavier than LIME's, but the guarantee is stronger. Shapley values are the unique attribution that satisfies a specific set of fairness properties from game theory, which is why SHAP has become the more widely trusted default in practice.

In regulated domains, lending, healthcare, criminal justice, the choice between an interpretable model and a more accurate but harder-to-explain black-box model is a real, consequential tradeoff, not a purely technical one. Sometimes a slightly less accurate but genuinely interpretable model is the right engineering choice specifically because of this.