Unit 02 of 03

Interpretability

Measuring whether what the model bases its number on can be read: how many of the questions that can be asked about a decision are answered by the output and how many by an explanation, in how many ranks the importance ranking read from the model and the ranking measured by permuting disagree, and how two correlated features split the importance, how many examples the local surrogate describes the decision of the original model correctly and how two methods give two different explanations for the same example, how a partial dependence that looks flat overall slopes in opposite directions in two subgroups, where the accuracy arrives as the rule list grows, and in which subgroup the best family cannot be approached.

Published October 7, 2026 Updated October 21, 2026

Lessons

01 The Need for Interpretability Counting how many of the twelve questions that can be asked about a decision are closed by the model's output, how many by an explanation, and how many by neither: the output alone closes 4 questions, the explanation raises the average to 8.58, and three questions are closed by no explanation. The one-step change at which the decision flips exists for 159 subscribers of the test set and for 2 of the thirty-two subscribers in the west region; thirteen subgroups were examined, and among the twelve that pass the threshold the number of closed questions ranges from 8.89 at best to 7.88 at worst, a gap of 1.02 questions. 8 min read
02 Feature Importance Measuring two global importance rankings side by side on the same model: the ranking read from the coefficients and the ranking measured by permuting the column disagree on ten of the twelve columns, and the largest shift is five ranks. Volatility is third in the coefficient ranking and eighth in the permutation ranking. Two correlated columns split the importance: the importance of consumption per person is 0.0018 in the full model and rises to 0.0387 when mean consumption is removed and the model is retrained. Across thirteen subgroups, the loss increase of the most important column ranges from 0.0162 to 0.1882, and in one subgroup the first place changes. 9 min read
03 Local Explanation Methods Asking two methods that explain a single prediction about the same subscribers and measuring the faithfulness of the explanation: the local surrogate gets an average of 0.9417 of the neighborhood decisions right, but for 41 of the 252 subscribers the neighborhood holds a single label, so the explanation comes out empty. The contribution share points to a different column than the surrogate in 145 of the 211 filled explanations. Faithfulness is highest in the west region at 0.9664, where the empty-explanation rate is 0.4688; the lowest faithfulness, 0.9027, is in the top tariff tier, and the gap is 0.0637. 5 min read
04 Partial Dependence and Interaction Printing what a column does to the model as a whole as numbers and counting what the average hides: in the decision tree, three of the five columns look flat in the whole, and in two of them the subgroups slope in opposite directions. The partial dependence of household size is flat in the whole at -0.0069, +0.1301 in the third tariff tier, -0.1891 in the east region, and the gap is 0.3192. In satisfaction the gap is 0.2189. The number of readings has a slope of exactly 0.0000 both in the whole and in all twelve subgroups: not every number that looks flat hides an interaction. 9 min read
05 Inherently Interpretable Models Measuring the cost of building the model readable from the start instead of producing the explanation afterward: a sequential list of two rules and four conditions gives 0.7897 on the test set, leaving a gap of 0.0159, that is, four subscribers, to the 0.8056 of the best family with thirty-three columns. Extending the list to three and four rules does not get closer; it drops back to 0.7857. Thirteen subgroups were examined, and in six of the twelve read, the best family cannot be approached within 0.02; the farthest is the central region, 0.6552 instead of 0.7414, a gap of 0.0862. In three subgroups the rule list beats the best family. 5 min read