Partial Dependence and Interaction

The previous two lessons tied the explanation either to the model as a whole or to a single subscriber. The question in between was not asked: while the value of a column is swept from one end to the other, how does the output of the model behave on average. This is partial dependence, and it summarizes the effect of a single column on the model as a curve. Here the curve is not drawn; the numbers at the grid points are printed, because the question of this lesson is precisely about what gets lost inside that average.

Partial dependence is an average, and like every average it can cancel two movements in opposite directions. If a column pushes the score up in one subgroup and down in another, the curve for the whole comes out flat, and it looks as if the column plays no role in the model. This is the measure of this lesson: in how many of the columns that look flat in the whole do the subgroups slope in opposite directions.

• IN19. The setup is that of the previous lessons: the same fictional subscriber table, seed 20260218, split 756/252/252, baseline 0.7579 on the test set. The subgroups and the reading threshold of 25 records continue; the number of subgroups examined is 13.
• IN20. This lesson changes the model being explained. The linear model does not multiply the columns by each other, so the partial dependence of a column carries the same sign in every subgroup and the question of this lesson cannot be asked there. The model being explained is the decision tree family of the Supervised Learning course; here an unpruned tree of depth 6 is used, and it gives 0.7897 on the test set. The number the family reaches with pruning in the Supervised Learning course is 0.7857.
• IN21. Partial dependence is computed as follows: one column is pulled to the value on the grid, the other columns are left at the set’s own values, and the scores are averaged. The grid is at most five points of the distinct values in the training set; for a column with fewer than five distinct values, the number of points is that many.
• IN22. The derived consumption per person is recomputed together with its source at each grid point.
• IN23. The slope is the partial dependence at the last point of the grid minus the one at the first point. A column whose absolute slope in the whole is below 0.05 counts as flat; a column whose absolute slope in a subgroup exceeds 0.02 counts as sloped. Both thresholds are on the score scale, and the score varies between 0 and 1.
• IN24. No chart is drawn; partial dependence is printed as numbers at the grid points.

Partial Dependence as Numbers

The setup is the same as in the previous lessons.

# interpret.py — MODEL: the FICTIONAL subscriber table of the first seven courses of the curriculum is rebuilt with the same seed;
# the twelve-column feature set and the split stay fixed throughout the course.
import math
import statistics

SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF
REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10),
          ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05),
          ("central", 0.18, 23, 0.05)]


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


SUBSCRIBERS, DATA = [], []
for i in range(RAW):
    r = rng(SEED + i)
    b = REGION[pick(r(), [x[1] for x in REGION])]
    household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05])
    rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17])
    if r() >= 0.046:
        SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household,
                            "satisfaction": rating})
for k in SUBSCRIBERS:
    r, v = rng(SEED + 7000 + k["id"]), []
    for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])):
        v.append(0.0 if r() < 0.038 else math.floor(
            k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62)
            * (1 - n * 0.05) * 100 + 0.5) / 100)
        r()
    if not v:
        continue
    mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"])
    z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol
         + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3]
         + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30)
    x = {"household": k["household"], "satisfaction": k["satisfaction"], "period": len(v),
         "mean_consumption": round(mean, 2), "volatility": round(vol, 2),
         "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35),
         "region_name": k["region"][0]}
    x["per_person"] = round(x["mean_consumption"] / x["household"], 3)
    for b in REGION:
        x["area_" + b[0]] = int(k["region"][0] == b[0])
    DATA.append(x)


def shuffle(data, seed):
    r, s = rng(seed), list(range(len(data)))
    for i in range(len(s) - 1, 0, -1):
        j = int(r() * (i + 1))
        s[i], s[j] = s[j], s[i]
    return [data[i] for i in s]


K = shuffle(DATA, SEED + 90000)
TRAIN, VAL, TEST = K[:756], K[756:1008], K[1008:]
FIELDS = ["mean_consumption", "volatility", "household", "satisfaction", "period",
          "zero_readings", "per_person", "area_north", "area_south", "area_east",
          "area_west", "area_central"]
BASELINE = sum(x["suspicious"] == 0 for x in TEST) / len(TEST)


def tariff_tier(x):
    return ("tariff 10" if x["mean_consumption"] <= 10 else
            "tariff 25" if x["mean_consumption"] <= 25 else "tariff 40")


def household_group(x):
    return "household 1-2" if x["household"] <= 2 else ("household 3-4" if x["household"] <= 4
                                                        else "household 5+")


def reading_group(x):
    return "readings few" if x["period"] <= 2 else "readings all"


def equals(f, value):
    return lambda x: f(x) == value


SUBGROUPS = ([("region " + b[0], equals(lambda x: x["region_name"], b[0])) for b in REGION]
             + [(t, equals(tariff_tier, t)) for t in ("tariff 10", "tariff 25", "tariff 40")]
             + [(h, equals(household_group, h)) for h in ("household 1-2", "household 3-4", "household 5+")]
             + [(o, equals(reading_group, o)) for o in ("readings few", "readings all")])
MIN_SIZE = 25
SIZES = {name: sum(1 for x in TEST if f(x)) for name, f in SUBGROUPS}
print(f"train {len(TRAIN)}, validation {len(VAL)}, test {len(TEST)}; subgroup "
      f"count {len(SUBGROUPS)}, below the threshold {[a for a in SIZES if SIZES[a] < MIN_SIZE]}")

train 756, validation 252, test 252; subgroup count 13, below the threshold ['tariff 10']

The tree is built in its form from the Supervised Learning course, and the partial dependence is printed at the grid points for five columns.

def positive_rate(s):
    return sum(x["suspicious"] for x in s) / len(s) if s else 0.0


def impurity(s):
    p = positive_rate(s)
    return 2 * p * (1 - p)


def best_split(s, min_leaf=2):
    best = None
    for a in FIELDS:
        d = sorted({x[a] for x in s})
        for v in (d[1:] if len(d) < 10 else
                  [d[int(i * len(d) / 10)] for i in range(1, 10)]):
            left = [x for x in s if x[a] < v]
            right = [x for x in s if x[a] >= v]
            if min(len(left), len(right)) < min_leaf:
                continue
            k = impurity(s) - (len(left) * impurity(left)
                               + len(right) * impurity(right)) / len(s)
            if best is None or k > best[0]:
                best = (k, a, v, left, right)
    return best


def tree(s, depth, min_leaf=2):       # MODEL: decision tree, the family of the Supervised Learning course
    p = positive_rate(s)
    best = (best_split(s, min_leaf) if depth and len(s) >= 2 * min_leaf
            and 0.0 < p < 1.0 else None)
    if best is None or best[0] <= 1e-9:
        return {"rate": p}
    return {"field": best[1], "threshold": best[2], "left": tree(best[3], depth - 1, min_leaf),
            "right": tree(best[4], depth - 1, min_leaf)}


def tree_score(d, x):
    while "rate" not in d:
        d = d["left"] if x[d["field"]] < d["threshold"] else d["right"]
    return d["rate"]


TREE = tree(TRAIN, 6)
TREE_ACCURACY = sum((tree_score(TREE, x) > 0.5) == x["suspicious"] for x in TEST) / len(TEST)
print(f"baseline {BASELINE:.4f}, test accuracy of the decision tree {TREE_ACCURACY:.4f}")

GRID_COLUMNS = ["mean_consumption", "volatility", "household", "satisfaction", "period"]


def grid(a, points=5):
    d = sorted({x[a] for x in TRAIN})
    k = min(points, len(d))
    return [d[int(i * (len(d) - 1) / (k - 1))] for i in range(k)]


def partial(rows, a, gr):             # partial dependence: as numbers, not drawn
    p = []
    for v in gr:
        t = 0.0
        for x in rows:
            d = dict(x)
            d[a] = v
            d["per_person"] = round(d["mean_consumption"] / max(1, d["household"]), 3)
            t += tree_score(TREE, d)
        p.append(t / len(rows))
    return p


FLAT, SLOPE = 0.05, {}
GRIDS = {a: grid(a) for a in GRID_COLUMNS}
for a in GRID_COLUMNS:
    p = partial(TEST, a, GRIDS[a])
    SLOPE[a] = p[-1] - p[0]
    print(f"{a:<17}" + "  ".join(f"{v:g}:{q:.4f}" for v, q in zip(GRIDS[a], p))
          + f"   slope {SLOPE[a]:+.4f}")
FLAT_COLUMNS = [a for a in GRID_COLUMNS if abs(SLOPE[a]) < FLAT]
print(f"\ncolumns that look flat in the whole (with the {FLAT} threshold) {FLAT_COLUMNS}")

baseline 0.7579, test accuracy of the decision tree 0.7897
mean_consumption 0:0.0848  15.23:0.1380  19.51:0.2234  23.81:0.2250  42.9:0.3838   slope +0.2990
volatility       0:0.1374  5.52:0.1374  9.89:0.1945  15.64:0.3158  40.57:0.7731   slope +0.6357
household        1:0.2546  2:0.2510  3:0.2274  4:0.2165  6:0.2477   slope -0.0069
satisfaction     1:0.2926  2:0.2588  3:0.2315  4:0.2354  5:0.2460   slope -0.0466
period           1:0.2491  2:0.2491  3:0.2491   slope +0.0000

columns that look flat in the whole (with the 0.05 threshold) ['household', 'satisfaction', 'period']

Mean consumption and volatility are two rows that are easy to read: along the grid the score rises in a single direction, with slopes of +0.2990 and +0.6357. The remaining three columns are flat. The slope of household size is -0.0069, that is, seven thousandths of the score; that of satisfaction is -0.0466; that of the number of readings is exactly zero.

Someone who looks at these three rows writes this sentence: the model does not base its decision on household size, satisfaction, or the number of readings. The sentence is written with the same justification for all three columns, but it does not describe the same thing in all three, and which one it describes cannot be read from the whole.

The computation itself carries an assumption, and the assumption must be written down here explicitly. While a column is pulled to a grid point, the other columns are left where they are; that is, the columns are assumed to be movable independently of each other. The mean consumption of a subscriber whose household size is pulled to 6 does not change, whereas in the fiction the two move together. The permutation importance of the previous lesson rested on the same assumption, and there two correlated columns had split the importance. Partial dependence also computes scores on combinations never seen in the set; the numbers at the ends of the grid are therefore less reliable than those in the middle.

The Interaction the Average Hides

The same computation is done separately in the twelve readable subgroups.

THRESHOLD = 0.02
SLOPES = {a: {} for a in FLAT_COLUMNS}
for name, f in SUBGROUPS:
    if SIZES[name] < MIN_SIZE:
        continue
    g = [x for x in TEST if f(x)]
    for a in FLAT_COLUMNS:
        p = partial(g, a, GRIDS[a])
        SLOPES[a][name] = p[-1] - p[0]
SHORT = [(name.replace("region ", "").replace("tariff ", "t").replace("household ", "h")
          .replace("readings ", "rd "), name) for name in SLOPES[FLAT_COLUMNS[0]]]
print("subgroup " + "".join(f"{a:>13}" for a in FLAT_COLUMNS))
print("whole    " + "".join(f"{SLOPE[a]:>+13.4f}" for a in FLAT_COLUMNS))
for short, name in SHORT:
    print(f"{short:<9}" + "".join(f"{SLOPES[a][name]:>+13.4f}" for a in FLAT_COLUMNS))
OPPOSITE = [a for a in FLAT_COLUMNS if max(SLOPES[a].values()) > THRESHOLD and min(SLOPES[a].values()) < -THRESHOLD]
print(f"\nsubgroups read {len(SLOPES[FLAT_COLUMNS[0]])}/{len(SUBGROUPS)}, slope threshold {THRESHOLD}")
print(f"columns that look flat in the whole {len(FLAT_COLUMNS)}, of which sloped in "
      f"opposite directions in the subgroups {len(OPPOSITE)}")
for a in OPPOSITE:
    hi = max(SLOPES[a], key=lambda k: SLOPES[a][k])
    lo = min(SLOPES[a], key=lambda k: SLOPES[a][k])
    print(f"  {a}: highest {hi} {SLOPES[a][hi]:+.4f}, lowest {lo} {SLOPES[a][lo]:+.4f}, "
          f"gap {SLOPES[a][hi] - SLOPES[a][lo]:.4f}")

subgroup     household satisfaction       period
whole          -0.0069      -0.0466      +0.0000
north          +0.0916      -0.1146      +0.0000
south          -0.0477      -0.0108      +0.0000
east           -0.1891      +0.0278      +0.0000
west           -0.0442      -0.0313      +0.0000
central        +0.0522      -0.0573      +0.0000
t25            -0.0264      -0.0143      +0.0000
t40            +0.1301      -0.1911      +0.0000
h1-2           -0.0354      -0.0665      +0.0000
h3-4           +0.0049      -0.0533      +0.0000
h5+            +0.0040      +0.0205      +0.0000
rd few         +0.0155      -0.0359      +0.0000
rd all         -0.0183      -0.0520      +0.0000

subgroups read 12/13, slope threshold 0.02
columns that look flat in the whole 3, of which sloped in opposite directions in the subgroups 2
  household: highest tariff 40 +0.1301, lowest region east -0.1891, gap 0.3192
  satisfaction: highest region east +0.0278, lowest tariff 40 -0.1911, gap 0.2189

Household size is flat in the whole at -0.0069. In the top tariff tier it is +0.1301, in the east region -0.1891; the gap is 0.3192, that is, forty-six times the slope in the whole. Among subscribers with high consumption, suspicion rises as the household grows; in the east region it falls as the household grows. The two movements cancel each other in the whole, and a number of seven thousandths is left behind. The flatness read in the whole is not an absence but a sum.

In satisfaction the same thing happens on a smaller scale: -0.0466 in the whole, -0.1911 in the top tariff tier, +0.0278 in the east region, a gap of 0.2189. The most extreme values of both columns stand in the same two subgroups and in opposite order; this means that the tree uses these two columns not independently but together with the consumption level and the region. This is exactly what interaction is.

The number of readings is a third kind of flatness, and it is the control of this lesson. The slope is exactly 0.0000 both in the whole and in all twelve subgroups: the tree has never placed a split on this column. Under every number that looks flat there is not an interaction; sometimes the column has not been used. The way to tell the two cases apart is not to look at the whole but to look at the subgroups, and when one does not look, the two read as the same row.

Whether the decision changes is read here too. Someone who looks at the whole proposes removing all three columns from the model. Someone who looks at the subgroups proposes this only for the number of readings; the other two are used, and it is only for whom, in which direction they are used that does not show in the whole.

Summary

• Partial dependence sweeps a column along the grid and averages the scores; here the curve was not drawn, and the numbers at the grid points were printed.
• The computation assumes that the columns can be moved independently and produces combinations never seen in the set; the numbers at the ends of the grid are less reliable than those in the middle.
• Two of the five columns clearly slope in the whole (+0.2990 and +0.6357), and three look flat.
• In two of the three columns that look flat in the whole, the subgroups slope in opposite directions: the gap is 0.3192 in household size and 0.2189 in satisfaction.
• The slope of household size is -0.0069 in the whole, +0.1301 in the third tariff tier, and -0.1891 in the east region; the flatness in the whole is not an absence but the sum of two movements in opposite directions.
• The number of readings has a slope of exactly 0.0000 both in the whole and in all twelve subgroups: not every number that looks flat hides an interaction; sometimes the column has really not been used.

Next Step

In four lessons the explanation was always produced from outside the model: the decision of a model that had already been built was read, permuted, imitated, and averaged afterward. Each time the explanation itself was an approximation, and its faithfulness had to be measured separately. There is another way: building the model readable from the start. A model that gives the decision by applying a few rules in order does not produce the explanation separately; the model itself is the explanation, and its faithfulness is complete by definition. The next lesson measures the cost of this way: with how many rules the number of the best family of the Supervised Learning course is approached, and in which subgroup it is not approached at all.