Local Explanation Methods

Both rankings in the previous lesson belonged to the model as a whole. When the fictional subscriber receives the suspicious flag, the question is not what the model relies on in general but what its own decision relies on. The change in the first place of the global ranking in the south region had shown that the answer for the whole may not be true for a single subscriber.

A local explanation answers this question with two separate structures. The first builds a small model that imitates the behavior of the original model around an example; the second distributes the output among the features. Both produce a column name; this lesson counts how often those two names are the same and whether the explanation really describes the original model.

• IN14. The setup is that of the previous two lessons: the same fictional subscriber table, seed 20260218, split 756/252/252, baseline 0.7579, linear model 0.8016 on the test set. The number of subgroups examined is 13, the reading threshold is 25 records.
• IN15. The local surrogate is a small model with a single split, fitted to 40 neighbors generated around the subscriber being explained. The neighbors are generated by shifting the numeric columns within a range of one standard deviation and by changing the two-valued column and the region with probability 0.2; their labels are not the true labels but the decision of the original model.
• IN16. The faithfulness of an explanation is the share of the neighborhood decisions the surrogate gets right. If the whole neighborhood carries a single label, the surrogate is constant: its faithfulness is 1 and its explanation is empty.
• IN17. The contribution share distribution starts from the base record, puts the subscriber’s columns in one by one in a random order, records the change in the score at each step, and averages over 24 orders. The base record is the mean of the training set, and its region is the training mean of the five indicators.
• IN18. The single-feature perturbation is the single-order form of the contribution share: only one column is pulled to its base value. When the two methods are compared, the first column each one gives is compared.

Two Methods, the Same Subscriber

The setup is the same as in the previous lessons.

# interpret.py — MODEL: the FICTIONAL subscriber table of the first seven courses of the curriculum is rebuilt with the same seed;
# the twelve-column feature set and the split stay fixed throughout the course.
import math
import statistics

SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF
REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10),
          ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05),
          ("central", 0.18, 23, 0.05)]


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


SUBSCRIBERS, DATA = [], []
for i in range(RAW):
    r = rng(SEED + i)
    b = REGION[pick(r(), [x[1] for x in REGION])]
    household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05])
    rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17])
    if r() >= 0.046:
        SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household,
                            "satisfaction": rating})
for k in SUBSCRIBERS:
    r, v = rng(SEED + 7000 + k["id"]), []
    for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])):
        v.append(0.0 if r() < 0.038 else math.floor(
            k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62)
            * (1 - n * 0.05) * 100 + 0.5) / 100)
        r()
    if not v:
        continue
    mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"])
    z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol
         + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3]
         + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30)
    x = {"household": k["household"], "satisfaction": k["satisfaction"], "period": len(v),
         "mean_consumption": round(mean, 2), "volatility": round(vol, 2),
         "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35),
         "region_name": k["region"][0]}
    x["per_person"] = round(x["mean_consumption"] / x["household"], 3)
    for b in REGION:
        x["area_" + b[0]] = int(k["region"][0] == b[0])
    DATA.append(x)


def shuffle(data, seed):
    r, s = rng(seed), list(range(len(data)))
    for i in range(len(s) - 1, 0, -1):
        j = int(r() * (i + 1))
        s[i], s[j] = s[j], s[i]
    return [data[i] for i in s]


K = shuffle(DATA, SEED + 90000)
TRAIN, VAL, TEST = K[:756], K[756:1008], K[1008:]
FIELDS = ["mean_consumption", "volatility", "household", "satisfaction", "period",
          "zero_readings", "per_person", "area_north", "area_south", "area_east",
          "area_west", "area_central"]
BASELINE = sum(x["suspicious"] == 0 for x in TEST) / len(TEST)


def squash(z):
    if z >= 0:
        return 1.0 / (1.0 + math.exp(-z)) if z < 700 else 1.0
    e = math.exp(z) if z > -700 else 0.0
    return e / (1.0 + e)


def scale(train, fields):
    return ({a: statistics.fmean(x[a] for x in train) for a in fields},
            {a: max(1e-9, statistics.pstdev([x[a] for x in train])) for a in fields})


def train_model(train, fields, o, s, rate=0.3, epoch=300):    # MODEL: logistic regression
    X = [[1.0] + [(x[a] - o[a]) / s[a] for a in fields] for x in train]
    Y = [x["suspicious"] for x in train]
    w = [0.0] * len(X[0])
    for _ in range(epoch):
        g = [0.0] * len(w)
        for i in range(len(X)):
            h = squash(sum(w[j] * X[i][j] for j in range(len(w)))) - Y[i]
            for j in range(len(w)):
                g[j] += h * X[i][j]
        for j in range(len(w)):
            w[j] -= rate * g[j] / len(X)
    return w


def score(w, fields, o, s, x):
    return squash(w[0] + sum(w[j + 1] * (x[a] - o[a]) / s[a]
                             for j, a in enumerate(fields)))


def decision(w, fields, o, s, x):
    return int(score(w, fields, o, s, x) > 0.5)


O, S = scale(TRAIN, FIELDS)
W = train_model(TRAIN, FIELDS, O, S)
ACCURACY = sum(decision(W, FIELDS, O, S, x) == x["suspicious"] for x in TEST) / len(TEST)
print(f"train {len(TRAIN)}, validation {len(VAL)}, test {len(TEST)}, baseline "
      f"{BASELINE:.4f}, test accuracy of the linear model {ACCURACY:.4f}")


def tariff_tier(x):
    return ("tariff 10" if x["mean_consumption"] <= 10 else
            "tariff 25" if x["mean_consumption"] <= 25 else "tariff 40")


def household_group(x):
    return "household 1-2" if x["household"] <= 2 else ("household 3-4" if x["household"] <= 4
                                                        else "household 5+")


def reading_group(x):
    return "readings few" if x["period"] <= 2 else "readings all"


def equals(f, value):
    return lambda x: f(x) == value


SUBGROUPS = ([("region " + b[0], equals(lambda x: x["region_name"], b[0])) for b in REGION]
             + [(t, equals(tariff_tier, t)) for t in ("tariff 10", "tariff 25", "tariff 40")]
             + [(h, equals(household_group, h)) for h in ("household 1-2", "household 3-4", "household 5+")]
             + [(o, equals(reading_group, o)) for o in ("readings few", "readings all")])
MIN_SIZE = 25
SIZES = {name: sum(1 for x in TEST if f(x)) for name, f in SUBGROUPS}
print(f"subgroup count {len(SUBGROUPS)}, threshold {MIN_SIZE} records, below the threshold "
      f"{[a for a in SIZES if SIZES[a] < MIN_SIZE]}")

train 756, validation 252, test 252, baseline 0.7579, test accuracy of the linear model 0.8016
subgroup count 13, threshold 25 records, below the threshold ['tariff 10']

The two methods are set up one after the other, and each produces one column name for every subscriber.

LOCAL = ["mean_consumption", "volatility", "household", "satisfaction", "period",
         "zero_readings", "region_name"]
BASE_FIELDS = {a: statistics.fmean(x[a] for x in TRAIN) for a in
               ("mean_consumption", "volatility", "household", "satisfaction", "period",
                "zero_readings")}
BASE_FIELDS["region_name"] = None    # in the base record the region is not a single region but the mean
B_MEAN = {"area_" + b[0]: statistics.fmean(x["area_" + b[0]] for x in TRAIN) for b in REGION}


def recompute(d):                    # recomputes the derived columns
    d["per_person"] = round(d["mean_consumption"] / max(1, d["household"]), 3)
    if d["region_name"] is None:
        d.update(B_MEAN)
    else:
        for b in REGION:
            d["area_" + b[0]] = int(d["region_name"] == b[0])
    return d


BASE_RECORD = recompute(dict(TEST[0], **BASE_FIELDS))


def neighborhood(x, count=40, seed=SEED + 600000):
    r, c = rng(seed + int(x["mean_consumption"] * 100) + x["household"]), []
    for _ in range(count):
        d = dict(x)
        for a in ("mean_consumption", "volatility", "satisfaction", "period"):
            d[a] = x[a] + (2 * r() - 1) * S[a]
        d["household"] = max(1, round(x["household"] + (2 * r() - 1) * S["household"]))
        if r() < 0.2:
            d["zero_readings"] = 1 - x["zero_readings"]
        if r() < 0.2:
            d["region_name"] = REGION[int(r() * 5)][0]
        c.append(recompute(d))
    return c


SORT_KEY = {"west": 0, "east": 1, "south": 2, "north": 3, "central": 4}   # keeps the source's order of the region names, because the first of two equally good splits wins


def surrogate(x):                    # MODEL: single-split decision tree, local surrogate
    c = neighborhood(x)
    y = [decision(W, FIELDS, O, S, k) for k in c]
    if len(set(y)) == 1:
        return None, 1.0             # the whole neighborhood has a single label: the explanation is empty
    best = (0.0, None)
    for a in LOCAL:
        d = sorted({k[a] for k in c}, key=lambda v: SORT_KEY.get(v, v))
        candidates = d[1:] if a in ("zero_readings", "region_name") else \
            [d[int(i * len(d) / 8)] for i in range(1, 8)]
        for v in candidates:
            left = {i for i in range(len(c))
                    if (c[i][a] != v if a == "region_name" else c[i][a] < v)}
            if not left or len(left) == len(c):
                continue
            pred_left = sum(y[i] for i in left) * 2 > len(left)
            pred_right = sum(y[i] for i in range(len(c)) if i not in left) * 2 > len(c) - len(left)
            hit = sum((pred_left if i in left else pred_right) == y[i] for i in range(len(c))) / len(c)
            if hit > best[0]:
                best = (hit, a)
    return best[1], best[0]


def contribution(x, epoch=24, seed=SEED + 700000):
    r = rng(seed + int(x["mean_consumption"] * 100) + x["household"])
    share = {a: 0.0 for a in LOCAL}
    for _ in range(epoch):
        order = list(LOCAL)
        for i in range(len(order) - 1, 0, -1):
            j = int(r() * (i + 1))
            order[i], order[j] = order[j], order[i]
        d, previous = dict(BASE_RECORD), score(W, FIELDS, O, S, BASE_RECORD)
        for a in order:
            d[a] = x[a]
            new = score(W, FIELDS, O, S, recompute(d))
            share[a] += new - previous
            previous = new
    return {a: share[a] / epoch for a in LOCAL}


def single_perturbation(x):          # single perturbation: one column is pulled to the base value
    p0 = score(W, FIELDS, O, S, x)
    return {a: p0 - score(W, FIELDS, O, S, recompute(dict(x, **{a: BASE_FIELDS[a]})))
            for a in LOCAL}


FAITHFULNESS, SUR_COL, CONTRIB_COL, SINGLE_COL = [], [], [], []
for x in TEST:
    a, u = surrogate(x)
    p, t = contribution(x), single_perturbation(x)
    FAITHFULNESS.append(u)
    SUR_COL.append(a)
    CONTRIB_COL.append(max(LOCAL, key=lambda k: abs(p[k])))
    SINGLE_COL.append(max(LOCAL, key=lambda k: abs(t[k])))
FILLED = [i for i in range(len(TEST)) if SUR_COL[i]]
DIFFER = [i for i in FILLED if SUR_COL[i] != CONTRIB_COL[i]]
print(f"mean faithfulness {sum(FAITHFULNESS) / len(FAITHFULNESS):.4f}; subscribers whose "
      f"neighborhood holds a single label, that is, who get an empty explanation {len(TEST) - len(FILLED)}")
print(f"filled explanations {len(FILLED)}, mean faithfulness among them "
      f"{sum(FAITHFULNESS[i] for i in FILLED) / len(FILLED):.4f}")
print(f"subscribers for whom the surrogate and the contribution share do not give the same column "
      f"{len(DIFFER)}/{len(FILLED)} ({len(DIFFER) / len(FILLED):.4f})")
print(f"subscribers for whom the contribution share and the single perturbation disagree "
      f"{sum(1 for i in range(len(TEST)) if CONTRIB_COL[i] != SINGLE_COL[i])}/{len(TEST)}")
i = min(DIFFER, key=lambda i: abs(score(W, FIELDS, O, S, TEST[i]) - 0.5))
x, p = TEST[i], contribution(TEST[i])
print(f"\nexample subscriber: region {x['region_name']}, mean consumption {x['mean_consumption']}, "
      f"zero readings {x['zero_readings']}, score {score(W, FIELDS, O, S, x):.4f}")
print(f"  surrogate split column {SUR_COL[i]}, faithfulness {FAITHFULNESS[i]:.4f}")
print("  contribution shares " + ", ".join(f"{a} {p[a]:+.4f}" for a in
                                           sorted(LOCAL, key=lambda a: -abs(p[a]))[:3]))

mean faithfulness 0.9417; subscribers whose neighborhood holds a single label, that is, who get an empty explanation 41
filled explanations 211, mean faithfulness among them 0.9303
subscribers for whom the surrogate and the contribution share do not give the same column 145/211 (0.6872)
subscribers for whom the contribution share and the single perturbation disagree 27/252

example subscriber: region east, mean consumption 24.67, zero readings 0, score 0.4885
  surrogate split column mean_consumption, faithfulness 0.7750
  contribution shares region_name +0.1631, mean_consumption +0.1335, volatility +0.0428

The example subscriber gets two explanations, and they say different things. The surrogate says that what separates the decision around that subscriber is mean consumption, and it gets 0.7750 of the neighborhood decisions right. The contribution share distribution says that the largest share that carries the subscriber from the base record to this score belongs to the region: being in the east region added 0.1631 to the score. Both are correct, and they answer two separate questions: the first is “which column leads out of here”, the second is “which column brought it here”. In 145 of the 211 filled explanations they do not give the same column. The single-feature perturbation, the single-order form of the contribution share, disagrees with it for 27 subscribers: what a column contributes depends on which columns were put in before it.

Faithfulness Is Not Read Alone

The mean faithfulness looks high at 0.9417. Inside this number are 41 subscribers whose neighborhood carries a single label: the surrogate is constant, its faithfulness is 1 by definition, and it says nothing at all. When the empty explanations are left out, faithfulness falls to 0.9303.

print(f"{'subgroup':<16}{'records':>8}{'faithfulness':>14}{'empty explanation':>19}{'disagreement':>14}")
READ = []
for name, f in SUBGROUPS:
    g = [i for i, x in enumerate(TEST) if f(x)]
    faith = sum(FAITHFULNESS[i] for i in g) / len(g)
    empty = sum(1 for i in g if SUR_COL[i] is None) / len(g)
    d = [i for i in g if SUR_COL[i]]
    dis = sum(1 for i in d if SUR_COL[i] != CONTRIB_COL[i]) / len(d)
    if SIZES[name] >= MIN_SIZE:
        READ.append((name, faith, dis))
    print(f"{name:<16}{len(g):>8}{faith:>14.4f}{empty:>19.4f}{dis:>14.4f}")
highest, lowest = max(READ, key=lambda t: t[1]), min(READ, key=lambda t: t[1])
top_dis = max(READ, key=lambda t: t[2])
print(f"\nsubgroups read {len(READ)}/{len(SUBGROUPS)}")
print(f"faithfulness highest {highest[0]} {highest[1]:.4f}, lowest {lowest[0]} {lowest[1]:.4f}, "
      f"gap {highest[1] - lowest[1]:.4f}")
print(f"disagreement highest {top_dis[0]} {top_dis[2]:.4f}, lowest "
      f"{min(READ, key=lambda t: t[2])[0]} {min(t[2] for t in READ):.4f}")

subgroup         records  faithfulness  empty explanation  disagreement
region north          68        0.9401             0.0441        0.7231
region south          58        0.9586             0.2414        0.7273
region east           36        0.9028             0.1389        0.7097
region west           32        0.9664             0.4688        0.4118
region central        58        0.9371             0.0690        0.6852
tariff 10             12        0.9688             0.5833        1.0000
tariff 25            193        0.9495             0.1503        0.7012
tariff 40             47        0.9027             0.1064        0.5952
household 1-2         73        0.9538             0.2603        0.6852
household 3-4        143        0.9320             0.1329        0.7097
household 5+          36        0.9556             0.0833        0.6061
readings few          85        0.9456             0.2353        0.8154
readings all         167        0.9397             0.1257        0.6301

subgroups read 12/13
faithfulness highest region west 0.9664, lowest tariff 40 0.9027, gap 0.0637
disagreement highest readings few 0.8154, lowest region west 0.4118

Thirteen subgroups were examined, and 12 were read. The subgroup with the highest faithfulness is the west region, 0.9664; on the same row the empty-explanation rate is 0.4688, which means fifteen of the thirty-two subscribers get an explanation that says nothing. Someone who reports faithfulness alone writes the west down as the best-explained group; when the two columns are put side by side, that sentence turns around. Faithfulness assumes that the explanation says something; when it says nothing, it still comes out high.

The lowest faithfulness, 0.9027, is in the top tariff tier, and the gap is 0.0637. Because subscribers with high consumption cluster near the decision boundary, their neighborhoods carry two labels and a single split is not enough; the same group is the one where the explanations are fullest. The disagreement of the two methods varies between 0.4118 and 0.8154 across the subgroups.

Summary

• The local surrogate gets an average of 0.9417 of the neighborhood decisions right; when the empty explanations are left out, it falls to 0.9303.
• For 41 of the 252 subscribers the neighborhood carries a single label: the surrogate is constant, its faithfulness is 1, and its explanation is empty. This is why faithfulness is not read alone.
• The surrogate and the contribution share distribution point to different columns in 145 of the 211 filled explanations; the two answer the questions “which column leads out of here” and “which column brought it here”.
• Thirteen subgroups were examined and 12 were read: faithfulness is between 0.9027 and 0.9664, the gap is 0.0637; in the group with the highest faithfulness, the empty-explanation rate is 0.4688.

Next Step

In this lesson the explanations belonged to individual subscribers. There is another way to see what a column does to the model as a whole: the value of the column is swept from one end to the other while the other columns are left at the set’s own values, and one looks at how the mean score changes. This is partial dependence, and it is usually drawn as a curve. Here it is not drawn, the numbers are printed; when the numbers are printed, what the average hides becomes visible. The next lesson counts, in how many columns it happens, that a partial dependence that looks flat for the whole can slope in opposite directions in two subgroups.