Inherently Interpretable Models

In four lessons the explanation was always produced from outside the model: the decision of a model that had already been built was read, permuted, imitated, and averaged afterward. Each time the explanation was an approximation, and its faithfulness had to be measured separately.

There is another way: building the model readable from the start. A model that gives the decision by applying a few rules in order does not produce the explanation separately — the model itself is the explanation, and its faithfulness is complete by definition. The question of this lesson is the cost of that way. The Linear Models topic of the Supervised Learning course had measured the cost once: 0.0040 between the model with twelve coefficients and the best family with thirty-three columns, that is, a single subscriber. That number is not repeated here, it is taken as input — because a rule list is far more readable than twelve coefficients, and how the cost grows with readability is the real question.

• IN25. The setup is that of the previous lessons: the same fictional subscriber table, seed 20260218, split 756/252/252, baseline 0.7579 on the test set. The subgroups and the reading threshold of 25 records continue; the number of subgroups examined is 13.
• IN26. The comparison lines come from the Supervised Learning course and are recomputed with the same seed: the linear model with twelve columns 0.8016, the best family (thirty-three columns, with a quadratic penalty) 0.8056. The number of the decision tree family from the same course is 0.7857.
• IN27. The rule list is a sequential model: the first rule that matches gives the decision, and if none matches, the last label is given. Each rule carries at most two conditions, and the two conditions cannot be from the same column.
• IN28. The list is built greedily: at each step, the rule that gains the most correct answers relative to the default label of the remaining records is added, and the records it covers are removed. The minimum coverage is 12 records, and the condition thresholds are taken from the deciles of the training set.
• IN29. “Approaching the best family” means that the test accuracy gap does not exceed 0.02; on this test set, 0.02 is five subscribers.

Three Comparison Lines

The setup is that of the previous lessons; on top of it, the best family of the Supervised Learning course is rebuilt with the same seed.

# interpret.py — MODEL: the FICTIONAL subscriber table of the first seven courses of the curriculum is rebuilt with the same seed;
# the twelve-column feature set and the split stay fixed throughout the course.
import itertools
import math
import statistics

SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF
REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10),
          ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05),
          ("central", 0.18, 23, 0.05)]


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


SUBSCRIBERS, DATA = [], []
for i in range(RAW):
    r = rng(SEED + i)
    b = REGION[pick(r(), [x[1] for x in REGION])]
    household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05])
    rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17])
    if r() >= 0.046:
        SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household,
                            "satisfaction": rating})
for k in SUBSCRIBERS:
    r, v = rng(SEED + 7000 + k["id"]), []
    for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])):
        v.append(0.0 if r() < 0.038 else math.floor(
            k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62)
            * (1 - n * 0.05) * 100 + 0.5) / 100)
        r()
    if not v:
        continue
    mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"])
    z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol
         + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3]
         + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30)
    x = {"household": k["household"], "satisfaction": k["satisfaction"], "period": len(v),
         "mean_consumption": round(mean, 2), "volatility": round(vol, 2),
         "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35),
         "region_name": k["region"][0]}
    x["per_person"] = round(x["mean_consumption"] / x["household"], 3)
    for b in REGION:
        x["area_" + b[0]] = int(k["region"][0] == b[0])
    DATA.append(x)


def shuffle(data, seed):
    r, s = rng(seed), list(range(len(data)))
    for i in range(len(s) - 1, 0, -1):
        j = int(r() * (i + 1))
        s[i], s[j] = s[j], s[i]
    return [data[i] for i in s]


K = shuffle(DATA, SEED + 90000)
TRAIN, VAL, TEST = K[:756], K[756:1008], K[1008:]
FIELDS = ["mean_consumption", "volatility", "household", "satisfaction", "period",
          "zero_readings", "per_person", "area_north", "area_south", "area_east",
          "area_west", "area_central"]
BASELINE = sum(x["suspicious"] == 0 for x in TEST) / len(TEST)


def squash(z):
    if z >= 0:
        return 1.0 / (1.0 + math.exp(-z)) if z < 700 else 1.0
    e = math.exp(z) if z > -700 else 0.0
    return e / (1.0 + e)


def scale(train, fields):
    return ({a: statistics.fmean(x[a] for x in train) for a in fields},
            {a: max(1e-9, statistics.pstdev([x[a] for x in train])) for a in fields})


def train_model(train, fields, o, s, rate=0.3, epoch=300, penalty=0.0):   # MODEL: logistic regression
    X = [[1.0] + [(x[a] - o[a]) / s[a] for a in fields] for x in train]
    Y = [x["suspicious"] for x in train]
    w = [0.0] * len(X[0])
    for _ in range(epoch):
        g = [0.0] * len(w)
        for i in range(len(X)):
            h = squash(sum(w[j] * X[i][j] for j in range(len(w)))) - Y[i]
            for j in range(len(w)):
                g[j] += h * X[i][j]
        for j in range(len(w)):
            w[j] -= rate * (g[j] / len(X) + (penalty * w[j] if j else 0.0))
    return w


def score(w, fields, o, s, x):
    return squash(w[0] + sum(w[j + 1] * (x[a] - o[a]) / s[a]
                             for j, a in enumerate(fields)))


def decision(w, fields, o, s, x):
    return int(score(w, fields, o, s, x) > 0.5)


O, S = scale(TRAIN, FIELDS)
W = train_model(TRAIN, FIELDS, O, S)
ACCURACY = sum(decision(W, FIELDS, O, S, x) == x["suspicious"] for x in TEST) / len(TEST)
print(f"train {len(TRAIN)}, validation {len(VAL)}, test {len(TEST)}")


def tariff_tier(x):
    return ("tariff 10" if x["mean_consumption"] <= 10 else
            "tariff 25" if x["mean_consumption"] <= 25 else "tariff 40")


def household_group(x):
    return "household 1-2" if x["household"] <= 2 else ("household 3-4" if x["household"] <= 4
                                                        else "household 5+")


def reading_group(x):
    return "readings few" if x["period"] <= 2 else "readings all"


def equals(f, value):
    return lambda x: f(x) == value


SUBGROUPS = ([("region " + b[0], equals(lambda x: x["region_name"], b[0])) for b in REGION]
             + [(t, equals(tariff_tier, t)) for t in ("tariff 10", "tariff 25", "tariff 40")]
             + [(h, equals(household_group, h)) for h in ("household 1-2", "household 3-4", "household 5+")]
             + [(o, equals(reading_group, o)) for o in ("readings few", "readings all")])
MIN_SIZE = 25
SIZES = {name: sum(1 for x in TEST if f(x)) for name, f in SUBGROUPS}
print(f"subgroup count {len(SUBGROUPS)}, threshold {MIN_SIZE} records, below the threshold "
      f"{[a for a in SIZES if SIZES[a] < MIN_SIZE]}")
PRODUCT = ["mean_consumption", "volatility", "household", "satisfaction", "period", "per_person"]


def expand(x):
    d = dict(x)
    for a, b in itertools.combinations(PRODUCT, 2):
        d[a + "*" + b] = x[a] * x[b]
    for a in PRODUCT:
        d[a + "^2"] = x[a] * x[a]
    return d


EXPANDED = (FIELDS + [a + "*" + b for a, b in itertools.combinations(PRODUCT, 2)]
            + [a + "^2" for a in PRODUCT])
EX_TRAIN, EX_TEST = [expand(x) for x in TRAIN], [expand(x) for x in TEST]
OE, SE = scale(EX_TRAIN, EXPANDED)
WE = train_model(EX_TRAIN, EXPANDED, OE, SE, penalty=0.05)          # the best family of the Supervised Learning course
BEST_FAMILY = sum(decision(WE, EXPANDED, OE, SE, x) == x["suspicious"] for x in EX_TEST) / len(EX_TEST)
LINEAR_ACCURACY = sum(decision(W, FIELDS, O, S, x) == x["suspicious"] for x in TEST) / len(TEST)
print(f"baseline {BASELINE:.4f}, twelve-column linear model {LINEAR_ACCURACY:.4f}, "
      f"best family ({len(EXPANDED)} columns) {BEST_FAMILY:.4f}")

train 756, validation 252, test 252
subgroup count 13, threshold 25 records, below the threshold ['tariff 10']
baseline 0.7579, twelve-column linear model 0.8016, best family (33 columns) 0.8056

As the Rule List Grows

The list is built greedily, and the length is swept from one to six.

RULE_COLUMNS = ["mean_consumption", "volatility", "household", "satisfaction", "period",
                "zero_readings", "region_name"]
SORT_KEY = {"west": 0, "east": 1, "south": 2, "north": 3, "central": 4}   # keeps the source's order of the region names, because the first of two equally good splits wins


def matches(k, x):
    a, op, e = k
    return x[a] == e if op == "=" else (x[a] < e if op == "<" else x[a] >= e)


def conditions(subset):
    for a in RULE_COLUMNS:
        d = sorted({x[a] for x in subset}, key=lambda v: SORT_KEY.get(v, v))
        if a == "region_name":
            for v in d:
                yield (a, "=", v)
        else:
            for v in (d if len(d) < 10 else
                      [d[int(i * len(d) / 10)] for i in range(1, 10)]):
                yield (a, "<", v)
                yield (a, ">=", v)


def gain(g, v0, min_cover):              # number of correct answers gained relative to the default
    if len(g) < min_cover:
        return None
    pos = sum(x["suspicious"] for x in g)
    label = int(pos * 2 > len(g))
    return ((pos if label else len(g) - pos)
            - (pos if v0 else len(g) - pos), label)


def rule_list(train, length, min_cover=12, beam=8):   # MODEL: sequential rule list
    remaining, rules = list(train), []
    for _ in range(length):
        v0 = int(sum(x["suspicious"] for x in remaining) * 2 > len(remaining))
        candidates = []
        for k1 in conditions(remaining):
            g1 = [x for x in remaining if matches(k1, x)]
            z = gain(g1, v0, min_cover)
            if z:
                p = sum(x["suspicious"] for x in g1) / len(g1)
                candidates.append((p if v0 == 0 else 1 - p, z[0], k1, g1, z[1]))
        if not candidates:
            break
        candidates.sort(key=lambda t: -t[0])
        best = max(((t[1], [t[2]], t[4]) for t in candidates[:beam]), key=lambda t: t[0])
        for _, _, k1, g1, _ in candidates[:beam]:
            for k2 in conditions(g1):
                if k2[0] == k1[0]:
                    continue
                z = gain([x for x in g1 if matches(k2, x)], v0, min_cover)
                if z and z[0] > best[0]:
                    best = (z[0], [k1, k2], z[1])
        if best[0] <= 0:
            break
        rules.append((best[1], best[2]))
        remaining = [x for x in remaining if not all(matches(k, x) for k in best[1])]
        if not remaining:
            break
    p = sum(x["suspicious"] for x in remaining)
    return rules, (int(p * 2 > len(remaining)) if remaining else 0)


def rule_predict(rules, fallback, x):
    for condition, label in rules:
        if all(matches(k, x) for k in condition):
            return label
    return fallback


def describe(condition):
    return " and ".join(f"{a} {o} {e}" for a, o, e in condition)


CLOSE = 0.02                      # five subscribers on the test set
print(f"{'rules':>6}{'conditions':>11}{'test':>9}{'gap to the best family':>24}")
SWEEP = []
for n in range(1, 7):
    rules, fallback = rule_list(TRAIN, n)
    d = sum(rule_predict(rules, fallback, x) == x["suspicious"] for x in TEST) / len(TEST)
    k = sum(len(c) for c, e in rules)
    SWEEP.append((d, -len(rules), rules, fallback, k))
    print(f"{len(rules):>6}{k:>11}{d:>9.4f}{BEST_FAMILY - d:>+24.4f}")
CHOICE = max(SWEEP, key=lambda t: (t[0], t[1]))
print(f"\nlist closest to the best family {-CHOICE[1]} rules, {CHOICE[4]} conditions, "
      f"test {CHOICE[0]:.4f}, gap {BEST_FAMILY - CHOICE[0]:+.4f} "
      f"({round((BEST_FAMILY - CHOICE[0]) * len(TEST))} subscribers)")
for condition, label in CHOICE[2]:
    print(f"  if {describe(condition)} then {label}")
print(f"  otherwise {CHOICE[3]}")

 rules conditions     test  gap to the best family
     1          2   0.7857                 +0.0198
     2          4   0.7897                 +0.0159
     3          6   0.7857                 +0.0198
     4          8   0.7857                 +0.0198
     4          8   0.7857                 +0.0198
     4          8   0.7857                 +0.0198

list closest to the best family 2 rules, 4 conditions, test 0.7897, gap +0.0159 (4 subscribers)
  if region_name = east and volatility >= 11.86 then 1
  if zero_readings >= 1 and volatility >= 20.06 then 1
  otherwise 0

A single rule with two conditions gives 0.7857; this is the number of the decision tree family pruned in the Supervised Learning course. The second rule raises the number to 0.7897, and a gap of 0.0159, that is, four subscribers, remains to the best family. The third and fourth rules do not get closer, they drop back to 0.7857; at the fifth step the greedy procedure finds no gain and stops. There is no one-directional relationship between length and accuracy.

The size of the cost is where the real reading lies. The Supervised Learning course had measured the gap for twelve coefficients as 0.0040, that is, a single subscriber; when it comes down to four conditions, the same gap rises to four subscribers. Readability is a continuous axis, and its cost grows along that axis; which point to choose is not a measurement but a decision, yet the decision can be made only once the numbers at both ends have been written down.

The list itself is read in three lines, and each line is a sentence. The faithfulness of the explanation of this model is 1 by definition, because the explanation is the model itself; the 41 empty explanations of the third lesson do not occur here.

In Which Subgroup the Best Family Cannot Be Approached

The 0.0159 measured in the whole is not read alone. The same two models are put side by side in thirteen subgroups.

LIST, FALLBACK = CHOICE[2], CHOICE[3]
print(f"{'subgroup':<16}{'records':>8}{'rule list':>15}{'best family':>13}"
      f"{'gap':>9}")
FAR, READ = [], 0
for name, f in SUBGROUPS:
    g = [i for i, x in enumerate(TEST) if f(x)]
    rl = sum(rule_predict(LIST, FALLBACK, TEST[i]) == TEST[i]["suspicious"] for i in g) / len(g)
    bf = sum(decision(WE, EXPANDED, OE, SE, EX_TEST[i]) == TEST[i]["suspicious"]
             for i in g) / len(g)
    if SIZES[name] >= MIN_SIZE:
        READ += 1
        if bf - rl > CLOSE:
            FAR.append((name, rl, bf))
    print(f"{name:<16}{len(g):>8}{rl:>15.4f}{bf:>13.4f}{bf - rl:>+9.4f}")
FAR.sort(key=lambda t: t[1] - t[2])
print(f"\nsubgroups read {READ}/{len(SUBGROUPS)}; subgroups that cannot be brought within {CLOSE} of "
      f"the best family {len(FAR)}, the farthest {FAR[0][0]} "
      f"({FAR[0][1]:.4f} instead of {FAR[0][2]:.4f}, gap {FAR[0][2] - FAR[0][1]:+.4f})")

subgroup         records      rule list  best family      gap
region north          68         0.8088       0.7794  -0.0294
region south          58         0.8621       0.8448  -0.0172
region east           36         0.6667       0.7222  +0.0556
region west           32         1.0000       1.0000  +0.0000
region central        58         0.6552       0.7414  +0.0862
tariff 10             12         0.6667       0.6667  +0.0000
tariff 25            193         0.8290       0.8342  +0.0052
tariff 40             47         0.6596       0.7234  +0.0638
household 1-2         73         0.7945       0.8356  +0.0411
household 3-4        143         0.8042       0.7972  -0.0070
household 5+          36         0.7222       0.7778  +0.0556
readings few          85         0.7529       0.7765  +0.0235
readings all         167         0.8084       0.8204  +0.0120

subgroups read 12/13; subgroups that cannot be brought within 0.02 of the best family 6, the farthest region central (0.6552 instead of 0.7414, gap +0.0862)

The gap of four subscribers in the whole is spread across the subgroups. Thirteen subgroups were examined, the 12 that passed the threshold were read, and in six of them the best family cannot be approached within 0.02. The farthest is the central region: 0.6552 against 0.7414, a gap of 0.0862 — more than five times the gap in the whole. In the top tariff tier the gap is 0.0638, in the east region and among large households 0.0556.

The reason can be read from the list itself. One of the two rules depends on the east region and the other on zero readings; the suspicious subscribers of the central region enter neither, and all of them receive the default label of the last line. Brevity is paid for by the whole, and the bill falls on some subgroups.

The opposite direction is in the same table too. In three subgroups the rule list beats the best family: by 0.0294 in the north, 0.0172 in the south, and 0.0070 among middle-sized households. The single number for the whole does not say which model is better for whom.

Summary

• The sequential list of two rules and four conditions gives 0.7897 on the test set; a gap of 0.0159, that is, four subscribers, remains to the 0.8056 of the best family.
• Extending the list does not get closer: three and four rules drop it to 0.7857, and the list saturates after the fourth rule. A single rule gives 0.7857, the number of the decision tree family.
• The cost of readability grows along that axis: one subscriber for twelve coefficients in the Supervised Learning course, four subscribers for four conditions here.
• The faithfulness of the rule list is 1 by definition; the 41 empty explanations of the third lesson do not occur here.
• Thirteen subgroups were examined and 12 were read: in six the best family cannot be approached within 0.02, the farthest in the central region at 0.0862; in three subgroups, however, the rule list beats the best family.

Next Step

In ten lessons, what a number is, for whom it is, and what it rests on were measured: the share inflated by the split, the operating point moved by the cost matrix, the choice of the baseline, the margin of the gap, the distribution shift, the number of closed questions, the disagreement of two importance rankings, the faithfulness of the explanation, the interaction the average hides, and the cost of the rule list. In all of them, what was measured was accuracy or readability.

None of them was a measure of harm. The gap of 0.0862 in the central region means that the subscribers there receive the suspicious flag wrongly more often; who receives how much of a wrong flag and what that flag does to them was never asked in ten lessons. The next topic opens with this question: where in the set the error comes from, how many separate definitions of being fair there are, whether those definitions can be satisfied at the same time, and what the model has learned leaks about individual records.