Feature Importance

The previous lesson said that three of the five explanation questions close for every subscriber, and the reason was that the linear model’s score can be decomposed into one term per column. Ranking those terms by size is read as the global explanation of which column the model relies on: it is an answer that belongs not to a single subscriber but to the model as a whole.

This ranking itself is not new. The Feature Selection topic of the Data Preparation and Feature Engineering course used importance as a selection criterion, and the Linear Models topic of the Supervised Learning course measured that the value of a coefficient depends on the set of columns beside it. Neither is repeated here. The question of this lesson is different and takes the answers of the earlier ones as input: importance is not measured in a single way. Reading a column from the model gives one ranking; permuting the column and looking at how much performance drops gives a second ranking. There is no reason for the two rankings to come out the same, and how far they disagree can be measured.

• IN8. The setup is that of the previous lesson: the same fictional subscriber table, seed 20260218, split 756/252/252, baseline 0.7579, the twelve-column linear model at 0.8016 on the test set. The subgroups and the 25-record reading threshold continue unchanged; the number of subgroups examined is 13.
• IN9. Importance read from the model is the ranking by the absolute value of the coefficients of the scaled columns. How this ranking is produced was measured in the earlier courses; here it is taken as input.
• IN10. Importance measured by permuting is the shuffling of a single column’s values among themselves within the test set and the measurement of the drop in performance. Shuffling changes the row order, not the set of values of the column; the model is not retrained.
• IN11. The drop is measured not in accuracy but in mean loss. On a set of 252 records, accuracy changes in steps of 0.004 and a single subscriber can move the ranking; loss is a continuous number and can be read in small subgroups too.
• IN12. Shuffling is done with 10 repeats and averaged; the seeds are derived from 20260218.
• IN13. An importance ranking shows what the model relies on; it does not show a causal effect. The limit of causal interpretation had been measured separately in the Applied Statistics topic of the Data Analytics curriculum.

Two Rankings

The setup is the same as in the previous lesson, and the subgroups are defined in the same place.

# interpret.py — MODEL: the FICTIONAL subscriber table of the first seven courses of the curriculum is rebuilt with the same seed;
# the twelve-column feature set and the split stay fixed throughout the course.
import math
import statistics

SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF
REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10),
          ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05),
          ("central", 0.18, 23, 0.05)]


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


SUBSCRIBERS, DATA = [], []
for i in range(RAW):
    r = rng(SEED + i)
    b = REGION[pick(r(), [x[1] for x in REGION])]
    household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05])
    rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17])
    if r() >= 0.046:
        SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household,
                            "satisfaction": rating})
for k in SUBSCRIBERS:
    r, v = rng(SEED + 7000 + k["id"]), []
    for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])):
        v.append(0.0 if r() < 0.038 else math.floor(
            k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62)
            * (1 - n * 0.05) * 100 + 0.5) / 100)
        r()
    if not v:
        continue
    mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"])
    z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol
         + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3]
         + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30)
    x = {"household": k["household"], "satisfaction": k["satisfaction"], "period": len(v),
         "mean_consumption": round(mean, 2), "volatility": round(vol, 2),
         "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35),
         "region_name": k["region"][0]}
    x["per_person"] = round(x["mean_consumption"] / x["household"], 3)
    for b in REGION:
        x["area_" + b[0]] = int(k["region"][0] == b[0])
    DATA.append(x)


def shuffle(data, seed):
    r, s = rng(seed), list(range(len(data)))
    for i in range(len(s) - 1, 0, -1):
        j = int(r() * (i + 1))
        s[i], s[j] = s[j], s[i]
    return [data[i] for i in s]


K = shuffle(DATA, SEED + 90000)
TRAIN, VAL, TEST = K[:756], K[756:1008], K[1008:]
FIELDS = ["mean_consumption", "volatility", "household", "satisfaction", "period",
          "zero_readings", "per_person", "area_north", "area_south", "area_east",
          "area_west", "area_central"]
BASELINE = sum(x["suspicious"] == 0 for x in TEST) / len(TEST)


def squash(z):
    if z >= 0:
        return 1.0 / (1.0 + math.exp(-z)) if z < 700 else 1.0
    e = math.exp(z) if z > -700 else 0.0
    return e / (1.0 + e)


def scale(train, fields):
    return ({a: statistics.fmean(x[a] for x in train) for a in fields},
            {a: max(1e-9, statistics.pstdev([x[a] for x in train])) for a in fields})


def train_model(train, fields, o, s, rate=0.3, epoch=300):    # MODEL: logistic regression
    X = [[1.0] + [(x[a] - o[a]) / s[a] for a in fields] for x in train]
    Y = [x["suspicious"] for x in train]
    w = [0.0] * len(X[0])
    for _ in range(epoch):
        g = [0.0] * len(w)
        for i in range(len(X)):
            h = squash(sum(w[j] * X[i][j] for j in range(len(w)))) - Y[i]
            for j in range(len(w)):
                g[j] += h * X[i][j]
        for j in range(len(w)):
            w[j] -= rate * g[j] / len(X)
    return w


def score(w, fields, o, s, x):
    return squash(w[0] + sum(w[j + 1] * (x[a] - o[a]) / s[a]
                             for j, a in enumerate(fields)))


def decision(w, fields, o, s, x):
    return int(score(w, fields, o, s, x) > 0.5)


O, S = scale(TRAIN, FIELDS)
W = train_model(TRAIN, FIELDS, O, S)
ACCURACY = sum(decision(W, FIELDS, O, S, x) == x["suspicious"] for x in TEST) / len(TEST)
print(f"train {len(TRAIN)}, validation {len(VAL)}, test {len(TEST)}")
print(f"baseline {BASELINE:.4f}, test accuracy of the linear model {ACCURACY:.4f}")


def tariff_tier(x):
    return ("tariff 10" if x["mean_consumption"] <= 10 else
            "tariff 25" if x["mean_consumption"] <= 25 else "tariff 40")


def household_group(x):
    return "household 1-2" if x["household"] <= 2 else ("household 3-4" if x["household"] <= 4
                                                        else "household 5+")


def reading_group(x):
    return "readings few" if x["period"] <= 2 else "readings all"


def equals(f, value):
    return lambda x: f(x) == value


SUBGROUPS = ([("region " + b[0], equals(lambda x: x["region_name"], b[0])) for b in REGION]
             + [(t, equals(tariff_tier, t)) for t in ("tariff 10", "tariff 25", "tariff 40")]
             + [(h, equals(household_group, h)) for h in ("household 1-2", "household 3-4", "household 5+")]
             + [(o, equals(reading_group, o)) for o in ("readings few", "readings all")])
MIN_SIZE = 25
SIZES = {name: sum(1 for x in TEST if f(x)) for name, f in SUBGROUPS}
print(f"subgroup count {len(SUBGROUPS)}, threshold {MIN_SIZE} records, below the threshold "
      f"{[a for a in SIZES if SIZES[a] < MIN_SIZE]}")

train 756, validation 252, test 252
baseline 0.7579, test accuracy of the linear model 0.8016
subgroup count 13, threshold 25 records, below the threshold ['tariff 10']

Importance measured by permuting is a single operation: the values of a column are shuffled among themselves within the test set, so the column keeps its own distribution but loses its link to the row; the model is measured again on that corrupted set, and the increase in loss is looked at.

def loss(w, fields, o, s, rows):     # accuracy changes in steps of 0.004 on 252 records
    t = 0.0
    for x in rows:
        p = min(max(score(w, fields, o, s, x), 1e-9), 1 - 1e-9)
        t -= math.log(p) if x["suspicious"] else math.log(1 - p)
    return t / len(rows)


def permute(w, fields, o, s, rows, permuted, repeats=10, seed=SEED + 400000):
    l0, t = loss(w, fields, o, s, rows), 0.0
    for r in range(repeats):
        order = shuffle(list(range(len(rows))), seed + 977 * r)
        new = []
        for i, x in enumerate(rows):
            d = dict(x)
            for a in permuted:
                d[a] = rows[order[i]][a]
            new.append(d)
        t += loss(w, fields, o, s, new)
    return t / repeats - l0


PERM = {a: permute(W, FIELDS, O, S, TEST, [a]) for a in FIELDS}
FROM_MODEL = sorted(FIELDS, key=lambda a: -abs(W[FIELDS.index(a) + 1]))
BY_PERMUTE = sorted(FIELDS, key=lambda a: -PERM[a])
print(f"{'column':<18}{'coefficient':>13}{'model rank':>11}{'loss increase':>15}"
      f"{'permutation rank':>18}{'shift':>7}")
disagree = 0
for a in FROM_MODEL:
    ra, rb = FROM_MODEL.index(a) + 1, BY_PERMUTE.index(a) + 1
    disagree += ra != rb
    print(f"{a:<18}{W[FIELDS.index(a) + 1]:>+13.3f}{ra:>11}{PERM[a]:>15.4f}"
          f"{rb:>18}{rb - ra:>+7}")
print(f"\nranks that disagree {disagree}/{len(FIELDS)}, largest shift "
      f"{max(abs(FROM_MODEL.index(a) - BY_PERMUTE.index(a)) for a in FIELDS)}")
print(f"top three of the model ranking {FROM_MODEL[:3]}")
print(f"top three of the permutation ranking {BY_PERMUTE[:3]}")

column              coefficient model rank  loss increase  permutation rank  shift
mean_consumption         +0.772          1         0.1332                 1     +0
zero_readings            +0.684          2         0.0564                 2     +0
volatility               +0.380          3         0.0012                 8     +5
area_east                +0.306          4         0.0146                 3     -1
satisfaction             -0.273          5         0.0016                 7     +2
household                +0.156          6         0.0053                 5     -1
period                   +0.148          7        -0.0004                10     +3
area_west                -0.147          8         0.0054                 4     -4
area_south               -0.109          9        -0.0011                11     +2
per_person               +0.071         10         0.0018                 6     -4
area_central             -0.048         11        -0.0014                12     +1
area_north               -0.005         12         0.0002                 9     -3

ranks that disagree 10/12, largest shift 5
top three of the model ranking ['mean_consumption', 'zero_readings', 'volatility']
top three of the permutation ranking ['mean_consumption', 'zero_readings', 'area_east']

Ten of the twelve columns stand in different places in the two rankings. The first two ranks are shared; the disagreement begins from the third rank. The largest shift is in volatility: third in the coefficient ranking, eighth in the permutation ranking. With 0.380 the coefficient carries a respectable size, but shuffling the column raises the loss by only 0.0012. The two numbers do not contradict each other; they say two different things. The coefficient says what an increase of one standard deviation in volatility adds to the score when the other columns are held constant. Permutation, on the other hand, measures whether the information volatility carries also exists in other columns: because volatility moves together with mean consumption, the model closes the gap when it is shuffled.

Three columns have a negative sign on their loss increase. If shuffling a column has lowered the loss, that column is harming the model; since their magnitudes are 0.0014 and below, what is to be read here is not the sign itself but that the three columns stand near zero.

How Two Correlated Columns Split the Importance

Mean consumption and consumption per person are two forms of the same quantity; the second is produced by dividing the first by household size. In the permutation ranking one is first and the other is sixth.

PAIR = ["mean_consumption", "per_person"]
together = permute(W, FIELDS, O, S, TEST, PAIR)
print(f"one by one {PERM[PAIR[0]]:.4f} and {PERM[PAIR[1]]:.4f}, total "
      f"{PERM[PAIR[0]] + PERM[PAIR[1]]:.4f}, together {together:.4f}")
for dropped in PAIR:
    remaining = [a for a in FIELDS if a != dropped]
    o2, s2 = scale(TRAIN, remaining)
    w2 = train_model(TRAIN, remaining, o2, s2)
    acc = sum(decision(w2, remaining, o2, s2, x) == x["suspicious"] for x in TEST) / len(TEST)
    other = PAIR[1] if dropped == PAIR[0] else PAIR[0]
    print(f"{dropped} removed: test {acc:.4f}, {other} importance "
          f"{permute(w2, remaining, o2, s2, TEST, [other]):.4f}")

one by one 0.1332 and 0.0018, total 0.1350, together 0.1437
mean_consumption removed: test 0.7778, per_person importance 0.0387
per_person removed: test 0.8016, mean_consumption importance 0.1441

The importance of consumption per person is 0.0018 in the full model, and someone looking at this number concludes that the column is standing there for nothing. When mean consumption is removed from the model and the model is retrained, the importance of the same column rises to 0.0387, that is, twenty-one times. The column did not change; the model changed. Importance is not a property of a column but of a column in a particular model.

Shuffling the two together gives 0.1437; the sum of the one-by-one shufflings is 0.1350. The difference, 0.0087, is the share the two carry redundantly: because the other fills the gap when each is shuffled alone, that share does not show up in any single measure. An importance ranking always loses a share between correlated columns.

The Most Important Column in the Subgroups

The overall ranking is the ranking of a set, and that set is made up of subgroups.

print(f"{'subgroup':<16}{'records':>8}{'most important column':>23}{'loss increase':>15}"
      f"{'second column':>18}")
READ = []
for name, f in SUBGROUPS:
    g = [x for x in TEST if f(x)]
    d = {a: permute(W, FIELDS, O, S, g, [a]) for a in FIELDS}
    order = sorted(FIELDS, key=lambda a: -d[a])
    if SIZES[name] >= MIN_SIZE:
        READ.append((name, order[0], d[order[0]]))
    print(f"{name:<16}{len(g):>8}{order[0]:>23}{d[order[0]]:>15.4f}{order[1]:>18}")
highest, lowest = max(READ, key=lambda t: t[2]), min(READ, key=lambda t: t[2])
print(f"\nsubgroups read {len(READ)}/{len(SUBGROUPS)}; subgroups with a column other than "
      f"{BY_PERMUTE[0]} in first place "
      f"{sum(1 for t in READ if t[1] != BY_PERMUTE[0])}")
print(f"highest {highest[0]} {highest[2]:.4f}, lowest {lowest[0]} {lowest[2]:.4f}, "
      f"gap {highest[2] - lowest[2]:.4f}")

subgroup         records  most important column  loss increase     second column
region north          68       mean_consumption         0.0719     zero_readings
region south          58          zero_readings         0.1882  mean_consumption
region east           36       mean_consumption         0.1753      satisfaction
region west           32       mean_consumption         0.0162     zero_readings
region central        58       mean_consumption         0.1879     zero_readings
tariff 10             12          zero_readings         0.1288         area_west
tariff 25            193       mean_consumption         0.0408     zero_readings
tariff 40             47       mean_consumption         0.1033        volatility
household 1-2         73       mean_consumption         0.1339     zero_readings
household 3-4        143       mean_consumption         0.1215     zero_readings
household 5+          36       mean_consumption         0.1513     zero_readings
readings few          85       mean_consumption         0.1213     zero_readings
readings all         167       mean_consumption         0.1181     zero_readings

subgroups read 12/13; subgroups with a column other than mean_consumption in first place 1
highest region south 0.1882, lowest region west 0.0162, gap 0.1720

Thirteen subgroups were examined, and the 12 that pass the threshold were read. In eleven of the twelve subgroups read, the first rank is mean consumption; in the south region the first rank moves to the zero-reading indicator, and its loss increase of 0.1882 is above the overall 0.1332. The sentence “the model relies most on mean consumption,” written for the whole, is wrong when asked of the subscribers of the south region.

The gap is sharper still: the loss increase of the most important column is 0.1882 in the south region and 0.0162 in the west region — an eleven-fold difference between them. In the west region, shuffling no column raises the loss to any notable degree, because the subscribers there stand far from the decision boundary and their scores are insensitive to column movements. The counterfactual count of the previous lesson had also collapsed in the same region; two separate measures point at the same fact from two separate places.

Summary

• For the same model, the ranking read from the coefficients and the ranking measured by permuting disagree on ten of the twelve columns; the largest shift is five ranks and it is in volatility.
• The coefficient measures what a column adds when the other columns are held constant, and permutation measures whether that column’s information exists in other columns; the coefficient of volatility is 0.380 while its loss increase is 0.0012.
• Importance is a property not of a column but of the column’s position in a particular model: the importance of consumption per person rises from 0.0018 to 0.0387 when mean consumption is removed and the model is retrained.
• Shuffling the two correlated columns together gives 0.1437 and the sum of the one-by-one shufflings gives 0.1350; the 0.0087 between them is the redundant share that shows up in no single measure.
• Thirteen subgroups were examined and 12 were read: in eleven the first rank is mean consumption, in the south region it is the zero-reading indicator; the loss increase of the most important column ranges from 0.0162 to 0.1882, and the gap is 0.1720.

Next Step

Both rankings belong to the whole model. When the flag of suspicion arrives, the subscriber in the setup asks not what the model relies on in general but what their own decision relies on, and the global ranking does not answer that question: the change of first place in the south region showed exactly this. The next lesson moves on to methods that explain a single prediction. A small surrogate model that imitates the behavior of the original model around an example is built, the output is distributed among the features to compute contribution shares, and the two methods are asked about the same subscriber. What will be measured is the faithfulness of the explanation: in how many examples the surrogate model describes the decision of the original model correctly, and how many times the two methods give two different explanations for the same subscriber.