--- title: 'Coefficient Interpretation' source: 'https://academia.sh/en/courses/supervised-learning/coefficient-interpretation' course: 'Supervised Learning' language: en updated: '2026-10-21T05:03:30+00:00' license: 'CC BY-SA 4.0' objectives: - 'Measures the held-out numbers of a model with readable coefficients and of the best family in this topic side by side and writes the difference in subscribers' - 'Reads which ranking the scaled coefficients give and compares that ranking with the generating structure in the fiction' - 'Counts, by removing columns and applying a penalty, that many different coefficient sets give the same held-out number' --- # Coefficient Interpretation Treating readability as a number rather than a praise: the readable model chosen from eight candidates on the validation set carries twelve coefficients and gives 0.8016 on the test set, the best family chosen from the same set gives 0.8056 with 33 columns, and the price of interpretability is 0.0040, that is, a single subscriber. The twelve coefficients give the order of the generating structure in the fiction correctly. The same coefficients are not fixed: when the per-person consumption column is removed, the coefficient of mean consumption rises from 0.772 to 0.808 and the test number does not change at all; when mean consumption is removed, the coefficient of per-person consumption jumps from 0.071 to 0.442, that is, to six times. The ridge penalty brings the coefficients down from 0.772 to 0.326 while the test number falls by only 0.0079. The model of the previous lesson carried twelve coefficients, and those coefficients are readable: each belongs to a column and has a sign and a magnitude. This is the most frequently cited advantage of linear models, and it is usually written down as a praise. This lesson treats it not as a praise but as **a number**: how many subscribers separate the held-out number of a model with readable coefficients from the number of the best family in this topic. The second question is separate from the first and harder. What has been said when a coefficient is read. The Applied Statistics topic of the Statistical Analysis and Exploratory Data Analysis course had measured the **causal** face of this question: a one-variable coefficient changing sign when a confounding variable is added, and the within-group slopes turning against the aggregated slope, were shown there with numbers. It is not repeated here. The question here is specific to the prediction target: **how many different coefficient sets give the same held-out number**, and how much the value of a coefficient depends on which of those sets was chosen. - **LM36.** The set, the split, the seed, the target and the fixed twelve-column feature set are as in the previous lesson; the baseline is 0.7579 on the test set. - **LM37.** All candidates are learned with **the same training setting**: learning rate 0.3, 300 epochs. The tuning sweep of the previous lesson is not repeated; the only thing that changes in this lesson is **the column set**. - **LM38.** A readable candidate is a model whose coefficients can be listed one by one and interpreted: the candidates with one, two, four and twelve columns. - **LM39.** The expanded candidate is the model raised to 33 columns with the pairwise products and the squares of the six numeric columns. Its coefficients can be listed but not read one by one, because the same magnitude is spread across several columns. - **LM40.** The ridge penalty is applied to the expanded candidate at four levels. The difference between the penalty forms was measured in the previous lessons; only the ridge penalty is used here. - **LM41.** The number of candidates tried is **8**: four readable candidates and four expanded candidates with a penalty. The choice is read from the validation set. - **LM42.** The coefficients are read on **scaled columns**, so they are comparable with one another. A coefficient in raw units is the change per cubic meter and cannot be ranked, because the columns have different units. - **LM43.** This lesson does not deal with **for whom** a coefficient is read, what happens in which subgroup, or how interpretability is built as a method; those questions belong to the Model Evaluation, Interpretability and Ethics course. Here interpretability is only the score paid. ## The Price of Interpretability The setup is the setup of the previous lesson. ```python # interpret.py — MODEL: the same FICTIONAL subscriber table, the same seed, the same split and # the same twelve-column feature set; the target is the suspicious label. import math import statistics SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10), ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05), ("central", 0.18, 23, 0.05)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 SUBSCRIBERS, DATA = [], [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() >= 0.046: SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household, "satisfaction": rating}) for k in SUBSCRIBERS: r, v = rng(SEED + 7000 + k["id"]), [] for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): v.append(0.0 if r() < 0.038 else math.floor( k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - n * 0.05) * 100 + 0.5) / 100) r() if not v: continue mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"]) z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3] + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30) x = {"household": k["household"], "satisfaction": k["satisfaction"], "period": len(v), "mean_consumption": round(mean, 2), "volatility": round(vol, 2), "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35)} x["per_person"] = round(x["mean_consumption"] / x["household"], 3) for b in REGION: x["area_" + b[0]] = int(k["region"][0] == b[0]) DATA.append(x) def shuffle(data, seed): r, s = rng(seed), list(range(len(data))) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return [data[i] for i in s] K = shuffle(DATA, SEED + 90000) TRAIN, VAL, TEST = K[:756], K[756:1008], K[1008:] FIELDS = ["mean_consumption", "volatility", "household", "satisfaction", "period", "zero_readings", "per_person", "area_north", "area_south", "area_east", "area_west", "area_central"] BASELINE = sum(x["suspicious"] == 0 for x in TEST) / len(TEST) print(f"train {len(TRAIN)}, validation {len(VAL)}, test {len(TEST)}, " f"baseline {BASELINE:.4f}") ``` ``` train 756, validation 252, test 252, baseline 0.7579 ``` The eight candidates are ranked in a single ladder: readable models from one column to twelve columns, then the expanded model with 33 columns at four penalty levels. ```python import itertools def squash(z): if z >= 0: return 1.0 / (1.0 + math.exp(-z)) if z < 700 else 1.0 e = math.exp(z) if z > -700 else 0.0 return e / (1.0 + e) PRODUCTS = ["mean_consumption", "volatility", "household", "satisfaction", "period", "per_person"] def expand(x): d = dict(x) for a, b in itertools.combinations(PRODUCTS, 2): d[a + "*" + b] = x[a] * x[b] for a in PRODUCTS: d[a + "^2"] = x[a] * x[a] return d EXPANDED = FIELDS + [a + "*" + b for a, b in itertools.combinations(PRODUCTS, 2)] \ + [a + "^2" for a in PRODUCTS] EX_TRAIN, EX_VAL, EX_TEST = [[expand(x) for x in s] for s in (TRAIN, VAL, TEST)] def scale(train, cols): return ({a: statistics.fmean(x[a] for x in train) for a in cols}, {a: max(1e-9, statistics.pstdev([x[a] for x in train])) for a in cols}) def train_model(train, cols, mu, sd, rate=0.3, epochs=300, penalty=0.0): # MODEL: logistic regression X = [[1.0] + [(x[a] - mu[a]) / sd[a] for a in cols] for x in train] Y = [x["suspicious"] for x in train] w = [0.0] * len(X[0]) for _ in range(epochs): g = [0.0] * len(w) for i in range(len(X)): h = squash(sum(w[j] * X[i][j] for j in range(len(w)))) - Y[i] for j in range(len(w)): g[j] += h * X[i][j] for j in range(len(w)): w[j] -= rate * (g[j] / len(X) + (penalty * w[j] if j else 0.0)) return w def score(w, cols, mu, sd, x): return w[0] + sum(w[j + 1] * (x[a] - mu[a]) / sd[a] for j, a in enumerate(cols)) def accuracy(w, cols, mu, sd, rows): return sum((squash(score(w, cols, mu, sd, x)) > 0.5) == x["suspicious"] for x in rows) / len(rows) LADDER = [("1 column", ["mean_consumption"]), ("2 columns", ["mean_consumption", "zero_readings"]), ("4 columns", ["mean_consumption", "zero_readings", "household", "satisfaction"]), ("12 columns", FIELDS)] RESULTS, tried = [], 0 print(f"{'model':<24} {'coefficients':>12} {'validation':>10} {'test':>8}") for name, cols in LADDER: mu, sd = scale(TRAIN, cols) w = train_model(TRAIN, cols, mu, sd) tried += 1 RESULTS.append((name, len(cols), accuracy(w, cols, mu, sd, VAL), accuracy(w, cols, mu, sd, TEST))) print(f"{name:<24} {len(cols):>12} {RESULTS[-1][2]:>10.4f} {RESULTS[-1][3]:>8.4f}") emu, esd = scale(EX_TRAIN, EXPANDED) for penalty in (0.0, 0.01, 0.05, 0.2): w = train_model(EX_TRAIN, EXPANDED, emu, esd, penalty=penalty) tried += 1 RESULTS.append((f"33 columns penalty {penalty}", len(EXPANDED), accuracy(w, EXPANDED, emu, esd, EX_VAL), accuracy(w, EXPANDED, emu, esd, EX_TEST))) print(f"{RESULTS[-1][0]:<24} {len(EXPANDED):>12} {RESULTS[-1][2]:>10.4f} " f"{RESULTS[-1][3]:>8.4f}") print(f"\nbaseline {BASELINE:.4f}, candidates tried {tried}") readable = max(RESULTS[:4], key=lambda t: t[2]) overall = max(RESULTS, key=lambda t: t[2]) print(f"readable candidate chosen from validation: {readable[0]}, test {readable[3]:.4f}") print(f"all candidates, chosen from validation: {overall[0]}, test {overall[3]:.4f}") print(f"price of interpretability {overall[3] - readable[3]:.4f}, that is " f"{round((overall[3] - readable[3]) * len(TEST))} subscriber") ``` ``` model coefficients validation test 1 column 1 0.8254 0.7976 2 columns 2 0.8095 0.7976 4 columns 4 0.8095 0.7897 12 columns 12 0.8333 0.8016 33 columns penalty 0.0 33 0.8333 0.8056 33 columns penalty 0.01 33 0.8373 0.7976 33 columns penalty 0.05 33 0.8492 0.8056 33 columns penalty 0.2 33 0.8214 0.7937 baseline 0.7579, candidates tried 8 readable candidate chosen from validation: 12 columns, test 0.8016 all candidates, chosen from validation: 33 columns penalty 0.05, test 0.8056 price of interpretability 0.0040, that is 1 subscriber ``` The last line is the number that carries the title of the lesson. Choosing among the readable models by looking at the validation set gives the twelve-column model, and 0.8016 is read on the test set. Choosing among all candidates by looking at the same set gives the expanded model with a penalty, and 0.8056 is read on the test set. The price of interpretability is 0.0040, that is, **one in 252 subscribers**. The smallness of this number is specific to this set and is not a general law; but what has to be read is not its size but **that it was measured**. When it is said that performance was given up for the sake of interpretability and the amount given up is not written, that sentence is not a measurement but a declaration of preference. Here the amount was written and its budget was put next to it: eight candidates. The ladder itself says something separate. The one-column model beats the four-column model on validation with 0.8254, and on the test set both stay below the twelve-column one. Adding columns is not a one-way improvement, and this was the finding of the twelve lessons of the Data Preparation and Feature Engineering course. ## What a Coefficient Says and Does Not Say The twelve coefficients of the chosen readable model are read on scaled columns, and in this form they are comparable with one another. ```python mu, sd = scale(TRAIN, FIELDS) W = train_model(TRAIN, FIELDS, mu, sd) print(f"{'column':<16} {'coefficient':>11} (scaled columns, intercept " f"{W[0]:+.3f})") for j in sorted(range(len(FIELDS)), key=lambda j: -abs(W[j + 1])): print(f"{FIELDS[j]:<16} {W[j + 1]:>+11.3f}") print() TWIN = [("full 12 columns", FIELDS), ("per_person removed", [a for a in FIELDS if a != "per_person"]), ("mean_consumption removed", [a for a in FIELDS if a != "mean_consumption"])] print(f"{'model':<24} {'mean_consumption':>16} {'per_person':>10} {'test':>8}") for name, cols in TWIN: mu2, sd2 = scale(TRAIN, cols) w = train_model(TRAIN, cols, mu2, sd2) k = {a: w[j + 1] for j, a in enumerate(cols)} print(f"{name:<24} {k.get('mean_consumption', 0.0):>+16.3f} " f"{k.get('per_person', 0.0):>+10.3f} {accuracy(w, cols, mu2, sd2, TEST):>8.4f}") print() print(f"{'ridge penalty':>13} {'mean_consumption':>16} {'volatility':>10} " f"{'zero_readings':>13} {'test':>8}") for penalty in (0.0, 0.02, 0.1, 0.5): w = train_model(TRAIN, FIELDS, mu, sd, penalty=penalty) k = {a: w[j + 1] for j, a in enumerate(FIELDS)} print(f"{penalty:>13} {k['mean_consumption']:>+16.3f} {k['volatility']:>+10.3f} " f"{k['zero_readings']:>+13.3f} {accuracy(w, FIELDS, mu, sd, TEST):>8.4f}") ``` ``` column coefficient (scaled columns, intercept -1.643) mean_consumption +0.772 zero_readings +0.684 volatility +0.380 area_east +0.306 satisfaction -0.273 household +0.156 period +0.148 area_west -0.147 area_south -0.109 per_person +0.071 area_central -0.048 area_north -0.005 model mean_consumption per_person test full 12 columns +0.772 +0.071 0.8016 per_person removed +0.808 +0.000 0.8016 mean_consumption removed +0.000 +0.442 0.7778 ridge penalty mean_consumption volatility zero_readings test 0.0 +0.772 +0.380 +0.684 0.8016 0.02 +0.573 +0.397 +0.523 0.7937 0.1 +0.326 +0.342 +0.310 0.7937 0.5 +0.133 +0.178 +0.127 0.7619 ``` The first table shows what the coefficient says, and what it says is true. Mean consumption, the zero-reading indicator and volatility are in the first three places; these are exactly the three heaviest terms that the fiction used when generating the label. The sign of satisfaction is minus, that of household size is plus, and among the regions east is the highest. The coefficient of north sits at zero with -0.005; in the fiction the region contribution of north is also exactly zero. The twelve numbers give the order of the generating structure in a readable form. The second table shows **what the same coefficients do not say**. In the twelve-column model per-person consumption stands near the end of the list with +0.071; someone who looks at this number concludes that the column is almost useless. When the column is removed from the model, the test number **does not change at all**, it stays at 0.8016; up to here the reading is correct. But when mean consumption is removed, the coefficient of the same column jumps from +0.071 to +0.442, that is, to six times. Per-person consumption was not a useless column; it **looked useless while mean consumption was next to it**, because the two are two forms of the same quantity, and the model divides the weight belonging to that quantity between the two as it pleases. The third table measures the same uncertainty through the penalty. The coefficient of mean consumption falls from 0.772 to 0.326, that is, to below half; the zero-reading indicator falls from 0.684 to 0.310. The test number, on the other hand, falls from 0.8016 to 0.7937, that is, by only two subscribers. **These three models are almost the same as predictions, and their coefficients hardly resemble one another.** The held-out number is not enough to determine which coefficient set was chosen. The rule that comes out of this is limited and has to be written exactly. The **place of a coefficient in the ranking** is readable information to the extent that the columns are independent of one another. The **value** of a coefficient, on the other hand, is meaningful only together with the set of columns beside it and the penalty applied. The **causal** reading of a coefficient does not come out of any of these tables; the Applied Statistics topic had measured that limit separately. ## Summary - The price of interpretability in this set is 0.0040, that is, one in 252 subscribers: the readable model chosen from validation gives 0.8016 on the test set, the best family chosen from the same set gives 0.8056, and the number of candidates tried is 8. - The ladder is not one-way: the one-column model passes the four-column one on validation, and on the test set both stay below the twelve-column one. - The twelve scaled coefficients give the order of the generating structure in the fiction correctly; the first three places are mean consumption, zero readings and volatility, and the coefficient of north sits at zero with -0.005. - The value of a coefficient depends on the columns beside it: per-person consumption is +0.071, and it jumps to +0.442 when mean consumption is removed, that is, it rises to six times. - The ridge penalty brings the coefficients of mean consumption and zero readings down to below half and hardly changes volatility, while the test number falls by only 0.0079; the coefficients of models that are almost the same as predictions hardly resemble one another. ## Next Step Across five lessons the model family changed but the **form** of the model never did. Least squares was a plane; polynomial regression bent the columns, but the decision was still a linear combination of the columns; the three penalties only shrank the coefficients of that combination; logistic regression squeezed the output of the plane, but the squeeze is a one-way increasing function, so the decision boundary is still a plane. In all five lessons the subscriber being classified fell on one side or the other according to a single weighted sum of twelve numbers. This is a limit, and it has never been tested. In a set where the boundary itself is **curved** — a set where suspicion rises in a certain range of two columns and falls on both sides of the range — no plane works, and polynomial columns catch that curve only if we know its shape in advance. The next topic opens with families that do not build the decision with a plane: an instance-based procedure that decides by looking at its neighbors, and what happens to the distance measure itself as the number of columns grows.