--- title: 'Data Preparation and Feature Engineering' source: 'https://academia.sh/en/courses/feature-engineering' language: en updated: '2026-10-07T04:31:19+00:00' license: 'CC BY-SA 4.0' lessons: 12 outcomes: - 'Prepares data for modeling in the right order and measures the contribution of each step on the held-out set' - 'Makes feature scaling and encoding decisions by model family and ties the justification to a number' - 'Applies the preparation steps without leakage by building a pipeline and counts the discipline of preparing inside the fold' --- # Data Preparation and Feature Engineering Measuring what each preparation decision adds on the held-out set when the model is held fixed and only the features are changed: a cleaning decision giving opposite signs in two model families, scaling producing a contribution in distance- and gradient-based procedures but not in a tree, the column count of one-hot encoding and the leakage of target encoding as cardinality grows, missingness itself carrying information, accuracy being a misleading baseline on an imbalanced class, the same outlier having opposite effects in two models, the contribution of a generated feature, the number of values that come out of one field, the inflation the selection itself produces, how interpretability changes with the number of components, the number of steps that leak without a pipeline, and the two numbers given by preparing inside the fold. ## Preparation Steps Counting how much each decision taken on the existing columns moves the number on the held-out set while the model stays fixed: the contribution of the cleaning decision and its opposite signs in two model families, in which model family standardization and normalization produce a contribution and why the scale is fitted only on the training set, the column count and leakage of one-hot, ordinal and target encoding, the three separate numbers of imputation, the missingness indicator and dropping the row, the result that oversampling, undersampling and class weight give on an imbalanced set, the opposite effect of the same outlier on a gradient-based and a tree-based method. ### Data Cleaning Source: https://academia.sh/en/courses/feature-engineering/data-cleaning Measuring on the held-out set what a cleaning decision adds to the model: the three defects in the raw record are fixed one at a time, the model is held fixed, and two numbers on the test set are printed side by side. Fixing the unit mix-up gains the distance-based method twelve subscribers and costs the decision tree two; when all three defects are fixed together, the tree's test accuracy rises from 0.7659 to 0.7778 and the distance-based method's from 0.7540 to 0.7857, and the baseline is 0.7579 for both. The previous course built the distinction over eleven lessons: the number measured on the data a model has seen and the number measured on data it has not seen are two different numbers, and each number is written next to a baseline computed on its own split. That course closed with something missing. The **features** given to the model were always taken as ready: mean consumption was a number, the region was five binary columns, volatility had already been computed. Which column was turned into a number in which way, and how much that conversion moved the difference from the baseline, was never asked. This lesson asks that question and sets up the unit of measurement for the course. The method is the same in every lesson: **the model is held fixed and only the features change.** A single preparation decision is switched on and off, the two numbers on the held-out set are printed side by side, and the difference between them is called the **contribution**. The rule of the course follows from this: **a preparation step exists to the extent of its contribution measured on the held-out set. A step whose contribution has not been measured is not a step but a habit.** - **PS1.** The dataset is inherited from the Introduction to Machine Learning course and is **fictional**: 1,260 subscribers derived from the regional measurement network, a label for suspected illicit use or a fault, seed 20260218, split 756/252/252. - **PS2.** In this course the **held-out set is the test set**, and the baseline there is 0.7579 — telling every subscriber the most frequent class of the training set. By the rule of the Introduction to Machine Learning course, every look accumulates a selection share; the accounting of that share is left to the end of the course. - **PS3.** The table used by the Introduction to Machine Learning course is the **cleaned** table. This lesson goes one step back: the raw record form of the same setup, before the cleaning decisions were made. - **PS4.** The raw record has three defects and all three are **fictional**. Some of the older meters report consumption in liters; the region name is recorded in eleven distinct spellings; an unreadable meter is read a second time, so the same subscriber appears on more than one row. - **PS5.** The defects corrupt only the **feature columns**. The label is an inspection result and is affected by none of the defects; this is why the baseline stays fixed at 0.7579. - **PS6.** The split is made **by subscriber**. Duplicate rows do not cross between splits; they only change the weight within the training set. - **PS7.** Two model families are compared: the depth 6 decision tree of the Introduction to Machine Learning course and a distance-based method that looks at the nearest fifteen neighbors. Both are written within the lesson, both are models, and their settings do not change for the whole course. - **PS8.** The measure is accuracy. The test set is 252 subscribers, so **one subscriber is worth 0.0040**, and differences smaller than that count as unmeasured. The cleaning procedure itself is not built in this lesson. Missing values, type conversion, duplicate records and category merging were measured in the Data Collection and Preparation course of the Data Analytics curriculum; the measure there was **the number of rows** — how many rows were fixed, how many were dropped. The question here is different: how much does the same decision move the number **on the held-out set**, and which model family is sensitive to it. #### The Raw Record and Three Defects The setup regenerates the dataset with the seed of the Introduction to Machine Learning course, splits it by subscriber, and builds the raw record form of the training and test splits. The label is not touched. ```python # preparation.py — MODEL: the fictional subscriber table of the Introduction to Machine Learning course is regenerated with the same # seed; the three defects of the RAW record are laid on top of it. import math SEED, CANDIDATES, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10), ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05), ("central", 0.18, 23, 0.05)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 SUBSCRIBERS = [] for i in range(CANDIDATES): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) satisfaction = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() >= 0.046: SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household, "satisfaction": satisfaction}) READINGS = {} for k in SUBSCRIBERS: r = rng(SEED + 7000 + k["id"]) for d in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): READINGS.setdefault(k["id"], []).append(0.0 if r() < 0.038 else math.floor( k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - d * 0.05) * 100 + 0.5) / 100) r() DATA = [] for k in SUBSCRIBERS: v = READINGS.get(k["id"]) if v is None: continue mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"]) z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3] + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30) DATA.append({"id": k["id"], "household": k["household"], "satisfaction": k["satisfaction"], "period": len(v), "mean_consumption": round(mean, 2), "volatility": round(vol, 2), "zero_readings": int(min(v) == 0.0), "region": k["region"][0], "suspicious": int(z > 1.35)}) def shuffle(data, seed): r, s = rng(seed), list(range(len(data))) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return [data[i] for i in s] K = shuffle(DATA, SEED + 90000) TRAIN0, TEST0 = K[:756], K[1008:] BASELINE = sum(x["suspicious"] == 0 for x in TEST0) / len(TEST0) SPELLING = {"north": ["north", "North", "NORTH"], "south": ["south", "South"], "east": ["east", "East"], "west": ["west", "WEST"], "central": ["central", "Central "]} def make_raw(rows, seed): # liter records + spelling inconsistency r, out = rng(seed), [] for x in rows: y = dict(x) if r() < 0.35: y["mean_consumption"] = round(y["mean_consumption"] * 1000, 1) y["volatility"] = round(y["volatility"] * 1000, 1) v = SPELLING[y["region"]] y["region"] = v[0] if r() < 0.70 else v[int(r() * (len(v) - 1)) + 1] out.append(y) return out def duplicate(rows, seed): # an unreadable meter is read a second time r, out = rng(seed), [] for x in rows: out.append(x) if x["zero_readings"] == 1 and r() < 0.85: out += [dict(x), dict(x)] return out RAW_TRAIN = duplicate(make_raw(TRAIN0, SEED + 61000), SEED + 62000) RAW_TEST = make_raw(TEST0, SEED + 63000) print(f"train {len(TRAIN0)} subscribers, {len(RAW_TRAIN)} rows in the raw record; " f"test {len(TEST0)} subscribers, {len(RAW_TEST)} rows") print(f"rows recorded in liters: train " f"{sum(1 for x in RAW_TRAIN if x['mean_consumption'] > 1000)}, test " f"{sum(1 for x in RAW_TEST if x['mean_consumption'] > 1000)}") print(f"{len({x['region'] for x in RAW_TRAIN})} distinct spellings for five regions, " f"baseline (most frequent class) on test {BASELINE:.4f}") ``` ``` train 756 subscribers, 864 rows in the raw record; test 252 subscribers, 252 rows rows recorded in liters: train 295, test 86 11 distinct spellings for five regions, baseline (most frequent class) on test 0.7579 ``` The size of the three defects can be measured: 756 subscribers have grown to 864 rows, 295 rows have shifted consumption by a factor of a thousand, and five regions are recorded in eleven distinct spellings. These are the same defects that were described by counting rows in the Data Analytics curriculum. What is new is that each of them is a **decision**, and that the price of the decision will be read off the model. #### The Same Decision, Two Model Families The code below writes the two models and applies the three cleaning decisions one at a time. The decision tree picks a feature and a threshold at each node, splits the set in two, and states the majority class at the leaf. The distance-based method finds the fifteen nearest neighbors of a subscriber and states their majority; closeness is the sum of the squared differences over all columns. ```python def gini(s): p = sum(x["suspicious"] for x in s) / len(s) if s else 0.0 return 2 * p * (1 - p) def tree(s, depth, fields, min_size=2): # MODEL: decision tree, depth 6 p = sum(x["suspicious"] for x in s) / len(s) best = None if depth and len(s) >= 2 * min_size and 0.0 < p < 1.0: for a in fields: d = sorted({x[a] for x in s}) for v in (d[1:] if len(d) < 10 else [d[int(i * len(d) / 10)] for i in range(1, 10)]): left = [x for x in s if x[a] < v] right = [x for x in s if x[a] >= v] if min(len(left), len(right)) < min_size: continue k = gini(s) - (len(left) * gini(left) + len(right) * gini(right)) / len(s) if best is None or k > best[0]: best = (k, a, v, left, right) if best is None or best[0] <= 1e-9: return {"prediction": int(p > 0.5)} return {"field": best[1], "threshold": best[2], "left": tree(best[3], depth - 1, fields, min_size), "right": tree(best[4], depth - 1, fields, min_size)} def tree_predict(d, x): while "prediction" not in d: d = d["left"] if x[d["field"]] < d["threshold"] else d["right"] return d["prediction"] def neighbor_predict(train, x, fields, k=15): # MODEL: distance-based method u = sorted((sum((x[a] - y[a]) ** 2 for a in fields), y["suspicious"]) for y in train)[:k] return int(sum(t[1] for t in u) * 2 > k) def expand_columns(train, test): # region spellings become binary columns cats = sorted({x["region"] for x in train}) fields = ["mean_consumption", "volatility", "household", "satisfaction", "period", "zero_readings"] expand = (lambda s: [dict(x, **{"b_" + k: int(x["region"] == k) for k in cats}) for x in s]) return fields + ["b_" + k for k in cats], expand(train), expand(test) def measure(train, test): fields, e, s = expand_columns(train, test) m = tree(e, 6, fields) return (sum(tree_predict(m, x) == x["suspicious"] for x in s) / len(s), sum(neighbor_predict(e, x, fields) == x["suspicious"] for x in s) / len(s)) def fix_units(s): # liter records are converted to cubic meters return [dict(x, mean_consumption=round(x["mean_consumption"] / 1000, 2), volatility=round(x["volatility"] / 1000, 2)) if x["mean_consumption"] > 1000 else dict(x) for x in s] def fix_spelling(s): return [dict(x, region=x["region"].strip().lower()) for x in s] def dedupe(s): # the first reading is kept, the copies are dropped g, out = set(), [] for x in s: if x["id"] not in g: g.add(x["id"]) out.append(x) return out DECISIONS = [("none (raw record)", lambda e, s: (e, s)), ("units fixed", lambda e, s: (fix_units(e), fix_units(s))), ("spelling merged", lambda e, s: (fix_spelling(e), fix_spelling(s))), ("duplicates dropped", lambda e, s: (dedupe(e), s)), ("all three", lambda e, s: (dedupe(fix_spelling(fix_units(e))), fix_spelling(fix_units(s))))] RESULT = {} print(f"{'decision applied':<22} {'tree':>8} {'distance':>9}") print(f"{'baseline':<22} {BASELINE:>8.4f} {BASELINE:>9.4f}") for name, f in DECISIONS: RESULT[name] = measure(*f(RAW_TRAIN, RAW_TEST)) print(f"{name:<22} {RESULT[name][0]:>8.4f} {RESULT[name][1]:>9.4f}") ``` ``` decision applied tree distance baseline 0.7579 0.7579 none (raw record) 0.7659 0.7540 units fixed 0.7579 0.8016 spelling merged 0.7738 0.7540 duplicates dropped 0.7778 0.7659 all three 0.7778 0.7857 ``` The second row is the starting point of the lesson. The distance-based method trained on the raw record reads 0.7540 on the test set, and this is **below** the baseline. A procedure that looks at no feature, learns nothing and gives every subscriber the same answer is better than it. The tree trained on the same raw record exceeds the baseline by 0.0079. Two models, the same table, two different signs. The third row is the real number of the lesson. Fixing the units takes the distance-based method from 0.7540 to 0.8016; it **lowers** the tree from 0.7659 to 0.7579. The same decision, the same data, the opposite sign. The reason is the way the models read a column. The tree looks only at the **order** in each column and splits with a single threshold; when the unit mix-up breaks the ordering, the tree can first split at the thousand boundary and then reuse the column in every branch, and the price it pays is one node depth. The distance-based method, on the other hand, **sums** the differences of all columns; a column that has grown a thousandfold determines the sum on its own, and the household, satisfaction, period and region columns all fall out of the calculation. #### How Many Subscribers a Contribution Is Worth Converting the differences into a number of subscribers separates the rows that are worth reading. ```python RAW_RESULT = RESULT["none (raw record)"] print(f"{'decision':<22} {'tree contribution':>19} {'subscribers':>12} " f"{'distance contribution':>23} {'subscribers':>12}") for name, _ in DECISIONS[1:]: ka, ku = RESULT[name][0] - RAW_RESULT[0], RESULT[name][1] - RAW_RESULT[1] print(f"{name:<22} {ka:>+19.4f} {round(ka * 252):>12} " f"{ku:>+23.4f} {round(ku * 252):>12}") print(f"\ndifference over the baseline, raw record: tree {RAW_RESULT[0] - BASELINE:+.4f}, " f"distance {RAW_RESULT[1] - BASELINE:+.4f}") u = RESULT["all three"] print(f"difference over the baseline, all three: tree {u[0] - BASELINE:+.4f}, " f"distance {u[1] - BASELINE:+.4f}") ``` ``` decision tree contribution subscribers distance contribution subscribers units fixed -0.0079 -2 +0.0476 12 spelling merged +0.0079 2 +0.0000 0 duplicates dropped +0.0119 3 +0.0119 3 all three +0.0119 3 +0.0317 8 difference over the baseline, raw record: tree +0.0079, distance -0.0040 difference over the baseline, all three: tree +0.0198, distance +0.0278 ``` The table has a single large number worth reading: the twelve subscribers that the unit fix adds to the distance-based method. The remaining cells are two or three subscribers, and on a set of 252 subscribers, two subscribers cannot be told apart from the randomness of the set itself. The −0.0079 harm the tree sees from the unit fix is of that size too; its sign is negative, but the sentence to read is "the contribution of this step to the tree has not been measured", not "it harms the tree". This distinction continues through the course: the sign of a contribution acquires meaning when its size exceeds the measurement resolution. The last two rows show where the differences come from. When all three defects are fixed together, the tree exceeds the baseline by 0.0198 and the distance-based method by 0.0278. These two differences come **from learning**: the defects were generated independently of the label, none of them is more frequent among suspicious subscribers, so no cleaning decision carries hidden information about the label into the model. Later in the course the opposite will be measured too — there are steps whose contribution comes not from learning but from leakage, and the two can be told apart only when the source is asked. There is one more check. When all three defects are fixed together, the tree's test accuracy comes out at 0.7778, and this is **exactly the same** as the number that the Introduction to Machine Learning course read with the same model, the same split and the same seed. This is no coincidence: the cleaned table is the table that course used from the very start. What was not visible throughout the previous course was that this table was itself the product of a series of decisions. #### Summary - The distance-based method trained on the raw record reads 0.7540 on the test set and stays 0.0040 below the baseline; the decision tree trained on the same table exceeds the baseline by 0.0079 with 0.7659. - Fixing the unit mix-up adds +0.0476 to the distance-based method, that is, twelve subscribers; the same decision takes −0.0079 from the decision tree, that is, two subscribers. - The reason the sign reverses is the way the models read a column: the tree looks only at the order in each column and makes up for the mixed units with one node of depth, while the distance-based method sums all the columns and a thousandfold column determines the sum on its own. - Spelling merging and duplicate record dropping move two or three subscribers in both models; since one subscriber is worth 0.0040 on a 252-subscriber test set, these differences count as unmeasured. - When all three defects are fixed together the tree reads 0.7778; this is the same number that the Introduction to Machine Learning course read with the same model and the same split, because that course was already using the cleaned table. #### Next Step This lesson showed that the unit mix-up breaks the distance-based method, but even after the fix that method stayed behind the tree and could exceed the baseline by only 0.0278. The reason is still in the table: the columns were converted to the same unit but not to the **same magnitude**. Mean consumption varies between zero and forty-three, household size between one and six, and the region columns between zero and one; for a method that sums the squares of the differences, this silently gives the columns weights. The next lesson measures two scaling decisions — standardization and normalization — separately in three model families, builds the discipline of learning the scale **only from the training set** and applying it to the held-out set, and counts in which model family the contribution comes out as zero. ### Feature Scaling Source: https://academia.sh/en/courses/feature-engineering/feature-scaling Measuring the standardization and normalization decisions separately in three model families: standardization adds +0.0437 to the gradient-based linear model, gives −0.0159 on the distance-based method, and its contribution on the decision tree is exactly zero. The scale is an object fitted on the training set; applied to the test set, two values fall outside the [0, 1] range and standard scores rise as high as 5.11. When all eleven columns are scaled, the distance-based method loses one more subscriber. The previous lesson fixed the three defects in the raw record and took the distance-based method from 0.7540 to 0.7857. That number exceeds the baseline by only 0.0278 — seven subscribers. The cause is not something left uncleaned but something never asked: the columns were converted to the same unit but not to the **same magnitude**. Mean consumption varies between zero and forty-three, while household size varies between one and six and the region columns between zero and one. For a method that sums the squares of the differences, this silently gives the columns weights. This lesson measures two scaling decisions. **Standardization** subtracts the column mean from each value and divides by its dispersion; the result is the standard score, and it was defined in the Data Collection and Preparation course of the Data Analytics curriculum. **Normalization** squeezes each column into a range so that its smallest value is zero and its largest is one; it shares a name with the text normalization of the Data Collection and Preparation course but does a different job: one unifies spelling, the other fits a number into a range. Neither is new as a procedure; what is measured is **in which model family** each one is worth how many subscribers. - **PS9.** The starting table of this lesson is the output of the previous lesson: the fictional table of 1,260 subscribers with the three defects fixed, the same 756/252/252 split, a baseline of 0.7579 on the test set. - **PS10.** Three model families are compared, and all three are written within the lesson: a distance-based method that looks at the nearest fifteen neighbors, a linear model that updates its weights in the direction of the gradient, and a depth 6 decision tree. Their settings are fixed for the whole course. - **PS11.** The output of the gradient-based model is squeezed between zero and one by an S-shaped function; a subscriber above the threshold of 0.5 counts as suspicious. The threshold is a choice and does not change in this course. - **PS12.** Scaling is applied only to the five numeric columns: mean consumption, volatility, household size, satisfaction, number of periods. Scaling the binary columns is measured at the end of the lesson as a separate decision. - **PS13.** The scale is **a fitted object** and has two faces: **fit**, extracting the scale numbers from the training set; **transform**, applying those numbers to a set. The second rule of the course turns into a procedure here for the first time: **every preparation step is fitted only on the training set and applied to the held-out set.** - **PS14.** A scale fitted on the whole set is a leak. The Introduction to Machine Learning course counted this as one of the four kinds of leakage; the job here is not to repeat those counts but to build the form that does not leak as a procedure. - **PS15.** The measure is accuracy and the test set is 252 subscribers; one subscriber is worth 0.0040. - **PS16.** Floating-point comparison is not done with `==`. The out-of-range count uses `0.0 <= v <= 1.0`; the boundary value itself counts as inside the range. #### Three Models, Two Scaling Decisions The setup produces the table and prints the raw ranges of the five numeric columns. ```python # scaling.py — MODEL. The fictional subscriber table of the Introduction to Machine Learning course is regenerated in the # form cleaned in the previous lesson. import math import statistics SEED, CANDIDATES, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10), ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05), ("central", 0.18, 23, 0.05)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 SUBSCRIBERS, READINGS = [], {} for i in range(CANDIDATES): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) satisfaction = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() >= 0.046: SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household, "satisfaction": satisfaction}) for k in SUBSCRIBERS: r = rng(SEED + 7000 + k["id"]) for d in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): READINGS.setdefault(k["id"], []).append(0.0 if r() < 0.038 else math.floor( k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - d * 0.05) * 100 + 0.5) / 100) r() DATA = [] for k in SUBSCRIBERS: v = READINGS.get(k["id"]) if v is None: continue mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"]) z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3] + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30) x = {"household": k["household"], "satisfaction": k["satisfaction"], "period": len(v), "mean_consumption": round(mean, 2), "volatility": round(vol, 2), "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35)} for b in REGION: x["b_" + b[0]] = int(k["region"][0] == b[0]) DATA.append(x) def shuffle(data, seed): r, s = rng(seed), list(range(len(data))) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return [data[i] for i in s] K = shuffle(DATA, SEED + 90000) TRAIN, TEST = K[:756], K[1008:] BASELINE = sum(x["suspicious"] == 0 for x in TEST) / len(TEST) NUMERIC = ["mean_consumption", "volatility", "household", "satisfaction", "period"] FIELDS = NUMERIC + ["zero_readings", "b_north", "b_south", "b_east", "b_west", "b_central"] print(f"train {len(TRAIN)}, test {len(TEST)}, baseline {BASELINE:.4f}") print(f"{'column':<16} {'minimum':>9} {'maximum':>9} {'mean':>9}") for a in NUMERIC: v = [x[a] for x in TRAIN] print(f"{a:<16} {min(v):>9.2f} {max(v):>9.2f} {statistics.fmean(v):>9.2f}") ``` ``` train 756, test 252, baseline 0.7579 column minimum maximum mean mean_consumption 0.00 42.90 19.77 volatility 0.00 40.57 9.62 household 1.00 6.00 3.22 satisfaction 1.00 5.00 3.37 period 1.00 3.00 2.53 ``` The ratio between the ranges reaches ninety times. Now the three models and the two scaling decisions are written. Splitting the scaling functions into **fit** and **transform** is not a matter of style; it is the rule of the course carried into code. ```python def gini(s): p = sum(x["suspicious"] for x in s) / len(s) if s else 0.0 return 2 * p * (1 - p) def tree(s, depth, fields, min_size=2): # MODEL: decision tree, depth 6 p = sum(x["suspicious"] for x in s) / len(s) best = None if depth and len(s) >= 2 * min_size and 0.0 < p < 1.0: for a in fields: d = sorted({x[a] for x in s}) for v in (d[1:] if len(d) < 10 else [d[int(i * len(d) / 10)] for i in range(1, 10)]): left = [x for x in s if x[a] < v] right = [x for x in s if x[a] >= v] if min(len(left), len(right)) < min_size: continue k = gini(s) - (len(left) * gini(left) + len(right) * gini(right)) / len(s) if best is None or k > best[0]: best = (k, a, v, left, right) if best is None or best[0] <= 1e-9: return {"prediction": int(p > 0.5)} return {"field": best[1], "threshold": best[2], "left": tree(best[3], depth - 1, fields, min_size), "right": tree(best[4], depth - 1, fields, min_size)} def tree_predict(d, x): while "prediction" not in d: d = d["left"] if x[d["field"]] < d["threshold"] else d["right"] return d["prediction"] def neighbor_predict(train, x, fields, k=15): # MODEL: distance-based method u = sorted((sum((x[a] - y[a]) ** 2 for a in fields), y["suspicious"]) for y in train)[:k] return int(sum(t[1] for t in u) * 2 > k) def squash(z): # S-shaped function with overflow guard if z >= 0: return 1.0 / (1.0 + math.exp(-z)) if z < 700 else 1.0 e = math.exp(z) if z > -700 else 0.0 return e / (1.0 + e) def linear_model(train, fields, step=0.5, rounds=400): # MODEL: gradient-based linear model w, b, n = [0.0] * len(fields), 0.0, len(train) for _ in range(rounds): gw, gb = [0.0] * len(fields), 0.0 for x in train: h = squash(sum(w[i] * x[a] for i, a in enumerate(fields)) + b) - x["suspicious"] for i, a in enumerate(fields): gw[i] += h * x[a] gb += h for i in range(len(fields)): w[i] -= step * gw[i] / n b -= step * gb / n return w, b def linear_predict(m, x, fields): return int(squash(sum(m[0][i] * x[a] for i, a in enumerate(fields)) + m[1]) > 0.5) def three_models(train, test, fields=None): fields = fields or FIELDS m1, m3 = tree(train, 6, fields), linear_model(train, fields) return (sum(neighbor_predict(train, x, fields) == x["suspicious"] for x in test) / len(test), sum(linear_predict(m3, x, fields) == x["suspicious"] for x in test) / len(test), sum(tree_predict(m1, x) == x["suspicious"] for x in test) / len(test)) def fit_standard(train, columns): # FIT: mean and dispersion return {a: (statistics.fmean(x[a] for x in train), statistics.pstdev([x[a] for x in train])) for a in columns} def apply_standard(p, s): # APPLY return [dict(x, **{a: (x[a] - p[a][0]) / p[a][1] for a in p}) for x in s] def fit_normal(train, columns): # FIT: minimum and maximum return {a: (min(x[a] for x in train), max(x[a] for x in train)) for a in columns} def apply_normal(p, s): # APPLY return [dict(x, **{a: (x[a] - p[a][0]) / (p[a][1] - p[a][0]) for a in p}) for x in s] PS, PN = fit_standard(TRAIN, NUMERIC), fit_normal(TRAIN, NUMERIC) CASES = [("unscaled (raw column)", TRAIN, TEST), ("standardization", apply_standard(PS, TRAIN), apply_standard(PS, TEST)), ("normalization", apply_normal(PN, TRAIN), apply_normal(PN, TEST))] S = {} print(f"{'decision':<30} {'distance':>8} {'linear':>13} {'tree':>8}") print(f"{'baseline':<30} {BASELINE:>8.4f} {BASELINE:>13.4f} {BASELINE:>8.4f}") for name, e, s in CASES: S[name] = three_models(e, s) print(f"{name:<30} {S[name][0]:>8.4f} {S[name][1]:>13.4f} {S[name][2]:>8.4f}") raw = S["unscaled (raw column)"] for name in ("standardization", "normalization"): k = [S[name][i] - raw[i] for i in range(3)] print(f"{name + ' contribution':<30} {k[0]:>+8.4f} {k[1]:>+13.4f} {k[2]:>+8.4f}") ``` ``` decision distance linear tree baseline 0.7579 0.7579 0.7579 unscaled (raw column) 0.7857 0.7579 0.7778 standardization 0.7698 0.8016 0.7778 normalization 0.7619 0.7817 0.7778 standardization contribution -0.0159 +0.0437 +0.0000 normalization contribution -0.0238 +0.0238 +0.0000 ``` The tree column gives the same number in all three cases, and this is not a small difference that has fallen below the measurement noise but **an exact zero**. The reason can be shown: in each column the tree looks only at the order of the values, picks its candidates from among the sorted distinct values and splits with a single threshold. Standardization and normalization are increasing transformations; they do not change the order, so the splits the tree can choose stay exactly the same. Putting a scaling step in front of a tree-based method does not change the result; its contribution is zero, and that is the definition of a step that is a habit. The gradient-based linear model is at the opposite end. On the unscaled table it reads 0.7579 — exactly the baseline. This is not a good number but a symptom of collapse: the model gives every subscriber the same answer. While mean consumption goes up to forty-three, household size does not exceed six, so the same step size is too large for one column and too small for the other, and in four hundred rounds the weights cannot reach a usable place. Standardization adds 0.0437 to this model, that is, eleven subscribers. The row of the distance-based method, on the other hand, breaks a common expectation. Standardization makes it **lose 0.0159**, normalization 0.0238. Scaling is not a step that helps distance-based methods; it is a step that **gives the columns equal weight**. In this table the raw scale had already given the widest ranges to the two most informative columns, mean consumption and volatility. Equalization takes that weight and distributes it to household size and the number of periods. The sign of the decision depends on the information the scale itself happens to carry, and this can be known only by measuring. #### The Scale Is a Fitted Object What the scaling step fits is a handful of numbers, and those numbers come only from the training set. ```python print(f"{'column':<16} {'mean':>9} {'dispersion':>12} " f"{'minimum':>9} {'maximum':>9}") for a in NUMERIC: print(f"{a:<16} {PS[a][0]:>9.3f} {PS[a][1]:>12.3f} " f"{PN[a][0]:>9.2f} {PN[a][1]:>9.2f}") sn = apply_normal(PN, TEST) print(f"\nvalues outside the [0, 1] range in test " f"{sum(1 for x in sn for a in NUMERIC if not 0.0 <= x[a] <= 1.0)}, " f"subscribers {sum(1 for x in sn if any(not 0.0 <= x[a] <= 1.0 for a in NUMERIC))}") ss = apply_standard(PS, TEST) print(f"{'column':<16} {'lowest in test':>18} {'highest':>10} (standard score)") for a in NUMERIC: v = [x[a] for x in ss] print(f"{a:<16} {min(v):>18.2f} {max(v):>10.2f}") PT = fit_standard(TRAIN, FIELDS) t = three_models(apply_standard(PT, TRAIN), apply_standard(PT, TEST)) print(f"\nall eleven columns standardized: distance {t[0]:.4f}, " f"linear {t[1]:.4f}, tree {t[2]:.4f}") ``` ``` column mean dispersion minimum maximum mean_consumption 19.767 6.200 0.00 42.90 volatility 9.622 8.120 0.00 40.57 household 3.224 1.270 1.00 6.00 satisfaction 3.372 1.147 1.00 5.00 period 2.532 0.715 1.00 3.00 values outside the [0, 1] range in test 2, subscribers 2 column lowest in test highest (standard score) mean_consumption -2.63 3.83 volatility -1.19 5.11 household -1.75 2.19 satisfaction -2.07 1.42 period -2.14 0.66 all eleven columns standardized: distance 0.7659, linear 0.8016, tree 0.7778 ``` The fitted object is ten numbers, and all of them come from the training set of 756 subscribers. At the point of application there is a visible consequence: normalization is defined as "fit every value into the range between zero and one", yet in the test set two values fall outside that range. This is not a bug and it is not fixed. The only way to fix it would be to choose the minimum and the maximum by looking at the test set as well; that would be fitting the scale on the held-out set, that is, leaking. Before the split, the Introduction to Machine Learning course had counted scaling as one of the four kinds of leakage; the job here is not to repeat those counts but to build the form that does not leak as an object. The standard score table shows the same thing from the other direction: in the volatility column the highest value in the test set is 5.11 standard scores, and the scale fitted on the training set lets it through without clipping. The last line measures a third decision: **which columns are scaled.** When all eleven columns, binary ones included, are standardized instead of the five numeric columns, the gradient-based model stays at 0.8016, the tree stays at 0.7778, and the distance-based method drops from 0.7698 to 0.7659. A difference of one subscriber is not read on its own, but its direction can be explained: when a binary column is standardized, the rare category is carried to a large standard score, and since the region difference is in the distance sum, it weighs heavily. The scaling decision is not a single switch; which column it is applied to is also a decision, and that too is measured on the held-out set. #### Summary - Standardization adds 0.0437 to the gradient-based linear model, that is, eleven subscribers; the same decision gives −0.0159 on the distance-based method and exactly 0.0000 on the decision tree. - The zero in the tree is not measurement noise but a provable result: standardization and normalization are increasing transformations, they do not change the order within a column, and the tree looks only at the order. - On the unscaled table the gradient-based model reads exactly the baseline, 0.7579; this is not a good number but a symptom that a single step size is not enough for ranges ninety times apart. - The scale is ten numbers fitted on the training set; applied to the test set, two values fall outside the [0, 1] range, and this is not fixed, because the only way to fix it would be to fit the scale on the held-out set. - The choice of which columns to scale is a separate decision: when the binary columns are scaled too, the distance-based method drops from 0.7698 to 0.7659 and the other two models do not change. #### Next Step In this lesson the region information was used as five ready-made binary columns, and where those five columns came from was not asked. Yet the region is a text field, and turning it into a number is a decision: one column can be opened for each category, the categories can be given sequence numbers, or each category can be replaced with the suspicious rate in its own training set. The three have different costs; the first grows the number of columns by the number of categories, the third settles for a single column but puts the label inside the encoding. The next lesson measures these three encodings on a reading-route field of two hundred forty codes, counts the categories never seen in the held-out set, and prints how much of the inflation that target encoding produces on the training set recedes when it is computed inside the fold. ### Categorical Encoding Source: https://academia.sh/en/courses/feature-engineering/categorical-encoding Measuring three encodings as column count and contribution when a categorical field is turned into numbers: one-hot encoding the region raises test accuracy from 0.7302 to 0.7778, and the 240-code reading route field gives +0.0119 for 231 extra columns when one-hot encoded and the same +0.0119 with a single column when ordinal encoded. Target encoding inflates training accuracy to 0.9114 while dropping the test to 0.7421; computed in-fold, 0.0595 of the inflation on the training set recedes and the test returns to 0.7540. The previous lesson used the region information as five ready-made binary columns and did not ask where those columns came from. The region is a text field. The input of a model is a number. The translation in between is a decision, and the decision has at least two costs: how many columns it produces and how much it moves the number on the held-out set. This lesson compares three encodings on the same table. **One-hot encoding** opens a separate binary column for each category. **Ordinal encoding** gives the categories one integer each and settles for a single column. **Target encoding** replaces each category with the suspicious rate of that category in the training set; it produces a single column but puts the label inside the encoding. The measurement is made on two categorical fields: the region with five categories and the reading route code, whose **cardinality** is two hundred forty. - **PS17.** The reading route code is a field of the raw record and is **fictional**: each subscriber is attached to one reading route, and routes were opened according to region and the expansion wave of the network. The Data Analytics curriculum did not use this field. - **PS18.** The route code is not produced directly from the label. The information it carries comes from the region and from the wave of joining the network; the second is absent from the feature columns. - **PS19.** The measurement set, the split and the baseline are the same as in the earlier lessons: 756/252/252 and 0.7579 on the test set. - **PS20.** The model is fixed: a depth 6 decision tree. No scaling is applied, because the previous lesson showed that its contribution on the tree is exactly zero. - **PS21.** The encoding is **a fitted object**. What is fitted is the category list in one-hot encoding, the category–integer mapping in ordinal encoding, and the category–rate table in target encoding. All three are extracted only from the training set. - **PS22.** An **unseen category** is a code that appears in the held-out set and does not exist in the training set. Every encoding has to give it an answer, and the answer it gives is part of the encoding. - **PS23.** Computing target encoding **in-fold** means this: the training set is divided into five folds, and the encoding value of each row is computed from the four folds that do not contain that row. This way no row's own label enters its own encoding value. - **PS24.** Cross-validation itself is not built in this lesson; the fold is used only as the split in which the encoding is computed. - **PS25.** The test set is 252 subscribers, and one subscriber is worth 0.0040. #### Three Encodings, Two Numbers The setup regenerates the table and adds the route field. ```python # encoding.py — MODEL. The fictional subscriber table is regenerated with the same seed; the # FICTIONAL reading route code from the raw record is added as a categorical column. import math SEED, CANDIDATES, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10), ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05), ("central", 0.18, 23, 0.05)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 SUBSCRIBERS, READINGS = [], {} for i in range(CANDIDATES): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) satisfaction = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() >= 0.046: SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household, "satisfaction": satisfaction}) for k in SUBSCRIBERS: r = rng(SEED + 7000 + k["id"]) for d in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): READINGS.setdefault(k["id"], []).append(0.0 if r() < 0.038 else math.floor( k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - d * 0.05) * 100 + 0.5) / 100) r() DATA, route_rng = [], rng(SEED + 41000) for k in SUBSCRIBERS: v = READINGS.get(k["id"]) if v is None: continue mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"]) z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3] + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30) x = {"household": k["household"], "satisfaction": k["satisfaction"], "period": len(v), "mean_consumption": round(mean, 2), "volatility": round(vol, 2), "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35), "region": k["region"][0]} wave = min(7, int((k["id"] - 10001) / 1399 * 8)) # wave of joining the network x["route"] = "R%03d" % ([b[0] for b in REGION].index(k["region"][0]) * 48 + wave * 6 + int(route_rng() * 6) + 1) DATA.append(x) def shuffle(data, seed): r, s = rng(seed), list(range(len(data))) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return [data[i] for i in s] K = shuffle(DATA, SEED + 90000) TRAIN, TEST = K[:756], K[1008:] BASELINE = sum(x["suspicious"] == 0 for x in TEST) / len(TEST) NUMERIC = ["mean_consumption", "volatility", "household", "satisfaction", "period", "zero_readings"] TRAIN_ROUTES = {x["route"] for x in TRAIN} ROUTE_SIZE = {} for x in TRAIN: ROUTE_SIZE[x["route"]] = ROUTE_SIZE.get(x["route"], 0) + 1 print(f"route code: {len({x['route'] for x in DATA})} in the table, {len(TRAIN_ROUTES)} in train, " f"most crowded route {max(ROUTE_SIZE.values())} subscribers, average " f"{sum(ROUTE_SIZE.values()) / len(ROUTE_SIZE):.1f}") print(f"unseen route codes in test {len({x['route'] for x in TEST} - TRAIN_ROUTES)}, " f"subscribers on these codes {sum(1 for x in TEST if x['route'] not in TRAIN_ROUTES)}") print(f"region {len({x['region'] for x in TRAIN})} categories, baseline {BASELINE:.4f}") ``` ``` route code: 240 in the table, 231 in train, most crowded route 9 subscribers, average 3.3 unseen route codes in test 6, subscribers on these codes 10 region 5 categories, baseline 0.7579 ``` On average 3.3 subscribers fall on each route, and the most crowded route has nine subscribers. This number will be decisive in the second half of the lesson. Now the four encodings are applied in turn. ```python def gini(s): p = sum(x["suspicious"] for x in s) / len(s) if s else 0.0 return 2 * p * (1 - p) def tree(s, depth, fields, min_size=2): # MODEL: decision tree, depth 6 p = sum(x["suspicious"] for x in s) / len(s) best = None if depth and len(s) >= 2 * min_size and 0.0 < p < 1.0: for a in fields: d = sorted({x[a] for x in s}) for v in (d[1:] if len(d) < 10 else [d[int(i * len(d) / 10)] for i in range(1, 10)]): left = [x for x in s if x[a] < v] right = [x for x in s if x[a] >= v] if min(len(left), len(right)) < min_size: continue k = gini(s) - (len(left) * gini(left) + len(right) * gini(right)) / len(s) if best is None or k > best[0]: best = (k, a, v, left, right) if best is None or best[0] <= 1e-9: return {"prediction": int(p > 0.5)} return {"field": best[1], "threshold": best[2], "left": tree(best[3], depth - 1, fields, min_size), "right": tree(best[4], depth - 1, fields, min_size)} def predict(d, x): while "prediction" not in d: d = d["left"] if x[d["field"]] < d["threshold"] else d["right"] return d["prediction"] def one_hot(column, train, test): # FIT: the category list in the training set cats = sorted({x[column] for x in train}) encode = (lambda s: [dict(x, **{column + "_" + k: int(x[column] == k) for k in cats}) for x in s]) return [column + "_" + k for k in cats], encode(train), encode(test) def ordinal(column, train, test, order=None): # FIT: category -> integer mapping cats = order or {k: i for i, k in enumerate(sorted({x[column] for x in train}))} encode = lambda s: [dict(x, route_ordinal=cats.get(x[column], -1)) for x in s] return ["route_ordinal"], encode(train), encode(test) def rate(rows): m = {} for x in rows: m.setdefault(x["route"], []).append(x["suspicious"]) return {k: sum(v) / len(v) for k, v in m.items()} def target(train, test): # FIT: suspicious rate per category g, t = sum(x["suspicious"] for x in train) / len(train), rate(train) encode = lambda s: [dict(x, route_target=t.get(x["route"], g)) for x in s] return ["route_target"], encode(train), encode(test) def target_in_fold(train, test, folds=5): # each row from outside its own fold g, t = sum(x["suspicious"] for x in train) / len(train), rate(train) tables = [rate([y for j, y in enumerate(train) if j % folds != i]) for i in range(folds)] e = [dict(x, route_target=tables[i % folds].get(x["route"], g)) for i, x in enumerate(train)] return ["route_target"], e, [dict(x, route_target=t.get(x["route"], g)) for x in test] RESULT = {} def measure(name, fields, e, s): m = tree(e, 6, fields) train_acc = sum(predict(m, x) == x["suspicious"] for x in e) / len(e) test_acc = sum(predict(m, x) == x["suspicious"] for x in s) / len(s) RESULT[name] = (len(fields), train_acc, test_acc) print(f"{name:<28} {len(fields):>7} {train_acc:>8.4f} {test_acc:>8.4f}") print(f"{'encoding':<28} {'columns':>7} {'train':>8} {'test':>8}") measure("no categorical column", NUMERIC, TRAIN, TEST) region_cols, eb, sb = one_hot("region", TRAIN, TEST) measure("region one-hot", NUMERIC + region_cols, eb, sb) for name, fn in (("+ route one-hot", lambda: one_hot("route", eb, sb)), ("+ route ordinal", lambda: ordinal("route", eb, sb)), ("+ route target", lambda: target(eb, sb)), ("+ route target, in-fold", lambda: target_in_fold(eb, sb))): a, e, s = fn() measure(name, NUMERIC + region_cols + a, e, s) ``` ``` encoding columns train test no categorical column 6 0.8492 0.7302 region one-hot 11 0.8466 0.7778 + route one-hot 242 0.8452 0.7897 + route ordinal 12 0.8651 0.7897 + route target 12 0.9114 0.7421 + route target, in-fold 12 0.8519 0.7540 ``` The first two rows say what the encoding itself is worth. When no categorical field is used, the model reads 0.7302 and stays 0.0277 below the baseline. When the five-category region field is opened into five binary columns, the number rises to 0.7778: twelve subscribers. This is the largest single contribution of the course so far, and the decision itself is invisible, because turning the region into a number is usually done without discussion. #### The Same Contribution, a Difference of Two Hundred Thirty-One Columns When the three encodings of the route field stand side by side in the same table, the price paid becomes visible. ```python t = RESULT["region one-hot"] print(f"{'encoding':<28} {'added columns':>14} {'train contribution':>19} {'test contribution':>18}") for name in ("+ route one-hot", "+ route ordinal", "+ route target", "+ route target, in-fold"): v = RESULT[name] print(f"{name:<28} {v[0] - t[0]:>14} {v[1] - t[1]:>+19.4f} {v[2] - t[2]:>+18.4f}") codes = sorted(TRAIN_ROUTES) code_rng = rng(SEED + 43000) perm = list(range(len(codes))) for i in range(len(perm) - 1, 0, -1): j = int(code_rng() * (i + 1)) perm[i], perm[j] = perm[j], perm[i] a, e, s = ordinal("route", eb, sb, {k: perm[i] for i, k in enumerate(codes)}) m = tree(e, 6, NUMERIC + region_cols + a) print(f"\nordinal encoding, code order shuffled: test " f"{sum(predict(m, x) == x['suspicious'] for x in s) / len(s):.4f}") g = sum(x["suspicious"] for x in TRAIN) / len(TRAIN) _, _, st = target(eb, sb) unseen = [x for x in st if x["route"] not in TRAIN_ROUTES] print(f"{len(unseen)} subscribers on unseen routes: in one-hot encoding all 231 columns are 0, " f"in ordinal encoding -1, in target encoding {g:.4f}") print(f"inflation on train of the in-fold encoding " f"{RESULT['+ route target'][1] - RESULT['+ route target, in-fold'][1]:+.4f}, " f"recovered on test " f"{RESULT['+ route target, in-fold'][2] - RESULT['+ route target'][2]:+.4f}") ``` ``` encoding added columns train contribution test contribution + route one-hot 231 -0.0013 +0.0119 + route ordinal 1 +0.0185 +0.0119 + route target 1 +0.0648 -0.0357 + route target, in-fold 1 +0.0053 -0.0238 ordinal encoding, code order shuffled: test 0.7778 10 subscribers on unseen routes: in one-hot encoding all 231 columns are 0, in ordinal encoding -1, in target encoding 0.2222 inflation on train of the in-fold encoding +0.0595, recovered on test +0.0119 ``` The first two rows give the same contribution at two different prices: one-hot encoding gives 0.0119 for 231 extra columns, ordinal encoding gives the same 0.0119 for a single column. A three-subscriber difference would not be read on its own; what is read is **the price difference.** The cost of one-hot encoding grows linearly with the cardinality, and on a training set of 756 rows, 231 columns means an average of three rows per column. There is a reason ordinal encoding works in this table, and the shuffling test reveals it. The code numbers were not given at random; they were ordered first by region and then by the wave of joining the network, so saying "the route number is smaller" says something. When the mapping is shuffled, the test number drops to 0.7778, that is, to where it would be if the route field had not been used at all. The contribution of ordinal encoding comes not from the categories but from **the order of the codes themselves**; when the order is made up, the contribution is zero. The third row is the real number of the lesson. Target encoding gains 0.0648 on the training set and loses 0.0357 on the test set. The opposite signs of the two numbers are no coincidence; they are the two faces of the same event. On average 3.3 subscribers fall on each route; the suspicious rate of a route with three subscribers is the mean of the labels of those three rows, and each row's own label is one third of that mean. When the tree reads this column, it reads a copy of the row's label. The 0.9114 on the training set is the result not of learning but of reading itself. In-fold computation corrects this. When each row's encoding value is derived from the four folds that do not contain that row, 0.0595 of the inflation on the training set disappears and the test number comes back by 0.0119. Even so, the remaining contribution is −0.0238: after the leak is pulled back, target encoding still stays below the baseline in this field. The measured result is this — removing the leak does not make the step good, it only makes the real contribution of the step visible. The last line is the silent face of the encoding. In the test set, ten subscribers are attached to six routes never seen in the training set. One-hot encoding gives them a row of 231 zeros, ordinal encoding gives a −1 outside the training range, and target encoding gives the overall rate of the training set, 0.2222. All three are a decision and all three are part of the encoding; none of them is right by itself. #### Summary - When no categorical field is used, the model reads 0.7302 and stays below the baseline; when the five-category region field is one-hot encoded it rises to 0.7778, a contribution of +0.0476. - The route field with a cardinality of 240 gives +0.0119 for 231 extra columns when one-hot encoded, and the same +0.0119 for a single column when ordinal encoded; the same contribution has two different prices. - The contribution of ordinal encoding comes from the order of the codes themselves: when the category–integer mapping is shuffled, the test number drops to 0.7778, that is, to the level where the field is not used at all. - Target encoding gives +0.0648 on train and −0.0357 on test; since 3.3 subscribers fall on each route, each row's own label is inside its own encoding value. - In-fold computation erases 0.0595 of the inflation on the training set and brings 0.0119 back on the test set, but the remaining contribution is −0.0238; removing the leak does not make the step good, it shows its real contribution. #### Next Step In this lesson the satisfaction column was assumed to be filled for every subscriber. In the raw record it is not so: satisfaction is a survey response, and there are subscribers who did not answer the survey. In all the lessons so far, that gap was silently filled with the median of the training set, and the filling itself was never measured. The next lesson puts the gap back and prints three options side by side: filling with the median, filling and adding a missingness indicator next to it, and dropping the missing rows from the training set. When the missingness carries information about the subscriber itself, the contribution of the indicator is measured, and why dropping rows cannot be applied on the held-out set is shown with numbers. ### Missing Value Strategies Source: https://academia.sh/en/courses/feature-engineering/missing-value-strategies Measuring three strategies for a column with missing values separately on the held-out set: median imputation gives 0.7619, adding a missingness indicator next to the imputation gives 0.8333, and dropping the missing rows from the training set gives 0.7659; the contribution of the indicator is +0.0714, that is, eighteen subscribers. The indicator also exceeds by 0.0555 the upper bound of 0.7778 given by the column with nothing missing, because the missingness itself carries information that the imputed value does not: the suspicious rate is 0.5276 among subscribers who did not answer the survey and 0.1383 among those who did. In the previous three lessons the satisfaction column was assumed to be filled for every subscriber. In the raw record it is not so. Satisfaction is a survey response, and there are subscribers who did not answer the survey. What to do about the gap is a decision, and until now that decision was never visible, because it was taken silently. The imputation procedure itself is not built in this lesson. Finding the missing value, imputing it with the median and dropping the row were measured in the Data Collection and Preparation course of the Data Analytics curriculum; the measure there was **how many rows were affected**. The question here is different: how much do the same three options move the number on the held-out set, and is **the missingness itself** a feature. - **PS26.** Not answering the survey is **fictional** and is not independent of the label: subscribers suspected of illicit use or a fault answer the survey less often. It was put into the setup deliberately, and it is what this lesson measures. - **PS27.** Missingness is only in the satisfaction column. The other ten columns and the label are complete. - **PS28.** The imputation value is **a fitted object**: the median is computed only from the filled rows in the training set, and that value is applied to both sets. - **PS29.** The imputation statistic is itself a decision: the median was chosen because satisfaction is measured on an ordinal scale, and on that scale the mean is not a legitimate summary. If the choice changes, the imputed value changes too. - **PS30.** The **missingness indicator** is a binary column that says whether that column is empty for the row. It is added next to the imputation, not in its place. - **PS31.** The measurement set, the split, the model and the baseline are the same as in the earlier lessons: 756/252/252, a depth 6 decision tree, 0.7579 on the test set. - **PS32.** The number measured with the complete column is printed as an **upper bound**: what would be read if there had been no missingness at all. This row is not a strategy but a comparison line. - **PS33.** The test set is 252 subscribers, and one subscriber is worth 0.0040. #### How Much the Gap Itself Says The setup produces the table, puts the survey missingness in, and prints the relation of missingness to the label in both sets. ```python # missing.py — MODEL. The fictional subscriber table is regenerated with the same seed; the # satisfaction column is made incomplete as a FICTIONAL survey response, as in the raw record. import math SEED, CANDIDATES, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10), ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05), ("central", 0.18, 23, 0.05)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 SUBSCRIBERS, READINGS = [], {} for i in range(CANDIDATES): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) satisfaction = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() >= 0.046: SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household, "satisfaction": satisfaction}) for k in SUBSCRIBERS: r = rng(SEED + 7000 + k["id"]) for d in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): READINGS.setdefault(k["id"], []).append(0.0 if r() < 0.038 else math.floor( k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - d * 0.05) * 100 + 0.5) / 100) r() DATA = [] for k in SUBSCRIBERS: v = READINGS.get(k["id"]) if v is None: continue mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"]) z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3] + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30) x = {"household": k["household"], "satisfaction": k["satisfaction"], "period": len(v), "mean_consumption": round(mean, 2), "volatility": round(vol, 2), "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35)} for b in REGION: x["b_" + b[0]] = int(k["region"][0] == b[0]) DATA.append(x) survey_rng = rng(SEED + 31000) # not answering the survey is FICTIONAL for x in DATA: x["survey"] = None if survey_rng() < (0.55 if x["suspicious"] else 0.15) else x["satisfaction"] def shuffle(data, seed): r, s = rng(seed), list(range(len(data))) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return [data[i] for i in s] K = shuffle(DATA, SEED + 90000) TRAIN, TEST = K[:756], K[1008:] BASELINE = sum(x["suspicious"] == 0 for x in TEST) / len(TEST) FIELDS = ["mean_consumption", "volatility", "household", "satisfaction", "period", "zero_readings", "b_north", "b_south", "b_east", "b_west", "b_central"] print(f"{'set':<8} {'rows':>6} {'missing':>7} {'missing rate':>12} " f"{'suspicious, missing':>19} {'complete':>9}") for name, s in (("train", TRAIN), ("test", TEST)): e = [x for x in s if x["survey"] is None] t = [x for x in s if x["survey"] is not None] print(f"{name:<8} {len(s):>6} {len(e):>7} {len(e) / len(s):>12.4f} " f"{sum(x['suspicious'] for x in e) / len(e):>19.4f} " f"{sum(x['suspicious'] for x in t) / len(t):>9.4f}") d = sorted(x["survey"] for x in TRAIN if x["survey"] is not None) MEDIAN = d[len(d) // 2] # FIT: only from the training set print(f"median fitted on train {MEDIAN}, baseline {BASELINE:.4f}") ``` ``` set rows missing missing rate suspicious, missing complete train 756 163 0.2156 0.5276 0.1383 test 252 58 0.2302 0.6207 0.1289 median fitted on train 4, baseline 0.7579 ``` The last two columns are the whole subject of the lesson. The suspicious rate is 0.5276 among subscribers who did not answer the survey and 0.1383 among those who did. The gap is not empty; it says something other than **the value the column carries**. The imputation decision either keeps this difference or erases it. The imputation value is itself a fitted object. The median of the filled rows in the training set comes out as 4, and this single number is applied to both sets. Looking at the median of the test set itself would have produced a more "correct" imputation, but computing that value would be looking at the held-out set; the discipline set up in scaling applies here exactly the same. It is also expected that the missingness rate does not come out the same in the two sets: 0.2156 in train, 0.2302 in test, and the difference comes from the randomness of the split. #### Three Strategies, Three Numbers The code below writes the model and measures five strategies together with an upper bound row. ```python def gini(s): p = sum(x["suspicious"] for x in s) / len(s) if s else 0.0 return 2 * p * (1 - p) def tree(s, depth, fields, min_size=2): # MODEL: decision tree, depth 6 p = sum(x["suspicious"] for x in s) / len(s) best = None if depth and len(s) >= 2 * min_size and 0.0 < p < 1.0: for a in fields: d = sorted({x[a] for x in s}) for v in (d[1:] if len(d) < 10 else [d[int(i * len(d) / 10)] for i in range(1, 10)]): left = [x for x in s if x[a] < v] right = [x for x in s if x[a] >= v] if min(len(left), len(right)) < min_size: continue k = gini(s) - (len(left) * gini(left) + len(right) * gini(right)) / len(s) if best is None or k > best[0]: best = (k, a, v, left, right) if best is None or best[0] <= 1e-9: return {"prediction": int(p > 0.5)} return {"field": best[1], "threshold": best[2], "left": tree(best[3], depth - 1, fields, min_size), "right": tree(best[4], depth - 1, fields, min_size)} def predict(d, x): while "prediction" not in d: d = d["left"] if x[d["field"]] < d["threshold"] else d["right"] return d["prediction"] def impute(s, value): # APPLY: a single value for the gap return [dict(x, satisfaction=(value if x["survey"] is None else x["survey"])) for x in s] def add_indicator(s, value): # APPLY: imputation + missingness column return [dict(x, satisfaction=(value if x["survey"] is None else x["survey"]), survey_missing=int(x["survey"] is None)) for x in s] RESULT = {} def measure(name, fields, e, s): m = tree(e, 6, fields) train_acc = sum(predict(m, x) == x["suspicious"] for x in e) / len(e) test_acc = sum(predict(m, x) == x["suspicious"] for x in s) / len(s) RESULT[name] = (len(e), len(fields), train_acc, test_acc) print(f"{name:<35} {len(e):>7} {len(fields):>7} {train_acc:>8.4f} {test_acc:>8.4f}") IMPUTED_TRAIN, IMPUTED_TEST = impute(TRAIN, MEDIAN), impute(TEST, MEDIAN) print(f"{'procedure':<35} {'rows':>7} {'columns':>7} {'train':>8} {'test':>8}") measure("column dropped entirely", [a for a in FIELDS if a != "satisfaction"], TRAIN, TEST) measure("imputed with median", FIELDS, IMPUTED_TRAIN, IMPUTED_TEST) measure("imputation + missingness indicator", FIELDS + ["survey_missing"], add_indicator(TRAIN, MEDIAN), add_indicator(TEST, MEDIAN)) measure("imputed with constant -1", FIELDS, impute(TRAIN, -1), impute(TEST, -1)) measure("missing rows dropped from train", FIELDS, [x for x in IMPUTED_TRAIN if x["survey"] is not None], IMPUTED_TEST) measure("complete column (upper bound)", FIELDS, TRAIN, TEST) ``` ``` procedure rows columns train test column dropped entirely 756 10 0.8439 0.7540 imputed with median 756 11 0.8466 0.7619 imputation + missingness indicator 756 12 0.8823 0.8333 imputed with constant -1 756 11 0.8823 0.8333 missing rows dropped from train 593 11 0.9191 0.7659 complete column (upper bound) 756 11 0.8466 0.7778 ``` Median imputation gives 0.7619: only one subscriber above the baseline. Dropping the column entirely gives 0.7540: below the baseline. Between the two there is a difference of two subscribers, and this is below the measurement resolution. The two common options cannot be told apart in this table. The train column should be read separately, because it divides the strategies into two groups. Imputation and column dropping stay around 0.84 on train; the indicator and the constant value rise to 0.8823; row dropping rises to 0.9191. The rise of the first two on train finds its counterpart on the held-out set, that of the third does not. What kind of increase it is on the training set can be understood only when the held-out set number is written next to it; this is the form in which the rule of the previous course works in this table. #### The Contribution Comes from the Gap Reading the differences against the imputation row separates which decision is really a step. ```python t = RESULT["imputed with median"] print(f"{'procedure':<35} {'contribution':>12} {'subscribers':>11}") for name in ("column dropped entirely", "imputation + missingness indicator", "imputed with constant -1", "missing rows dropped from train", "complete column (upper bound)"): k = RESULT[name][3] - t[3] print(f"{name:<35} {k:>+12.4f} {round(k * 252):>11}") ae = [x for x in TRAIN if x["survey"] is None] print(f"\nof the {len(ae)} rows dropped from train, {sum(x['suspicious'] for x in ae)} are " f"suspicious; suspicious left in train " f"{sum(x['suspicious'] for x in TRAIN) - sum(x['suspicious'] for x in ae)} / " f"{sum(x['suspicious'] for x in TRAIN)}") print(f"in test, a prediction must still be produced for the " f"{sum(1 for x in TEST if x['survey'] is None)} subscribers whose survey is missing; " f"row dropping cannot be applied on the held-out set") ``` ``` procedure contribution subscribers column dropped entirely -0.0079 -2 imputation + missingness indicator +0.0714 18 imputed with constant -1 +0.0714 18 missing rows dropped from train +0.0040 1 complete column (upper bound) +0.0159 4 of the 163 rows dropped from train, 86 are suspicious; suspicious left in train 82 / 168 in test, a prediction must still be produced for the 58 subscribers whose survey is missing; row dropping cannot be applied on the held-out set ``` One row is worth reading: the missingness indicator is +0.0714, that is, eighteen subscribers. This is the largest contribution of the course so far, and where it comes from is clear. While imputation closes the gap, it also erases which rows were empty; the indicator puts that information back. In a table where the suspicious rate among subscribers who did not answer the survey is 0.5276, "not having answered" says more in itself than the satisfaction score. The proof of this is in the upper bound row. Without any missingness the model would read 0.7778; the indicator **exceeds** it by 0.0555. If a feature gives a better number than its complete form, the contribution comes not from the value of the column but from the way the column is missing. Imputation with a constant value gives the same number, 0.8333. The reason is the way the tree works: since the −1 given to the missing rows is a single value outside the scale, the tree can separate it with a single threshold and turns the column into an implicit indicator. This equality has been measured here **for the tree**; for a method that computes distance by summing the columns or multiplies them by a weight, −1 is an ordinary number, and its result should be measured separately. The row dropping line carries two separate problems. It reads 0.9191 on train but stays at 0.7659 on test; on a set reduced to 593 rows the tree memorizes more. The real problem is in the second output: 86 of the 163 rows dropped are suspicious, that is, more than half of the minority class is dropped, and of the 168 suspicious in the training set only 82 remain. The third and decisive obstacle is this: there are 58 subscribers in the test set whose survey is missing, and a prediction has to be produced for them too. Dropping rows is an option on training, it is not an option on the held-out set; this is why it can never be a strategy on its own. The source of the contribution must also be asked. Not answering the survey is a fact known at prediction time and, as it stands, a real feature; the contribution comes from learning. But if the missingness arises **as a result** of the inspection — if a subscriber found suspicious stops answering the survey — then the same column turns into a feature that sees the future and the contribution comes from leakage. The two cases are identical in the table. The distinction is made only by knowing **when** the missingness arises, and that information is not in the data. #### Summary - The suspicious rate is 0.5276 among subscribers who did not answer the survey and 0.1383 among those who did; the missingness is related to the label, and imputation erases this relation. - Median imputation gives 0.7619 and dropping the column entirely gives 0.7540; the two-subscriber difference between them is below the measurement resolution, and the two options cannot be told apart. - Adding a missingness indicator next to the imputation gives 0.8333, a contribution of +0.0714, that is, eighteen subscribers; this number also exceeds by 0.0555 the upper bound of 0.7778 that would be read if there had been no missingness at all. - Imputation with a constant value gives the same number as the indicator on the tree, because a single value outside the scale is an implicit indicator for the tree; this equality has not been measured for other model families. - Dropping missing rows from training removes more than half of the minority class (82 of 168 suspicious remain) and cannot be applied on the held-out set at all, because a prediction is needed for the 58 subscribers in the test set. #### Next Step In this lesson the dropping of half of the minority class was seen as a side effect, but the size difference between the classes itself was never measured. In the training set 168 of 756 subscribers are suspicious, and in the test set 61 of 252. At this rate accuracy is a misleading measure: the baseline that marks no one as suspicious reads 0.7579, and the number of suspicious subscribers it catches is zero. The next lesson prints these two numbers side by side and measures the three answers given to the imbalance — oversampling the minority, undersampling the majority and class weight — both in accuracy and in the number of suspicious subscribers caught. The theory of measures and the definitions of precision and recall are not the subject of this course and are left to the Supervised Learning course. ### Imbalanced Classes Source: https://academia.sh/en/courses/feature-engineering/imbalanced-classes Reading three answers to class imbalance with two measures at once: the baseline gives 0.7579 in accuracy while the number of suspicious subscribers it catches is zero; the model on the raw training set gives 0.7778 accuracy and catches 21 of 61 suspicious subscribers. Oversampling the minority twice lowers accuracy to 0.7619 and raises the caught count to 33, undersampling the majority gives 0.7143 and 36, and class weight 3.5 gives 0.6349 and 39. Oversampling the minority twice and class weight 2.0 produce the same prediction for all 252 subscribers in the test set. The previous lesson recorded as a side effect that dropping the missing rows took more than half of the minority class. The size difference between the classes itself was never measured. In the training set 168 of 756 subscribers are suspicious, and in the test set 61 of 252. On this table **accuracy** is a misleading measure, and the size of the deception can be counted: the baseline that marks no one as suspicious reads 0.7579, and the number of suspicious subscribers it catches is zero. This lesson measures three answers to imbalance: **oversampling** the minority, **undersampling** the majority and giving a **class weight**. Each answer is read with two numbers at once; the theory of measures is not the subject of this course. The share of correctly flagged minority is called **precision**, and the share of caught minority is called **recall**; their definitions, how they are read against each other and the choice of a threshold are left to the Supervised Learning course. Here only raw numbers are printed. - **PS34.** The set, the split, the features and the model are the same as in the earlier lessons: 756/252/252, a depth 6 decision tree, eleven features. - **PS35.** Two measures are printed: **accuracy** and the **number of suspicious subscribers caught**, that is, how many of the 61 suspicious subscribers in the test set were flagged as suspicious. Two more raw numbers are written next to them: the suspicious subscribers missed, and the subscribers flagged as suspicious who are not. - **PS36.** The baseline does not change: telling every subscriber the most frequent class of the training set. This procedure gives 0.7579 in accuracy and 0 in caught. - **PS37.** Oversampling rewrites the minority rows as they are; no new rows are **generated**. - **PS38.** Undersampling reduces the majority rows at random and shrinks the training set; the discarded rows do not come back. - **PS39.** The class weight makes the tree compute its split criterion with weights: the contribution of each row is multiplied by its own weight. In this lesson the model code is written so that it accepts weights. - **PS40.** All three answers are applied **only to the training set**. The test set is never oversampled, undersampled or weighted in any case; the measure is read on the real distribution. #### Two Numbers, One Baseline The setup produces the set and prints the baseline in two measures at once. ```python # imbalanced.py — MODEL. The fictional subscriber table is regenerated with the same seed. import math SEED, CANDIDATES, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10), ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05), ("central", 0.18, 23, 0.05)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 SUBSCRIBERS, READINGS = [], {} for i in range(CANDIDATES): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) satisfaction = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() >= 0.046: SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household, "satisfaction": satisfaction}) for k in SUBSCRIBERS: r = rng(SEED + 7000 + k["id"]) for d in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): READINGS.setdefault(k["id"], []).append(0.0 if r() < 0.038 else math.floor( k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - d * 0.05) * 100 + 0.5) / 100) r() DATA = [] for k in SUBSCRIBERS: v = READINGS.get(k["id"]) if v is None: continue mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"]) z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3] + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30) x = {"household": k["household"], "satisfaction": k["satisfaction"], "period": len(v), "mean_consumption": round(mean, 2), "volatility": round(vol, 2), "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35)} for b in REGION: x["b_" + b[0]] = int(k["region"][0] == b[0]) DATA.append(x) def shuffle(data, seed): r, s = rng(seed), list(range(len(data))) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return [data[i] for i in s] K = shuffle(DATA, SEED + 90000) TRAIN, TEST = K[:756], K[1008:] FIELDS = ["mean_consumption", "volatility", "household", "satisfaction", "period", "zero_readings", "b_north", "b_south", "b_east", "b_west", "b_central"] N_TRAIN, N_TEST = sum(x["suspicious"] for x in TRAIN), sum(x["suspicious"] for x in TEST) print(f"train {len(TRAIN)} subscribers, suspicious {N_TRAIN} ({N_TRAIN / len(TRAIN):.4f}); " f"test {len(TEST)} subscribers, suspicious {N_TEST} ({N_TEST / len(TEST):.4f})") print(f"majority / minority ratio in train {(len(TRAIN) - N_TRAIN) / N_TRAIN:.2f}") print(f"baseline (most frequent class): accuracy " f"{sum(x['suspicious'] == 0 for x in TEST) / len(TEST):.4f}, " f"suspicious caught 0 / {N_TEST}, suspicious flagged 0") ``` ``` train 756 subscribers, suspicious 168 (0.2222); test 252 subscribers, suspicious 61 (0.2421) majority / minority ratio in train 3.50 baseline (most frequent class): accuracy 0.7579, suspicious caught 0 / 61, suspicious flagged 0 ``` The last line shows the baseline of the course in a second measure, and the picture changes. 0.7579 in accuracy is a number that looks good; zero in the number of suspicious subscribers caught is an unusable result. The same procedure, the same set, two measures, two different verdicts. On an imbalanced set, not having chosen the measure is the same as not having chosen the baseline. #### Three Answers The code below writes the tree so that it accepts weights and measures the three answers in turn. ```python def gini(s): n = sum(x.get("w", 1.0) for x in s) if n <= 0: return 0.0 p = sum(x.get("w", 1.0) * x["suspicious"] for x in s) / n return 2 * p * (1 - p) def tree(s, depth, fields, min_size=2): # MODEL: weighted decision tree n = sum(x.get("w", 1.0) for x in s) p = sum(x.get("w", 1.0) * x["suspicious"] for x in s) / n best = None if depth and len(s) >= 2 * min_size and 0.0 < p < 1.0: for a in fields: d = sorted({x[a] for x in s}) for v in (d[1:] if len(d) < 10 else [d[int(i * len(d) / 10)] for i in range(1, 10)]): left = [x for x in s if x[a] < v] right = [x for x in s if x[a] >= v] if min(len(left), len(right)) < min_size: continue k = gini(s) - (sum(x.get("w", 1.0) for x in left) * gini(left) + sum(x.get("w", 1.0) for x in right) * gini(right)) / n if best is None or k > best[0]: best = (k, a, v, left, right) if best is None or best[0] <= 1e-9: return {"prediction": int(p > 0.5)} return {"field": best[1], "threshold": best[2], "left": tree(best[3], depth - 1, fields, min_size), "right": tree(best[4], depth - 1, fields, min_size)} def predict(d, x): while "prediction" not in d: d = d["left"] if x[d["field"]] < d["threshold"] else d["right"] return d["prediction"] def oversample(train, times): # each minority row is written `times` times out = [] for x in train: out += [x] * (times if x["suspicious"] else 1) return out def undersample(train, seed): # the majority is reduced to the minority size minority = [x for x in train if x["suspicious"]] majority = shuffle([x for x in train if not x["suspicious"]], seed)[:len(minority)] return shuffle(minority + majority, seed + 1) def weighted(train, w): return [dict(x, w=(w if x["suspicious"] else 1.0)) for x in train] RESULT = {} def measure(name, train): m = tree(train, 6, FIELDS) t = [predict(m, x) for x in TEST] d = sum(t[i] == TEST[i]["suspicious"] for i in range(len(TEST))) / len(TEST) c = sum(1 for i, x in enumerate(TEST) if t[i] == 1 and x["suspicious"] == 1) RESULT[name] = (len(train), d, c, sum(t)) print(f"{name:<36} {len(train):>7} {d:>9.4f} {c:>10} {sum(t):>10}") print(f"{'procedure':<36} {'train':>7} {'accuracy':>9} {'caught':>10} " f"{'flagged':>10}") print(f"{'baseline (most frequent class)':<36} {'-':>7} " f"{sum(x['suspicious'] == 0 for x in TEST) / len(TEST):>9.4f} {0:>10} {0:>10}") measure("raw training set", TRAIN) measure("oversampling, minority twice", oversample(TRAIN, 2)) measure("oversampling, minority three times", oversample(TRAIN, 3)) measure("undersampling, majority to minority", undersample(TRAIN, SEED + 22000)) measure("class weight 2.0", weighted(TRAIN, 2.0)) measure("class weight 3.5", weighted(TRAIN, 3.5)) ``` ``` procedure train accuracy caught flagged baseline (most frequent class) - 0.7579 0 0 raw training set 756 0.7778 21 37 oversampling, minority twice 924 0.7619 33 65 oversampling, minority three times 1092 0.7341 34 74 undersampling, majority to minority 336 0.7143 36 83 class weight 2.0 756 0.7619 33 65 class weight 3.5 756 0.6349 39 109 ``` The two columns move in opposite directions. While accuracy falls from 0.7778 to 0.6349, the number of suspicious subscribers caught rises from 21 to 39. With the raw training set the model misses 40 of the 61 suspicious subscribers; with the harshest weight it misses 22 of them, but the number of subscribers it flags as suspicious rises from 37 to 109. The flagged column says something on its own, too. There are 61 suspicious subscribers in the test set, while the model trained on the raw training set flags only 37 subscribers. The model flags the minority **too little**, and this is not a tuning mistake but the direct result of the split criterion: the criterion rewards saying the majority at a leaf, and as long as the minority share stays below fifty percent, a leaf saying minority pays off only when a very pure region is found. All three answers to imbalance break exactly this calculation: some by writing a row a second time, some by reducing the opposite class, some by changing the weight directly, they pull down the threshold at which leaves say minority. The Statistical Analysis and Exploratory Data Analysis course of the Data Analytics curriculum measured the traps of reading proportions in imbalanced sets; the difference here is that it is not the proportion but the **decision** that changes. #### In Which Number the Contribution Is Measured When the same decisions are read against the raw training set, the meaning of the table becomes clear. ```python h = RESULT["raw training set"] print(f"{'procedure':<36} {'accuracy change':>15} {'caught +':>12} " f"{'missed':>10} {'false flags':>14}") for name in ("oversampling, minority twice", "oversampling, minority three times", "undersampling, majority to minority", "class weight 2.0", "class weight 3.5"): v = RESULT[name] print(f"{name:<36} {v[1] - h[1]:>+15.4f} {v[2] - h[2]:>+12} " f"{N_TEST - v[2]:>10} {v[3] - v[2]:>14}") a = tree(oversample(TRAIN, 2), 6, FIELDS) b = tree(weighted(TRAIN, 2.0), 6, FIELDS) print(f"\nsubscribers given the same prediction by oversampling twice and class weight 2.0 " f"{sum(predict(a, x) == predict(b, x) for x in TEST)} / {len(TEST)}") ``` ``` procedure accuracy change caught + missed false flags oversampling, minority twice -0.0159 +12 28 32 oversampling, minority three times -0.0437 +13 27 40 undersampling, majority to minority -0.0635 +15 25 47 class weight 2.0 -0.0159 +12 28 32 class weight 3.5 -0.1429 +18 22 70 subscribers given the same prediction by oversampling twice and class weight 2.0 252 / 252 ``` In the accuracy column all five rows are negative. If the rule of the course is applied to the letter, none of these five steps has a contribution and none of them should be taken. But the caught column rises by between twelve and eighteen subscribers, and each of those subscribers is a missed suspicion of illicit use or a fault. The contradiction is not in the procedures but in the choice of the measure. As long as accuracy is chosen, the contribution of every answer to imbalance is negative; when the measure changes, the sign changes too. Which number matters is not read from the data but from the job — how many wrong addresses an inspection team can go to is a question of resources. The undersampling row also opens a question of stability. Had the same undersampling been done with another seed, which 168 majority rows remain would change, and the model would change with it; oversampling and weighting contain no random choice and produce the same tree on the same set in every run. Besides the contribution of a preparation step, **how much that contribution moves from run to run** should also be asked. The cost side of the last two rows is clear. Class weight 3.5 catches eighteen more suspicious subscribers while raising the number of wrongly flagged subscribers from 16 to 70. Undersampling pays a different price: the training set falls from 756 rows to 336, and the information in the 420 discarded majority rows does not come back. On small sets this price is reflected directly in the stability of the model. The last output shows that two procedures are the same thing. Oversampling the minority twice and giving the minority a class weight of 2.0 produce the same prediction for all 252 subscribers in the test set. The reason is in the split criterion of the tree: the criterion is computed with the weighted sum of the rows, and writing a row twice gives the same sum as making its weight two. The only place where the two part ways is the stopping condition that looks at the number of rows — with oversampling the set grows to 924 rows, with weighting it stays at 756 — and in this table that condition did not bind at any node. In memory and time weighting is cheap; oversampling produces the same result with a larger set. #### Summary - The baseline gives two separate numbers on an imbalanced set: 0.7579 in accuracy, zero out of 61 in the number of suspicious subscribers caught; accuracy alone is a misleading baseline. - The model on the raw training set gives 0.7778 accuracy and catches 21 of the 61 suspicious subscribers, and misses 40. - Oversampling, undersampling and class weight all give a negative contribution in accuracy (between −0.0159 and −0.1429) and a positive one of between twelve and eighteen subscribers in the number of suspicious subscribers caught; the sign turns with the measure. - Undersampling takes the training set from 756 rows to 336, and the information in the 420 discarded majority rows does not come back; oversampling and weighting discard no rows. - Oversampling the minority twice and giving a class weight of 2.0 produce the same prediction for all 252 subscribers in the test set, because the split criterion of the tree is computed with the weighted sum. #### Next Step In this lesson the number of suspicious subscribers was a problem because of its scarcity; in the next lesson a few subscribers will be a problem because of their **value**. In the mean consumption column there are subscribers outside the fence computed from the interquartile range of the training set; some of them are real extremes, and some are recording errors produced by stuck meters. The two look the same in the table. The next lesson measures a single clipping decision in two model families, shows that the same decision gives a positive contribution in the slope-based procedure and a negative one in the tree-based procedure, and puts forward with numbers that an outlier can be managed without deleting any row. ### Outlier Management Source: https://academia.sh/en/courses/feature-engineering/outlier-management Measuring that a single clipping decision gives opposite signs in two model families: clipping with a fence learned from the training set adds +0.0040 to the gradient-based linear model on real extremes only and +0.0317 when the stuck meter records are also present; the same decision gives −0.0278 and −0.0357 on the decision tree. The suspicious rate of the 34 subscribers outside the fence is 0.6765, so clipping also erases real information, and 41 values are clipped without deleting a single row. The previous lesson measured the scarcity of suspicious subscribers **in number**. This lesson measures how far a few subscribers are **in value**. In the mean consumption and volatility columns there are subscribers outside the fence computed from the interquartile range of the training set. Some of them are real extremes — they come from a crowded household or from a connection with a high tariff. Some are recording errors produced by stuck meters. In the table the two look the same: a large number. The interquartile range, the fence and the finding of extreme values were set up in the Data Collection and Preparation course of the Data Analytics curriculum; the measure there was how many rows stayed outside the fence. The question here is different: how much does the **clipping decision** move the number on the held-out set, and in which model. - **PS41.** The stuck meter fault is **fictional**: the mean consumption of a small set of subscribers is recorded as thirty times higher. It is chosen independently of the label. - **PS42.** This fault is different from the three defects in `01` and cannot be caught by their fixes: the value is in the right unit, written correctly and in a single row. It is only wrong. - **PS43.** **The fence is a fitted object**: the quartiles are computed only from the training set, the fence is built with 1.5 times the interquartile range, and the same fence is applied to both sets. - **PS44.** Clipping pulls a value outside the fence to the fence. **No row is deleted**; a prediction is produced for every subscriber in the test set. - **PS45.** Two model families are compared: a gradient-based linear model and a depth 6 decision tree. The linear model is trained with standardized columns according to the discipline set up in the previous course; the tree is trained with raw columns, because the contribution of scaling on a tree is zero. #### The Two Sources of an Extreme The setup produces the table, puts the fault in and learns the fence from the training set. ```python # outlier.py — MODEL. The fictional subscriber table is regenerated with the same seed; the # consumption of a small set of subscribers is recorded thirty times higher through a FICTIONAL # stuck meter fault. import math import statistics SEED, CANDIDATES, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10), ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05), ("central", 0.18, 23, 0.05)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 SUBSCRIBERS, READINGS = [], {} for i in range(CANDIDATES): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) satisfaction = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() >= 0.046: SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household, "satisfaction": satisfaction}) for k in SUBSCRIBERS: r = rng(SEED + 7000 + k["id"]) for d in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): READINGS.setdefault(k["id"], []).append(0.0 if r() < 0.038 else math.floor( k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - d * 0.05) * 100 + 0.5) / 100) r() DATA = [] for k in SUBSCRIBERS: v = READINGS.get(k["id"]) if v is None: continue mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"]) z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3] + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30) x = {"household": k["household"], "satisfaction": k["satisfaction"], "period": len(v), "mean_consumption": round(mean, 2), "volatility": round(vol, 2), "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35)} for b in REGION: x["b_" + b[0]] = int(k["region"][0] == b[0]) DATA.append(x) fault_rng = rng(SEED + 81000) # the stuck meter fault is FICTIONAL for x in DATA: x["fault"] = (round(x["mean_consumption"] * 30, 2) if fault_rng() < 0.010 else x["mean_consumption"]) def shuffle(data, seed): r, s = rng(seed), list(range(len(data))) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return [data[i] for i in s] K = shuffle(DATA, SEED + 90000) TRAIN, TEST = K[:756], K[1008:] BASELINE = sum(x["suspicious"] == 0 for x in TEST) / len(TEST) NUMERIC = ["mean_consumption", "volatility", "household", "satisfaction", "period"] FIELDS = NUMERIC + ["zero_readings", "b_north", "b_south", "b_east", "b_west", "b_central"] EXTREME = ["mean_consumption", "volatility"] def quantile(v, q): s = sorted(v) i = q * (len(s) - 1) a = int(i) return s[a] if a + 1 >= len(s) else s[a] + (i - a) * (s[a + 1] - s[a]) def fit_fence(train, columns, k=1.5): # FIT: interquartile range and fence d = {} for a in columns: q1, q3 = quantile([x[a] for x in train], 0.25), quantile([x[a] for x in train], 0.75) d[a] = (q1 - k * (q3 - q1), q3 + k * (q3 - q1)) return d def clip(c, s): # APPLY: no row is deleted return [dict(x, **{a: min(max(x[a], c[a][0]), c[a][1]) for a in c}) for x in s] faulty = [x for x in TRAIN if x["fault"] != x["mean_consumption"]] print(f"stuck meter: {len(faulty)} subscribers in train, " f"{sum(1 for x in TEST if x['fault'] != x['mean_consumption'])} in test; " f"{sum(x['suspicious'] for x in faulty)} of them are suspicious") c = fit_fence(TRAIN, EXTREME) print(f"{'column':<16} {'upper fence':>11} {'largest':>9} {'outside fence, train':>20} " f"{'test':>7}") for a in EXTREME: print(f"{a:<16} {c[a][1]:>11.2f} {max(x[a] for x in TRAIN):>9.2f} " f"{sum(1 for x in TRAIN if x[a] > c[a][1]):>20} " f"{sum(1 for x in TEST if x[a] > c[a][1]):>7}") u = [x for x in TRAIN if any(x[a] > c[a][1] for a in EXTREME)] print(f"suspicious rate of the {len(u)} subscribers outside the fence " f"{sum(x['suspicious'] for x in u) / len(u):.4f}, baseline {BASELINE:.4f}") ``` ``` stuck meter: 9 subscribers in train, 1 in test; 2 of them are suspicious column upper fence largest outside fence, train test mean_consumption 35.80 42.90 11 4 volatility 29.41 40.57 23 7 suspicious rate of the 34 subscribers outside the fence 0.6765, baseline 0.7579 ``` The last line states the cost of the clipping decision in advance. The suspicious rate of the 34 subscribers outside the fence is 0.6765; over the whole training set this rate is 0.2222. Two thirds of the subscribers at the extreme are suspicious; that is, the extreme values are not noise but the most informative region of the set. #### The Same Clipping, Two Models ```python def gini(s): p = sum(x["suspicious"] for x in s) / len(s) if s else 0.0 return 2 * p * (1 - p) def tree(s, depth, fields, min_size=2): # MODEL: decision tree, depth 6 p = sum(x["suspicious"] for x in s) / len(s) best = None if depth and len(s) >= 2 * min_size and 0.0 < p < 1.0: for a in fields: d = sorted({x[a] for x in s}) for v in (d[1:] if len(d) < 10 else [d[int(i * len(d) / 10)] for i in range(1, 10)]): left = [x for x in s if x[a] < v] right = [x for x in s if x[a] >= v] if min(len(left), len(right)) < min_size: continue k = gini(s) - (len(left) * gini(left) + len(right) * gini(right)) / len(s) if best is None or k > best[0]: best = (k, a, v, left, right) if best is None or best[0] <= 1e-9: return {"prediction": int(p > 0.5)} return {"field": best[1], "threshold": best[2], "left": tree(best[3], depth - 1, fields, min_size), "right": tree(best[4], depth - 1, fields, min_size)} def tree_predict(d, x): while "prediction" not in d: d = d["left"] if x[d["field"]] < d["threshold"] else d["right"] return d["prediction"] def squash(z): if z >= 0: return 1.0 / (1.0 + math.exp(-z)) if z < 700 else 1.0 e = math.exp(z) if z > -700 else 0.0 return e / (1.0 + e) def linear_model(train, fields, step=0.5, rounds=400): # MODEL: gradient-based linear model w, b, n = [0.0] * len(fields), 0.0, len(train) for _ in range(rounds): gw, gb = [0.0] * len(fields), 0.0 for x in train: h = squash(sum(w[i] * x[a] for i, a in enumerate(fields)) + b) - x["suspicious"] for i, a in enumerate(fields): gw[i] += h * x[a] gb += h for i in range(len(fields)): w[i] -= step * gw[i] / n b -= step * gb / n return w, b def standardize(train, sets): # FIT + APPLY (previous lesson) p = {a: (statistics.fmean(x[a] for x in train), statistics.pstdev([x[a] for x in train])) for a in NUMERIC} return [[dict(x, **{a: (x[a] - p[a][0]) / p[a][1] for a in NUMERIC}) for x in k] for k in sets] def build(source, clipping): e = [dict(x, mean_consumption=x[source]) for x in TRAIN] s = [dict(x, mean_consumption=x[source]) for x in TEST] if clipping: c = fit_fence(e, EXTREME) e, s = clip(c, e), clip(c, s) return e, s RESULT = {} print(f"{'case':<38} {'gradient-based':>14} {'tree':>8}") print(f"{'baseline':<38} {BASELINE:>14.4f} {BASELINE:>8.4f}") for name, source, clipping in (("real extremes only, no clipping", "mean_consumption", False), ("real extremes only, clipped with fence", "mean_consumption", True), ("stuck meter too, no clipping", "fault", False), ("stuck meter too, clipped with fence", "fault", True)): e, s = build(source, clipping) es, ss = standardize(e, [e, s]) m = linear_model(es, FIELDS) d = sum(int(squash(sum(m[0][i] * x[a] for i, a in enumerate(FIELDS)) + m[1]) > 0.5) == x["suspicious"] for x in ss) / len(ss) t = tree(e, 6, FIELDS) g = sum(tree_predict(t, x) == x["suspicious"] for x in s) / len(s) RESULT[name] = (d, g) print(f"{name:<38} {d:>14.4f} {g:>8.4f}") ``` ``` case gradient-based tree baseline 0.7579 0.7579 real extremes only, no clipping 0.8016 0.7778 real extremes only, clipped with fence 0.8056 0.7500 stuck meter too, no clipping 0.7698 0.7857 stuck meter too, clipped with fence 0.8016 0.7500 ``` The tree column gives three different numbers in four cases, and what is interesting is that the two clipped rows read the same number. Clipping brings both tables to the same place: when all the values above the fence are gathered at a single point, the faulty record and the real extreme become indistinguishable. #### How Many Subscribers the Contribution Is Worth ```python print(f"{'clipping decision':<20} {'gradient-based':>14} {'subscribers':>11} " f"{'tree':>8} {'subscribers':>11}") for name, a, b in (("real extremes only", "real extremes only, no clipping", "real extremes only, clipped with fence"), ("stuck meter too", "stuck meter too, no clipping", "stuck meter too, clipped with fence")): kd, kg = RESULT[b][0] - RESULT[a][0], RESULT[b][1] - RESULT[a][1] print(f"{name:<20} {kd:>+14.4f} {round(kd * 252):>11} " f"{kg:>+8.4f} {round(kg * 252):>11}") e, s = build("fault", False) c = fit_fence(e, EXTREME) print(f"\nupper fence learned on the faulty table: mean_consumption " f"{c['mean_consumption'][1]:.2f}, volatility {c['volatility'][1]:.2f}") print(f"values clipped: train {sum(1 for x in e for a in EXTREME if x[a] > c[a][1])}, " f"test {sum(1 for x in s for a in EXTREME if x[a] > c[a][1])}; " f"rows deleted 0") print(f"cost of the fault to the gradient-based model " f"{RESULT['stuck meter too, no clipping'][0] - RESULT['real extremes only, no clipping'][0]:+.4f}, " f"to the tree {RESULT['stuck meter too, no clipping'][1] - RESULT['real extremes only, no clipping'][1]:+.4f}") print(f"features used in six lessons {len(FIELDS)}, new quantities produced 0") ``` ``` clipping decision gradient-based subscribers tree subscribers real extremes only +0.0040 1 -0.0278 -7 stuck meter too +0.0317 8 -0.0357 -9 upper fence learned on the faulty table: mean_consumption 36.20, volatility 29.41 values clipped: train 41, test 12; rows deleted 0 cost of the fault to the gradient-based model -0.0317, to the tree +0.0079 features used in six lessons 11, new quantities produced 0 ``` The signs are opposite in both rows. Clipping gives a positive contribution in the gradient-based model and a negative one in the tree, and when the stuck meter is present both directions rise to a readable size: eight subscribers and nine subscribers. The mechanism is in the way the two models handle an extreme value. The gradient-based model sums the error of all rows in every round and shifts the weights according to that sum; a row thirty times away from the mean determines that sum alone and pulls all eleven weights at once. The tree, on the other hand, can set an extreme row aside with a single split and leaves the rest of the set untouched — but clipping removes that split, because the ordering above the fence is erased, and the suspicious rate of 0.6765 of the 34 subscribers remaining there can no longer be turned into a threshold. The fault row confirms this from the second side. A recording error of nine rows makes the gradient-based model lose 0.0317 and makes the tree gain 0.0079 — that is, for the tree an unmeasured difference. When clipping is applied the gradient-based model returns exactly to 0.8016, the number it would have read if the fault had never existed. The same step on the same data fully undoes the damage in one model, while it lowers the other down to 0.0079 above the baseline. The procedural side of clipping is also worth noting. The fence was learned only from the training set, the upper fence moved to 36.20 on the faulty table, 41 values were clipped and **zero rows were deleted**. For an extreme of unknown source there is no single right answer; the only thing that can be measured is how many subscribers that answer is worth in which model. #### Summary - The suspicious rate of the 34 subscribers outside the fence is 0.6765, over the whole set 0.2222; in this table the extreme values are not noise but the most informative region. - The same clipping decision gives a positive contribution in the gradient-based linear model and a negative one in the decision tree: +0.0040 and −0.0278 on real extremes only, +0.0317 and −0.0357 when the stuck meter record is also present. - The gradient-based model updates its weights by summing the error of all rows, and a row thirty times away from the mean determines that sum alone; while the tree can separate an extreme row with a single split, clipping removes that split. - A recording error of nine rows makes the gradient-based model lose 0.0317 and leaves an unmeasured difference of 0.0079 on the tree; clipping brings the gradient-based model back exactly to 0.8016 of the fault-free state. - The fence was learned only from the training set, 41 values were clipped and no row was deleted; a prediction was produced for every subscriber in the test set. #### Next Step The last line of the last output states the common limit of the six lessons: the number of features used is eleven and the number of new quantities produced is zero. Everything done in the six lessons was changing **the way an existing field is read** — fixing the unit, learning the scale from training, turning a category into a number, writing the gap itself into a column, changing the weight of rows, pulling the extreme to the fence. In no lesson was the ratio, difference or product of two columns taken; no seasonal indicator was extracted from a date and no length from a text. And nothing was done in the opposite direction either: no column was removed from the table on the grounds that "its contribution was not measured", and the eleven features stayed as they were throughout six lessons. The next topic takes on these two jobs. First, new columns are derived from domain knowledge and the contribution of each is measured one by one — those that come out at zero are written down too. Then the opposite question is asked: which columns should be removed from the table, how many features do three selection methods choose on the same set, and how much does **the selection itself** inflate the number on the held-out set. ## Feature Generation and Selection Counting what generating new columns, dropping columns and binding the preparation to an object add on the held-out set: the individual contribution of ratio, difference, interaction and threshold indicators and those whose contribution comes out zero, the number of numbers that come out of a date and a short text, the number of features chosen by filter, wrapper and embedded methods and the inflation produced when the selection is made from the whole set, the relation between the number of components and the held-out number and what is lost from interpretability, the number of steps learned from the whole set without a pipeline, the two cross-validation numbers given by preparing outside the folds and inside the folds. ### Feature Generation Source: https://academia.sh/en/courses/feature-engineering/feature-generation Counting one by one what new columns derived from existing columns add on the held-out set: the ratio feature carries the test number from 0.7778 to 0.7897 while the difference and the indicator learned from the training set add 0.0040, the hand-set 25 m³ threshold loses 0.0159 and the product of two columns loses 0.0238, and the 40 m³ threshold produces exactly zero contribution. When the six columns are added cumulatively the training number rises from 0.8466 to 0.8677 while the held-out number peaks at 0.7897 and falls to 0.7698; the three columns that each add a positive contribution on their own add a total contribution of 0.0000 when added together. The previous six lessons worked on existing columns: missing values were imputed, the scale was learned only from the training set, categories were encoded, imbalance was handled, outliers were assessed according to the model. In all of them the set of columns stayed the same. No **new column** was generated and no column was dropped; in every lesson the model looked at the same eleven numbers. This lesson increases the number of columns for the first time. Its question is this: whether a new number derived from existing columns by arithmetic **adds information** to the model or only **widens the search space**. The rule of the course holds here too — a preparation step exists to the extent of its contribution measured on the held-out set — and in this lesson six features are generated one by one and measured one by one. Those whose contribution comes out zero or negative are also written into the table. - **FS1.** The dataset is the **fictional** subscriber table of the Introduction to Machine Learning course: the same seed (20260218), the same split (756 training, 252 validation, 252 test), the same label. The held-out set is the test set. - **FS2.** The model is held **fixed** throughout the course: a decision tree of depth 6. The only thing that changes is the set of columns. The baseline is telling the most frequent class and it is 0.7579 on the test set. - **FS3.** The line before the decision is the line of the eleven columns the preparation steps left behind: 0.8466 on training, 0.7778 on test. - **FS4.** The raw readings are still kept. `mean_consumption` and `volatility` are a summary of those readings, and that summarizing was itself a feature generation decision; the lesson reopens this decision. - **FS5.** Features computed at the row level (ratio, difference, product, fixed threshold) learn nothing from the dataset and are independent of the split. - **FS6.** A feature that learns a number from the dataset — being above the region median — reads that number **only from the training set** and applies it to all three sets with the same number. - **FS7.** Contribution is the difference of the same model's number on the held-out set from the line before the decision. In a 252-row set, one row corresponds to 0.0040. #### Six New Numbers from the Same Columns The setup is the same as in the Introduction to Machine Learning course and produces the same set with the same seed. This time the list of raw readings is also kept, because one of the features to be generated depends on it. ```python # generation.py — MODEL: the FICTIONAL subscriber table of the Introduction to Machine Learning course is rebuilt with the same seed; # the model is a decision tree of depth 6 and stays fixed throughout the course. import math SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10), ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05), ("central", 0.18, 23, 0.05)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 SUBSCRIBERS, DATA = [], [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() >= 0.046: SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household, "satisfaction": rating}) for k in SUBSCRIBERS: r, v = rng(SEED + 7000 + k["id"]), [] for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): v.append(0.0 if r() < 0.038 else math.floor( k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - n * 0.05) * 100 + 0.5) / 100) r() if not v: continue mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"]) z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3] + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30) x = {"id": k["id"], "household": k["household"], "satisfaction": k["satisfaction"], "period": len(v), "mean_consumption": round(mean, 2), "volatility": round(vol, 2), "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35), "readings": [round(q, 2) for q in v], "region_name": k["region"][0]} for b in REGION: x["area_" + b[0]] = int(k["region"][0] == b[0]) DATA.append(x) BASE_FIELDS = ["mean_consumption", "volatility", "household", "satisfaction", "period", "zero_readings", "area_north", "area_south", "area_east", "area_west", "area_central"] def shuffle(data, seed): r, s = rng(seed), list(range(len(data))) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return [data[i] for i in s] K = shuffle(DATA, SEED + 90000) TRAIN, VAL, TEST = K[:756], K[756:1008], K[1008:] def gini(rows): p = sum(x["suspicious"] for x in rows) / len(rows) if rows else 0.0 return 2 * p * (1 - p) def tree(rows, depth, fields, min_size=2): p, best = sum(x["suspicious"] for x in rows) / len(rows), None if depth and len(rows) >= 2 * min_size and 0.0 < p < 1.0: for f in fields: values = sorted({x[f] for x in rows}) for v in (values[1:] if len(values) < 10 else [values[int(i * len(values) / 10)] for i in range(1, 10)]): left = [x for x in rows if x[f] < v] right = [x for x in rows if x[f] >= v] if min(len(left), len(right)) < min_size: continue k = gini(rows) - (len(left) * gini(left) + len(right) * gini(right)) / len(rows) if best is None or k > best[0]: best = (k, f, v, left, right) if best is None or best[0] <= 1e-9: return {"prediction": int(p > 0.5)} return {"field": best[1], "threshold": best[2], "left": tree(best[3], depth - 1, fields, min_size), "right": tree(best[4], depth - 1, fields, min_size)} def predict(node, x): while "prediction" not in node: node = node["left"] if x[node["field"]] < node["threshold"] else node["right"] return node["prediction"] def accuracy(node, rows): return sum(predict(node, x) == x["suspicious"] for x in rows) / len(rows) def baseline(train, rows): c = int(sum(x["suspicious"] for x in train) * 2 > len(train)) return sum(x["suspicious"] == c for x in rows) / len(rows) def measure(fields, depth=6): m = tree(TRAIN, depth, fields) return accuracy(m, TRAIN), accuracy(m, TEST) ``` The six features to be generated represent five types. **Ratio** is the quotient of two columns: consumption per person separates the size of the household from consumption. **Difference** is the change between the last and the first of the raw readings; `volatility` carries only the span, not the direction. The **interaction feature** is the product of two columns. The **threshold indicator** comes from domain knowledge: the tariff tiers stop at 25 and 40 m³. The last one **learns** a number from the dataset — whether the subscriber is above the median of their own region — and that median is read from the training set only. ```python for x in DATA: o = x["readings"] x["per_person"] = round(x["mean_consumption"] / x["household"], 3) x["last_first_diff"] = round(o[-1] - o[0], 2) x["household_satisfaction"] = x["household"] * x["satisfaction"] x["threshold_25"] = int(x["mean_consumption"] > 25) x["threshold_40"] = int(x["mean_consumption"] > 40) MEDIAN = {} for b in REGION: v = sorted(x["mean_consumption"] for x in TRAIN if x["region_name"] == b[0]) MEDIAN[b[0]] = v[len(v) // 2] for x in DATA: x["region_above"] = int(x["mean_consumption"] > MEDIAN[x["region_name"]]) BEFORE = measure(BASE_FIELDS) NEW = ["per_person", "last_first_diff", "region_above", "threshold_25", "household_satisfaction", "threshold_40"] print(f"{'feature':<22} {'type':<12} {'train':>9} {'held_out':>9} {'contrib':>8}") print(f"{'-- none --':<22} {'11 columns':<12} {BEFORE[0]:>9.4f} {BEFORE[1]:>9.4f}") for a, t in zip(NEW, ["ratio", "difference", "learned", "threshold", "interaction", "threshold"]): e, s = measure(BASE_FIELDS + [a]) print(f"{a:<22} {t:<12} {e:>9.4f} {s:>9.4f} {s - BEFORE[1]:>+8.4f}") print(f"\nbaseline {baseline(TRAIN, TEST):.4f}; above 25 m3 {sum(x['threshold_25'] for x in DATA)} " f"rows, above 40 m3 {sum(x['threshold_40'] for x in DATA)} rows") ``` ``` feature type train held_out contrib -- none -- 11 columns 0.8466 0.7778 per_person ratio 0.8519 0.7897 +0.0119 last_first_diff difference 0.8571 0.7817 +0.0040 region_above learned 0.8492 0.7817 +0.0040 threshold_25 threshold 0.8479 0.7619 -0.0159 household_satisfaction interaction 0.8492 0.7540 -0.0238 threshold_40 threshold 0.8466 0.7778 +0.0000 baseline 0.7579; above 25 m3 243 rows, above 40 m3 2 rows ``` The contribution column carries three signs at once. The largest gain comes from the ratio feature: 0.0119, that is, three subscribers in a 252-row set. This is a column that arithmetically carries no new information — `mean_consumption` and `household` are already in the set — but the tree makes its splits **on a single column** and cannot build the quotient of two columns with one split. What the ratio adds is not information but the **splittable form** of that information. The difference feature and the indicator learned from the training set earn one row each: 0.0040. That is the correct classification of a single subscriber and is too small to be told apart from chance. A number like this cannot be reported as "there is a contribution"; at most it can be reported as "measured and could not be told apart from zero". There is a difference of kind between these two columns, and the second rule of the course passes exactly through it. `last_first_diff` is computed inside each row on its own; it sees no other row in the dataset and gives the same number before and after the split. `region_above`, on the other hand, rests on a number — the region median — and that number is **read** from the data. In the code this distinction sits on a single line: the median dictionary is built over `TRAIN`, not over `DATA`. Had the source been `DATA`, the values of the column would not have changed at all, the 252 rows in the test set would have contributed to determining their own thresholds, and a share of the measured contribution would have come from leakage rather than from learning. A generated feature is also **a learned object**, and writing where it learned is as obligatory as writing the number it contributed. This distinction will become the founding principle of the pipeline in the fifth lesson. #### Two Separate Causes of Zero The most instructive row of the table is the 40 m³ threshold: the contribution is **exactly zero**, and not even the training number has moved by a hair. The cause shows in the last row — there are only 2 rows above 40 m³ in the set. A near-constant column cannot produce two sides of at least two rows each in any split, so the tree never selects it. The column was added, it was measured, and the tree the model sees is exactly the same. The 25 m³ threshold is a different kind of zero and in fact is not even zero, it is negative: a loss of 0.0159, four subscribers. This column separates 243 rows, so it is not constant. But the information it carries is already inside `mean_consumption`, and the tree finds its own threshold by **searching** the data. The hand-set threshold says nothing new; in return it competes as a split candidate at every node, and gets a split that looks slightly better on the training set selected, producing a worse tree on the held-out set. The largest loss comes from the interaction feature: 0.0238, six subscribers. The product of `household` and `satisfaction` squeezes the two columns into one number, and its relationship with the label is weaker than that of the two columns themselves. When the tree selects the product it makes the separation coarser. This is the typical behavior of interactions that do not come from domain knowledge and are generated on a "might work" basis. Three separate causes of zero-or-below must be told apart: the column **cannot be split at all** (40 m³), its information is **already there** (25 m³), or it carries the information **by distorting it** (the product). If all three are summarized as "no contribution", what is lost in the table is which fix will work. #### Contributions Do Not Add Up The contributions measured one by one gave three positive values: 0.0119, 0.0040, 0.0040. Their sum is 0.0199. One needs to look at what happens when the columns are added cumulatively. ```python cumulative = list(BASE_FIELDS) print(f"{'added':<22} {'columns':>7} {'train':>9} {'held_out':>9} {'contrib':>8}") for a in NEW: cumulative.append(a) e, s = measure(cumulative) print(f"{a:<22} {len(cumulative):>7} {e:>9.4f} {s:>9.4f} {s - BEFORE[1]:>+8.4f}") POSITIVE = ["per_person", "last_first_diff", "region_above"] e, s = measure(BASE_FIELDS + POSITIVE[:2]) print(f"\nonly two contributing columns {e:.4f} {s:.4f} {s - BEFORE[1]:+.4f}") e, s = measure(BASE_FIELDS + POSITIVE) print(f"only three contributing columns {e:.4f} {s:.4f} {s - BEFORE[1]:+.4f}") ``` ``` added columns train held_out contrib per_person 12 0.8519 0.7897 +0.0119 last_first_diff 13 0.8598 0.7897 +0.0119 region_above 14 0.8611 0.7778 +0.0000 threshold_25 15 0.8664 0.7698 -0.0079 household_satisfaction 16 0.8677 0.7698 -0.0079 threshold_40 17 0.8677 0.7698 -0.0079 only two contributing columns 0.8598 0.7897 +0.0119 only three contributing columns 0.8611 0.7778 +0.0000 ``` If the two columns were independent of each other, their contributions would add up. They do not. Two columns that add 0.0119 and 0.0040 one by one add 0.0119 together; when the third is added the total contribution falls to **0.0000**. Three features that measured positive one by one return, together, to the line before the decision. The cause is that the tree is greedy: at each node it selects the split that looks best at that moment, and the selection is not undone. A new column can offer a better-looking split at the root node and send the rest of the tree down an entirely different path. The contribution is not the column's own information but **the effect of the column on the model's search path**, and this effect is not an additive quantity. The two numbers must be read separately. The training number rises in one direction: from 0.8466 to 0.8677, each of the six columns pulls it a bit higher. The held-out number, on the other hand, peaks at twelve columns and falls, and at seventeen columns it stays below the line before the decision. The Overfitting and Underfitting lesson of the Introduction to Machine Learning course produced this divergence by increasing capacity; here the model is fixed and the only thing growing is the **number of columns**. Adding features is also increasing capacity. The rule of the course takes concrete form in this table. Six columns were generated, all six with a reasonable justification, and when measured on the held-out set only one of them remained. Had it not been measured, all six would have stayed in the set and the model would have worked with seventeen columns at 0.7698, 0.0080 below the line before the decision — and moreover the table would have looked better because the training number rose to 0.8677. #### Summary - The ratio feature carries the held-out number from 0.7778 to 0.7897 (0.0119, three subscribers); the difference feature and the region indicator learned from the training set add 0.0040, that is, one subscriber each. - Those with zero and negative contribution are written as well: the 40 m³ threshold is exactly 0.0000, the 25 m³ threshold loses 0.0159, the product of two columns loses 0.0238. - The cause of the zero is separated: the column cannot be split because it is near-constant, its information is already in another column, or it carries the information by distorting it. - Contributions do not add up: three columns that add 0.0119, 0.0040 and 0.0040 one by one add 0.0000 together, because the contribution is not the column's information but its effect on the model's search path. - While the number of columns goes from 11 to 17, the training number rises in one direction from 0.8466 to 0.8677 and the held-out number peaks at 0.7897 and falls to 0.7698; adding features is also increasing capacity. #### Next Step The six features of this lesson were derived from numerical columns by arithmetic. In the subscriber table, however, there are two more non-numerical fields: the date the contract started and the short note left by the field team. Neither can be given to a model as it is; multiple numbers can be derived from each. How many numbers come out of a date, how many numbers come out of a short text, and what the **total** contribution of these numbers is — the next lesson measures it. ### Time and Text Features Source: https://academia.sh/en/courses/feature-engineering/time-and-text-features Extracting five numbers each from a date and from a short text and counting their total contributions on the held-out set: while the line before the decision is 0.7897, the five numbers from the date lose 0.0198, the five numbers from the note add 0.0317, and the ten numbers together add 0.0198. The whole contribution comes from a single indicator; the seal trace indicator alone adds 0.0357 while the text length loses 0.0357 and the word count loses 0.0516. How many numbers a field yields does not determine the contribution; the relation of the field to the label does. The previous lesson generated six features, measured all six one by one, and left a single column behind: consumption per person, the ratio feature that carried the line before the decision from 0.7778 to 0.7897. The other five were removed from the set. From this lesson on, the line before the decision is the line of **twelve columns**. The subscriber table has two more fields that have not been given to the model, and neither is a number: the **date** on which the contract started and the **short note** the field crew left during the reading. A decision tree cannot split either of them as they are; a date is an ordered value, but its ordering does not carry the seasonality of the year, and a text is not ordered at all. Several numbers can be extracted from each of them, and this is exactly the measure of this lesson: **how many numbers a field yields and what the total contribution of those numbers is.** - **FS8.** The line before the decision is twelve columns: the eleven columns of the Introduction to Machine Learning course and `per_person`. The model does not change: a decision tree of depth 6. The baseline is 0.7579. - **FS9.** The contract date and the field note are **fictional**; they are generated from the same seed (20260218) depending on the subscriber id and added to the table in this lesson. - **FS10.** The field note is written **during the reading period**; the suspicion label becomes final after the period closes. The content of the note is at hand at prediction time. - **FS11.** The seal trace record enters the note with probability 0.42 for suspicious subscribers and 0.10 for the others; the access record enters with probability 0.30 if the period count is below three and 0.08 otherwise. - **FS12.** The extra note enters with probability 0.25 and is one of three equally likely topics; the vehicle record thus falls into the note with a probability of about 0.08 and is **independent** of the label; it is a trap indicator. - **FS13.** String operations were built in the data frames topic of the Data Analysis with Python course; the string routine is not repeated here, only the number extracted from the text is measured. The bag of words and the embedding space are not the subject of this course. - **FS14.** Contribution is the difference of the twelve columns' number on the held-out set from the line before the decision; one row is 0.0040. #### Five Numbers from a Date, Five from a Note The setup produces the same set as the previous lessons; the only difference is that the `per_person` column, whose persistence has been measured, is added from the start. ```python # COMMON — MODEL: the FICTIONAL subscriber table of the Introduction to Machine Learning course is rebuilt with the same seed; # the model is a decision tree of depth 6. The only new column left from the previous lesson is per_person. import math SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10), ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05), ("central", 0.18, 23, 0.05)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 SUBSCRIBERS, DATA = [], [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() >= 0.046: SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household, "satisfaction": rating}) for k in SUBSCRIBERS: r, v = rng(SEED + 7000 + k["id"]), [] for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): v.append(0.0 if r() < 0.038 else math.floor( k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - n * 0.05) * 100 + 0.5) / 100) r() if not v: continue mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"]) z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3] + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30) x = {"id": k["id"], "household": k["household"], "satisfaction": k["satisfaction"], "period": len(v), "mean_consumption": round(mean, 2), "volatility": round(vol, 2), "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35), "region_name": k["region"][0]} for b in REGION: x["area_" + b[0]] = int(k["region"][0] == b[0]) DATA.append(x) BASE = ["mean_consumption", "volatility", "household", "satisfaction", "period", "zero_readings", "area_north", "area_south", "area_east", "area_west", "area_central"] BASE_FIELDS = BASE + ["per_person"] for x in DATA: x["per_person"] = round(x["mean_consumption"] / x["household"], 3) def shuffle(data, seed): r, s = rng(seed), list(range(len(data))) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return [data[i] for i in s] K = shuffle(DATA, SEED + 90000) TRAIN, VAL, TEST = K[:756], K[756:1008], K[1008:] def gini(rows): p = sum(x["suspicious"] for x in rows) / len(rows) if rows else 0.0 return 2 * p * (1 - p) def tree(rows, depth, fields, min_size=2): p, best = sum(x["suspicious"] for x in rows) / len(rows), None if depth and len(rows) >= 2 * min_size and 0.0 < p < 1.0: for f in fields: values = sorted({x[f] for x in rows}) for v in (values[1:] if len(values) < 10 else [values[int(i * len(values) / 10)] for i in range(1, 10)]): left = [x for x in rows if x[f] < v] right = [x for x in rows if x[f] >= v] if min(len(left), len(right)) < min_size: continue k = gini(rows) - (len(left) * gini(left) + len(right) * gini(right)) / len(rows) if best is None or k > best[0]: best = (k, f, v, left, right) if best is None or best[0] <= 1e-9: return {"prediction": int(p > 0.5)} return {"field": best[1], "threshold": best[2], "left": tree(best[3], depth - 1, fields, min_size), "right": tree(best[4], depth - 1, fields, min_size)} def predict(node, x): while "prediction" not in node: node = node["left"] if x[node["field"]] < node["threshold"] else node["right"] return node["prediction"] def accuracy(node, rows): return sum(predict(node, x) == x["suspicious"] for x in rows) / len(rows) def baseline(train, rows): c = int(sum(x["suspicious"] for x in train) * 2 > len(train)) return sum(x["suspicious"] == c for x in rows) / len(rows) def measure(fields, depth=6): m = tree(TRAIN, depth, fields) return accuracy(m, TRAIN), accuracy(m, TEST) ``` The five numbers extracted from the date correspond to two different questions. The month, the quarter, the day of the week and the day of the year are **seasonal** indicators: each is a number that cycles within itself, and their purpose is to catch a seasonal pattern. The month and the quarter give the same value on the same calendar date every year; the day of the week shifts from year to year, and the day of the year moves by one day in a leap year. The contract age, on the other hand, is a **linear** number and grows every day. The five numbers extracted from the note also fall into two groups: two of them measure the **form** of the text (character count, word count), and three show the presence of a particular **keyword**. ```python import datetime START = datetime.date(2024, 1, 1) CORE = ["meter access completed", "reading okay", "subscriber was out", "dirty cover", "readings repeated", "valve was inspected"] EXTRA = ["car parked nearby", "doorbell not working", "garden gates open"] for x in DATA: r = rng(SEED + 61000 + x["id"]) x["contract_day"] = int(r() * 1460) g = START + datetime.timedelta(days=x["contract_day"]) x["month"], x["quarter"] = g.month, (g.month - 1) // 3 + 1 x["weekday"], x["day_of_year"] = g.weekday(), g.timetuple().tm_yday x["contract_age"] = 1460 - x["contract_day"] p = [CORE[int(r() * len(CORE))]] if r() < (0.42 if x["suspicious"] else 0.10): p.append("seal was torn") if r() < (0.30 if x["period"] < 3 else 0.08): p.append("access unavailable") if r() < 0.25: p.append(EXTRA[int(r() * len(EXTRA))]) n = ", ".join(p) x["field_note"], x["note_length"], x["note_words"] = n, len(n), len(n.split()) x["kw_seal"] = int("seal" in n) x["kw_access"] = int("access unavailable" in n) x["kw_car"] = int("car" in n) TIME = ["month", "quarter", "weekday", "day_of_year", "contract_age"] TEXT = ["note_length", "note_words", "kw_seal", "kw_access", "kw_car"] for i in (4, 9, 10): print(f"note example: {DATA[i]['field_note']}") print(f"seal trace recorded in {sum(x['kw_seal'] for x in DATA)} rows; in " f"{sum(x['kw_seal'] for x in DATA if x['suspicious']) / sum(x['suspicious'] for x in DATA):.3f}" f" of the suspicious, in " f"{sum(x['kw_seal'] for x in DATA if not x['suspicious']) / sum(1 - x['suspicious'] for x in DATA):.3f}" f" of the others\n") BEFORE = measure(BASE_FIELDS) print(f"{'block':<16} {'columns':>7} {'train':>9} {'held_out':>9} {'contrib':>8}") print(f"{'-- none --':<16} {0:>7} {BEFORE[0]:>9.4f} {BEFORE[1]:>9.4f}") for name, fs in (("from date", TIME), ("from text", TEXT), ("both", TIME + TEXT)): e, s = measure(BASE_FIELDS + fs) print(f"{name:<16} {len(fs):>7} {e:>9.4f} {s:>9.4f} {s - BEFORE[1]:>+8.4f}") ``` ``` note example: readings repeated note example: subscriber was out, access unavailable, car parked nearby note example: dirty cover, seal was torn seal trace recorded in 219 rows; in 0.453 of the suspicious, in 0.096 of the others block columns train held_out contrib -- none -- 0 0.8519 0.7897 from date 5 0.8611 0.7698 -0.0198 from text 5 0.8942 0.8214 +0.0317 both 10 0.8955 0.8095 +0.0198 ``` Side by side, the three numbers show that there is no link between the number of fields and the contribution. Five numbers were extracted from each of two fields; one earned 0.0317, the other lost 0.0198. When ten numbers are given together, the total contribution falls to 0.0198 — that is, the date field takes back three of the eight subscribers the text field earned. The training number, on the other hand, rises in all three setups: from 0.8519 to 0.8955. That the training number rises as the column count grows is a requirement of the definition, not a measurement. That the date block comes out negative is not surprising, because in this fiction no link has been built between the contract date and the suspicion. The cause of the loss is not the absence of a link but **five columns competing as split candidates**: the day-of-year column has 366 distinct values, and among that many threshold candidates it is easy to find one that looks good on the training set. A noise column with many values is more dangerous than a noise column with few. #### Where the Contribution Comes From When the blocks are opened one by one, it becomes visible which column the total numbers come from. ```python print(f"{'single feature':<16} {'source':<8} {'train':>9} {'held_out':>9} {'contrib':>8}") for f in TIME + TEXT: e, s = measure(BASE_FIELDS + [f]) print(f"{f:<16} {'date' if f in TIME else 'text':<8} {e:>9.4f} {s:>9.4f} " f"{s - BEFORE[1]:>+8.4f}") for name, fs in (("only three keywords", ["kw_seal", "kw_access", "kw_car"]), ("only two counters", ["note_length", "note_words"]), ("only the seal", ["kw_seal"])): e, s = measure(BASE_FIELDS + fs) print(f"\n{name:<20} {e:.4f} {s:.4f} {s - BEFORE[1]:+.4f}", end="") print() ``` ``` single feature source train held_out contrib month date 0.8519 0.7897 +0.0000 quarter date 0.8519 0.7897 +0.0000 weekday date 0.8532 0.7778 -0.0119 day_of_year date 0.8519 0.7897 +0.0000 contract_age date 0.8585 0.7817 -0.0079 note_length text 0.8823 0.7540 -0.0357 note_words text 0.8783 0.7381 -0.0516 kw_seal text 0.8876 0.8254 +0.0357 kw_access text 0.8519 0.7857 -0.0040 kw_car text 0.8519 0.7897 +0.0000 only three keywords 0.8876 0.8254 +0.0357 only two counters 0.8942 0.7381 -0.0516 only the seal 0.8876 0.8254 +0.0357 ``` Nine of the ten features give zero or negative contribution. The whole gain comes from the `kw_seal` column: 0.0357 on its own, that is, nine subscribers. The three keyword indicators together also give 0.0357, and the seal indicator alone also gives 0.0357 — the other two keywords add nothing. The source of the number is clear: the seal trace is recorded in 0.453 of the suspicious and 0.096 of the others, so this indicator is related to the label. The vehicle record, on the other hand, was generated independently of the label and its contribution came out exactly zero; the trap indicator behaved as a trap. The harshest rows are the **form** measures of the text. The character count loses 0.0357 and the word count loses 0.0516; the two together lose 0.0516 and raise the training number to 0.8942. These two columns are in fact a blurred copy of the keyword indicators: if the note gets longer, there is an extra record in it, but which record is not known. The model uses the length to predict the seal record indirectly on the training set, and on the held-out set this indirect path does not hold. The easy way of extracting a number from a text and the way that works are not the same way. The number of values extracted from a field does not measure the contribution of that field. Ten numbers were extracted, the entire contribution came from one of them, and the other nine took 0.0159 off in total. Increasing the number of columns per field does not give the model more information, it gives it a larger **search space**. #### The Time of the Note The seal indicator has statistically the same form as the audit column in the Data Leakage lesson of the Introduction to Machine Learning course: both are indicator columns that appear more often in suspicious subscribers. The only difference between them is **time**. The audit assignment was written after the label became final, and that is why it was leakage; the field note is written during the reading period, before the label becomes final, and is on record at the moment the prediction is to be given. This distinction cannot be made by looking at the table. Inflation and contribution carry the same sign on the held-out set and appear in the same column. Only the **writing order** of the source system gives the distinction: after which event is this value recorded. The question itself was built in the Introduction to Machine Learning course; this lesson makes it part of a feature generation decision. The question is asked again for every indicator extracted from text, because the same text field can carry both sentences written during the period and sentences written after it. Methods that turn a text entirely into numbers, such as the bag of words and the embedding space, are not the subject of this course; they are covered in the Natural Language Processing and Computer Vision course. What is measured here is a narrower and more common decision: extracting **a few indicators** from a free-text field when one is at hand. The measurement shows that this decision, too, can come out without contribution. #### Summary - Five features were extracted from a date and five from a short note; the date block lost 0.0198, the text block gave a contribution of 0.0317, and the ten numbers together gave 0.0198. - The whole contribution comes from a single indicator: the seal trace alone adds 0.0357, the three keyword indicators together give the same number, and the vehicle indicator, generated independently of the label, gives exactly zero. - The form measures of the text are the columns that lose the most: the character count 0.0357, the word count 0.0516; both are a blurred copy of the keyword indicator and raise the training number to 0.8942. - A noise column with many values is more dangerous than one with few; the day-of-year column offers 366 threshold candidates, and one that looks good on the training set is easy to find. - The seal indicator and the audit column of the Introduction to Machine Learning course have the same form; only the time of the record separates them, and this question is asked again for every text feature. #### Next Step In two lessons the column count rose from eleven to twenty-two, and each time the measurement said the same thing: most columns either add nothing or lose. Up to here the columns were eliminated **one by one by hand**. The elimination itself can be tied to a routine — columns can be filtered by their relation to the label, tried by training the model again and again, or the model's own feature importance can be read. How many columns the three methods select on the same set, how much the held-out number inflates when the selection is made from **the whole set**, and what happens when it is made **from training only** — the next lesson measures this. ### Feature Selection Source: https://academia.sh/en/courses/feature-engineering/feature-selection Counting how many features the filter, wrapper and embedded methods select from the same 25 candidate columns, and the inflation produced by the selection itself: while the 25 columns without selection give 0.8135 on the test set, the filter gives 0.8056 with 10 columns, the wrapper 0.7579 with 4 columns and the embedded method 0.8254 with 14 columns. The wrapper reports 0.8413 on the validation set it reads its score from and drops to 0.7579 on the test set; the inflation is 0.0834. When the selection from 1,200 noise columns is made from the whole set, the held-out number inflates by 0.0198, 0.0040 and 0.0714, with the same sign in all three measurements. In two lessons the column count rose from eleven to twenty-two, and each time the elimination was done by hand: a column was added, the number on the held-out set was read, and if it had no contribution it was removed. This lesson takes the eliminated ones back into the pool as well; the procedure works on twenty-five candidates, it does not work on two hundred. The elimination decision itself can be tied to a procedure. There are three families. The **filter method** compares each column alone with the label and eliminates by a score independent of the model. The **wrapper method** trains the model again and again and grows the column set according to a performance number. The **embedded method** reads the model's feature importance. This lesson runs the three on the same pool and asks a second question: how much does the selection procedure itself inflate the number on the held-out set. - **FS15.** The candidate pool is 25 columns: the eleven columns of the Introduction to Machine Learning course and the fourteen columns generated by the previous two lessons. The line before the decision is the setup **without selection**, that is, all twenty-five columns. - **FS16.** The model does not change: a decision tree of depth 6. The baseline is 0.7579. - **FS17.** The filter score is the largest purity gain the column provides in a single split, and the threshold is 0.004; both sides must contain at least 20 rows. The threshold is chosen **beforehand**. - **FS18.** The wrapper does forward selection: in each round it adds the column that gains the most, stops when no gain is left, and runs at most eight rounds. It reads its score from the **validation set**. - **FS19.** The embedded method builds a tree of depth 8, weights the gain of each split by the number of rows split and writes it to the column, and selects the columns that carry more than one percent of the total importance. - **FS20.** All three methods make their decision **from the training and validation sets only**; the test set takes part in no selection. The multiple comparison calculation of the Statistical Analysis and Exploratory Data Analysis course is not repeated. - **FS21.** The inflation measurement uses a noise pool of 1,200 columns. The columns are generated from the same seed depending on the subscriber id and have no relation to the label; their contribution is a known zero. #### Three Methods, Three Feature Sets ```python # COMMON — MODEL: the FICTIONAL subscriber table of the Introduction to Machine Learning course is rebuilt with the same seed; # the model is a decision tree of depth 6. The only new column left from the previous lesson is per_person. import math SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10), ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05), ("central", 0.18, 23, 0.05)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 SUBSCRIBERS, DATA = [], [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() >= 0.046: SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household, "satisfaction": rating}) for k in SUBSCRIBERS: r, v = rng(SEED + 7000 + k["id"]), [] for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): v.append(0.0 if r() < 0.038 else math.floor( k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - n * 0.05) * 100 + 0.5) / 100) r() if not v: continue mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"]) z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3] + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30) x = {"id": k["id"], "household": k["household"], "satisfaction": k["satisfaction"], "period": len(v), "mean_consumption": round(mean, 2), "volatility": round(vol, 2), "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35), "region_name": k["region"][0]} for b in REGION: x["area_" + b[0]] = int(k["region"][0] == b[0]) DATA.append(x) BASE = ["mean_consumption", "volatility", "household", "satisfaction", "period", "zero_readings", "area_north", "area_south", "area_east", "area_west", "area_central"] BASE_FIELDS = BASE + ["per_person"] for x in DATA: x["per_person"] = round(x["mean_consumption"] / x["household"], 3) def shuffle(data, seed): r, s = rng(seed), list(range(len(data))) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return [data[i] for i in s] K = shuffle(DATA, SEED + 90000) TRAIN, VAL, TEST = K[:756], K[756:1008], K[1008:] def gini(rows): p = sum(x["suspicious"] for x in rows) / len(rows) if rows else 0.0 return 2 * p * (1 - p) def tree(rows, depth, fields, min_size=2): p, best = sum(x["suspicious"] for x in rows) / len(rows), None if depth and len(rows) >= 2 * min_size and 0.0 < p < 1.0: for f in fields: values = sorted({x[f] for x in rows}) for v in (values[1:] if len(values) < 10 else [values[int(i * len(values) / 10)] for i in range(1, 10)]): left = [x for x in rows if x[f] < v] right = [x for x in rows if x[f] >= v] if min(len(left), len(right)) < min_size: continue k = gini(rows) - (len(left) * gini(left) + len(right) * gini(right)) / len(rows) if best is None or k > best[0]: best = (k, f, v, left, right) if best is None or best[0] <= 1e-9: return {"prediction": int(p > 0.5)} return {"field": best[1], "threshold": best[2], "left": tree(best[3], depth - 1, fields, min_size), "right": tree(best[4], depth - 1, fields, min_size)} def predict(node, x): while "prediction" not in node: node = node["left"] if x[node["field"]] < node["threshold"] else node["right"] return node["prediction"] def accuracy(node, rows): return sum(predict(node, x) == x["suspicious"] for x in rows) / len(rows) def baseline(train, rows): c = int(sum(x["suspicious"] for x in train) * 2 > len(train)) return sum(x["suspicious"] == c for x in rows) / len(rows) def measure(fields, depth=6): m = tree(TRAIN, depth, fields) return accuracy(m, TRAIN), accuracy(m, TEST) ``` The candidate pool includes the fourteen columns generated by the previous two lessons — including those whose contribution was measured and came out negative, because the job of the selection procedure is exactly to eliminate them. ```python import datetime START = datetime.date(2024, 1, 1) CORE = ["meter access completed", "reading okay", "subscriber was out", "dirty cover", "readings repeated", "valve was inspected"] EXTRA = ["car parked nearby", "doorbell not working", "garden gates open"] MEDIAN = {} for b in REGION: v = sorted(x["mean_consumption"] for x in TRAIN if x["region_name"] == b[0]) MEDIAN[b[0]] = v[len(v) // 2] for x in DATA: x["household_satisfaction"] = x["household"] * x["satisfaction"] x["threshold_25"] = int(x["mean_consumption"] > 25) x["region_above"] = int(x["mean_consumption"] > MEDIAN[x["region_name"]]) r = rng(SEED + 61000 + x["id"]) g = START + datetime.timedelta(days=int(r() * 1460)) x["month"], x["quarter"] = g.month, (g.month - 1) // 3 + 1 x["weekday"], x["day_of_year"] = g.weekday(), g.timetuple().tm_yday x["contract_age"] = 1460 - (g - START).days p = [CORE[int(r() * len(CORE))]] if r() < (0.42 if x["suspicious"] else 0.10): p.append("seal was torn") if r() < (0.30 if x["period"] < 3 else 0.08): p.append("access unavailable") if r() < 0.25: p.append(EXTRA[int(r() * len(EXTRA))]) n = ", ".join(p) x["note_length"], x["note_words"] = len(n), len(n.split()) x["kw_seal"] = int("seal" in n) x["kw_access"] = int("access unavailable" in n) x["kw_car"] = int("car" in n) GENERATED = ["per_person", "region_above", "threshold_25", "household_satisfaction", "month", "quarter", "weekday", "day_of_year", "contract_age", "note_length", "note_words", "kw_seal", "kw_access", "kw_car"] CANDIDATES = BASE + GENERATED MANUAL = BASE_FIELDS + ["kw_seal"] def score(source, f): values, g0, best = sorted({x[f] for x in source}), gini(source), 0.0 for v in (values[1:] if len(values) < 10 else [values[int(i * len(values) / 10)] for i in range(1, 10)]): left = [x for x in source if x[f] < v] right = [x for x in source if x[f] >= v] if min(len(left), len(right)) >= 20: best = max(best, g0 - (len(left) * gini(left) + len(right) * gini(right)) / len(source)) return best def filter_select(pool, source, threshold=0.004): return [f for f in pool if score(source, f) >= threshold] def wrapper_select(pool, train, target, rounds=8): chosen, best_score = [], 0.0 for _ in range(rounds): cand = max((accuracy(tree(train, 6, chosen + [f]), target), f) for f in pool if f not in chosen) if cand[0] <= best_score: break best_score = cand[0] chosen.append(cand[1]) return chosen, best_score def embedded_select(pool, train, depth=8, share=0.01): importance = dict.fromkeys(pool, 0.0) def walk(rows, d): p = sum(x["suspicious"] for x in rows) / len(rows) if not d or len(rows) < 4 or not 0.0 < p < 1.0: return best = None for f in pool: vals = sorted({x[f] for x in rows}) for v in (vals[1:] if len(vals) < 10 else [vals[int(i * len(vals) / 10)] for i in range(1, 10)]): left = [x for x in rows if x[f] < v] right = [x for x in rows if x[f] >= v] if min(len(left), len(right)) < 2: continue k = gini(rows) - (len(left) * gini(left) + len(right) * gini(right)) / len(rows) if best is None or k > best[0]: best = (k, f, left, right) if best is None or best[0] <= 1e-9: return importance[best[1]] += best[0] * len(rows) walk(best[2], d - 1) walk(best[3], d - 1) walk(train, depth) t = sum(importance.values()) return [f for f in pool if importance[f] / t >= share], importance, t S1 = filter_select(CANDIDATES, TRAIN) S2, REPORTED = wrapper_select(CANDIDATES, TRAIN, VAL) S3, IMPORTANCE, TOTAL = embedded_select(CANDIDATES, TRAIN) BEFORE = measure(CANDIDATES) print(f"{'method':<17} {'columns':>7} {'train':>9} {'held_out':>9} {'contrib':>8}") print(f"{'no selection':<17} {len(CANDIDATES):>7} {BEFORE[0]:>9.4f} {BEFORE[1]:>9.4f}") for name, s in (("filter", S1), ("wrapper", S2), ("embedded", S3), ("manually measured", MANUAL)): e, t = measure(s) print(f"{name:<17} {len(s):>7} {e:>9.4f} {t:>9.4f} {t - BEFORE[1]:>+8.4f}") print(f"\nthe number the wrapper reported on the validation set {REPORTED:.4f}, " f"on the test set {measure(S2)[1]:.4f}") print("embedded importance order:", ", ".join( f"{f} {IMPORTANCE[f] / TOTAL:.3f}" for _, f in sorted((-IMPORTANCE[f], f) for f in CANDIDATES)[:5])) ``` ``` method columns train held_out contrib no selection 25 0.8968 0.8135 filter 10 0.8823 0.8056 -0.0079 wrapper 4 0.8241 0.7579 -0.0556 embedded 14 0.8995 0.8254 +0.0119 manually measured 13 0.8876 0.8254 +0.0119 the number the wrapper reported on the validation set 0.8413, on the test set 0.7579 embedded importance order: volatility 0.216, kw_seal 0.204, mean_consumption 0.145, contract_age 0.076, note_length 0.056 ``` The three methods select three sets of different sizes: 10, 4 and 14 columns. Their contributions also carry three different signs. The embedded method adds 0.0119 and reaches 0.8254 with fourteen columns — exactly the number of the thirteen-column set that the previous two lessons found by measuring by hand. The column sets are not exactly the same, but the number on the held-out set is the same. The ten columns of the filter method lose 0.0079. The reason lies in the definition of the method: it looks at each column **alone**, so it cannot see columns that carry information only together, and it keeps columns that are copies of each other together. Both `note_length` and `note_words` are among the ten columns it selected; the two are blurred copies of the same information. The filter is cheap — twenty-five scores are computed for twenty-five columns — and this is the price of its cheapness. #### The Selection's Own Number The wrapper row carries two separate numbers, and the real subject of the course in this lesson lies between those two. The method selected four columns and stopped; while making the selection it looked at the accuracy on the **validation set** at every step, and in the last step it saw 0.8413. The model built with the same four columns gives 0.7579 on the test set, that is, it stands exactly on the baseline. The difference is 0.0834. This difference is not a mistake; it is the consequence of the definition of the procedure. Forward selection builds twenty-five models in the first round and twenty-four in the second; taking the highest on the validation set among forty-nine models is taking the highest of the random fluctuation on that set. The moment the validation set takes part in the selection, it ceases to be held out, and the number read on it is no longer a measurement, it is the selection's own target. The result is the second rule of this course read for selection: **the selected feature set is also a trained object.** Like the scale, like the category mapping, it is learned from data, and which data it was learned from has to be written down. The importance order of the embedded method stands in the last line: the three most important columns are the volatility, the seal indicator and the mean consumption, carrying 0.565 of the total importance. The contract age in fourth place, on the other hand, is a column with no relation to the label at all, and it has still taken 0.076 of the importance. Feature importance is a ranking, not evidence. #### If the Selection Is Made from the Whole Set To measure cleanly what the selection inflates, a pool whose contribution is a **known zero** is needed. The code below generates 1,200 noise columns; none of them has a relation to the label. The filter selects from this pool the few columns that separate the most, once looking at the training set only and once looking at the whole set. ```python NOISE = ["noise_" + str(j) for j in range(1200)] for x in DATA: r = rng(SEED + 41000 + x["id"]) for f in NOISE: x[f] = round(r(), 4) def separation(source, f): v1 = [x[f] for x in source if x["suspicious"]] v0 = [x[f] for x in source if not x["suspicious"]] return abs(sum(v1) / len(v1) - sum(v0) / len(v0)) def top_k(source, k): return [f for _, f in sorted((-separation(source, f), f) for f in NOISE)[:k]] print(f"{'selected':>8} {'source':<12} {'train':>9} {'held_out':>9} {'inflation':>9}") for k in (2, 4, 8): e1, s1 = measure(top_k(TRAIN, k)) e2, s2 = measure(top_k(K, k)) print(f"{k:>8} {'train':<12} {e1:>9.4f} {s1:>9.4f}") print(f"{k:>8} {'whole set':<12} {e2:>9.4f} {s2:>9.4f} {s2 - s1:>+9.4f}") print(f"\nbaseline {baseline(TRAIN, TEST):.4f}; the pool has {len(NOISE)} noise columns") ``` ``` selected source train held_out inflation 2 train 0.8320 0.7262 2 whole set 0.8069 0.7460 +0.0198 4 train 0.8307 0.7421 4 whole set 0.8241 0.7460 +0.0040 8 train 0.8704 0.6667 8 whole set 0.8479 0.7381 +0.0714 baseline 0.7579; the pool has 1200 noise columns ``` All three values of the inflation column carry the same sign: +0.0198, +0.0040, +0.0714. In the Data Leakage lesson of the Introduction to Machine Learning course, the inflation of the four leakage types was also of the same sign. The channel here is the same: when the selection looks at the whole set, the labels of the 252 rows of the test set contribute to which columns will be selected, the selected columns come out more consistent on those rows, and the measurement rises. The difference grows with the column count; with eight columns it is 0.0714, that is, eighteen subscribers. One detail makes this table dangerous: none of the inflated numbers **passes the baseline**. No model comes out of noise; the leakage only pulls the number up. In a real set the same channel is added to the number of a model that already passes the baseline, and it looks like an ordinary improvement in the table. Making the selection from the training set only is the one arrangement that makes the number come out correct at the price of coming out small. #### Summary - On the same 25 candidate columns the filter selected 10, the wrapper 4 and the embedded method 14 columns; while the setup without selection gives 0.8135, they gave 0.8056, 0.7579 and 0.8254 respectively. - With a contribution of 0.0119 the embedded method reached 0.8254, the number the previous two lessons found by measuring by hand, with a single procedure; the filter lost 0.0079 because it kept two columns that are copies of each other. - The wrapper reported 0.8413 on the validation set it reads its score from and gave 0.7579 on the test set; the inflation is 0.0834 and its cause is that it picked the highest on that set among forty-nine models. - The selected feature set is a trained object; unless it is written down which set it was learned from, the number read on it is not a measurement. - On the 1,200 noise columns with a known zero contribution, the inflation came out +0.0198, +0.0040 and +0.0714 when the selection was made from the whole set; all three have the same sign and none passes the baseline. #### Next Step Selection keeps and drops columns one at a time, and the names of the ones it keeps can be read. There is another way to reduce the column count — instead of dropping columns, turning them into a few new **components**. On this path every new column is a mixture of the old columns and the mixture has no name. At how many components the held-out number of the raw features is caught up with, and what is lost when the feature importance list loses its readability — the next lesson measures this. ### Dimensionality Reduction as a Feature Tool Source: https://academia.sh/en/courses/feature-engineering/feature-role-of-dimensionality-reduction Measuring what happens to the held-out number and to interpretability when a component-based representation replaces thirteen raw features: while the raw columns give 0.8254 on the test set, two components give 0.8175, three components 0.8056, five components 0.7778 and eight components 0.7460, and no number of components catches up with the raw representation. The component carrying the largest component share falls below the baseline on its own with 0.7381, while the fifth component with a share of 0.093 gives 0.7659. In the raw representation 0.35 of the splits are read from the mean consumption, while in the component representation the same share falls on a mixture with no name. The previous lesson measured one way of reducing the column count: keeping or dropping the columns one by one. The embedded method kept fourteen of the twenty-five candidates and gave 0.8254 on the held-out set; the thirteen-column set built by hand-measuring over two lessons gave the same number. Every one of the kept columns has a name, and what they mean is known. The second way of reducing the column count is **mixing** the columns instead of dropping them. Thirteen columns can be turned into a few new columns, each a weighted sum of the thirteen; these are called **components**, and the components are ordered starting from the directions that carry the most variability in the set. The Statistical Analysis and Exploratory Data Analysis course measured the component share there, and the theory is not repeated here. The only question this lesson asks is this: what do components do as a **feature** — at how many components is the held-out number of the raw features caught up with, and what is lost in return. - **FS22.** The raw representation is thirteen columns: the eleven columns of the Introduction to Machine Learning course, `per_person` and `kw_seal`. Its number on the held-out set is 0.8254 and it is the line before the decision of this lesson. - **FS23.** The scale and the component directions are learned **from the training set only** and applied to all three sets with the same numbers. If they were learned from the whole set, the measurement would inflate. - **FS24.** The components are written within the lesson: the covariance matrix of the standardized training columns is built, its largest direction is found by iteration, the direction found is subtracted from the matrix, and the operation is repeated thirteen times. No library is called. - **FS25.** The number of iterations is 200 for every component; this number was not chosen by looking at the result. If the remaining magnitude drops below $10^{-12}$, the iteration stops early. - **FS26.** The component share is the ratio of the variability the component carries to the total variability. - **FS27.** The components **do not see the label**. Their ranking is by variability, not by discriminating power. - **FS28.** The model does not change: a decision tree of depth 6. The baseline is 0.7579. - **FS29.** The interpretability measure is which column the splits of the tree are read from; each split is weighted by the number of rows it splits. #### Thirteen Columns, Thirteen Components ```python # COMMON — MODEL: the FICTIONAL subscriber table of the Introduction to Machine Learning course is rebuilt with the same seed; # the model is a decision tree of depth 6. The only new column left from the previous lesson is per_person. import math SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10), ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05), ("central", 0.18, 23, 0.05)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 SUBSCRIBERS, DATA = [], [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() >= 0.046: SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household, "satisfaction": rating}) for k in SUBSCRIBERS: r, v = rng(SEED + 7000 + k["id"]), [] for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): v.append(0.0 if r() < 0.038 else math.floor( k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - n * 0.05) * 100 + 0.5) / 100) r() if not v: continue mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"]) z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3] + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30) x = {"id": k["id"], "household": k["household"], "satisfaction": k["satisfaction"], "period": len(v), "mean_consumption": round(mean, 2), "volatility": round(vol, 2), "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35), "region_name": k["region"][0]} for b in REGION: x["area_" + b[0]] = int(k["region"][0] == b[0]) DATA.append(x) BASE = ["mean_consumption", "volatility", "household", "satisfaction", "period", "zero_readings", "area_north", "area_south", "area_east", "area_west", "area_central"] BASE_FIELDS = BASE + ["per_person"] for x in DATA: x["per_person"] = round(x["mean_consumption"] / x["household"], 3) def shuffle(data, seed): r, s = rng(seed), list(range(len(data))) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return [data[i] for i in s] K = shuffle(DATA, SEED + 90000) TRAIN, VAL, TEST = K[:756], K[756:1008], K[1008:] def gini(rows): p = sum(x["suspicious"] for x in rows) / len(rows) if rows else 0.0 return 2 * p * (1 - p) def tree(rows, depth, fields, min_size=2): p, best = sum(x["suspicious"] for x in rows) / len(rows), None if depth and len(rows) >= 2 * min_size and 0.0 < p < 1.0: for f in fields: values = sorted({x[f] for x in rows}) for v in (values[1:] if len(values) < 10 else [values[int(i * len(values) / 10)] for i in range(1, 10)]): left = [x for x in rows if x[f] < v] right = [x for x in rows if x[f] >= v] if min(len(left), len(right)) < min_size: continue k = gini(rows) - (len(left) * gini(left) + len(right) * gini(right)) / len(rows) if best is None or k > best[0]: best = (k, f, v, left, right) if best is None or best[0] <= 1e-9: return {"prediction": int(p > 0.5)} return {"field": best[1], "threshold": best[2], "left": tree(best[3], depth - 1, fields, min_size), "right": tree(best[4], depth - 1, fields, min_size)} def predict(node, x): while "prediction" not in node: node = node["left"] if x[node["field"]] < node["threshold"] else node["right"] return node["prediction"] def accuracy(node, rows): return sum(predict(node, x) == x["suspicious"] for x in rows) / len(rows) def baseline(train, rows): c = int(sum(x["suspicious"] for x in train) * 2 > len(train)) return sum(x["suspicious"] == c for x in rows) / len(rows) def measure(fields, depth=6): m = tree(TRAIN, depth, fields) return accuracy(m, TRAIN), accuracy(m, TEST) ``` Three steps are needed to find the components: bringing the columns to a common scale, building the covariance matrix, and finding the direction that carries the largest variability and subtracting it from the matrix. When the third step is repeated thirteen times, thirteen components come out. The scale and all the directions are read from the training set only. ```python for x in DATA: r = rng(SEED + 61000 + x["id"]) r(), r() x["kw_seal"] = int(r() < (0.42 if x["suspicious"] else 0.10)) RAW_FIELDS = BASE_FIELDS + ["kw_seal"] MEAN = {f: sum(x[f] for x in TRAIN) / len(TRAIN) for f in RAW_FIELDS} SPREAD = {f: max(1e-9, (sum((x[f] - MEAN[f]) ** 2 for x in TRAIN) / len(TRAIN)) ** 0.5) for f in RAW_FIELDS} for x in DATA: x["_z"] = [(x[f] - MEAN[f]) / SPREAD[f] for f in RAW_FIELDS] n = len(RAW_FIELDS) C = [[sum(x["_z"][i] * x["_z"][j] for x in TRAIN) / len(TRAIN) for j in range(n)] for i in range(n)] def multiply(m, v): return [sum(m[i][j] * v[j] for j in range(len(v))) for i in range(len(m))] COMPONENTS, SHARE = [], [] A = [row[:] for row in C] for _ in range(n): v = [1.0 / (i + 1) for i in range(n)] for _ in range(200): w = multiply(A, v) u = sum(q * q for q in w) ** 0.5 if u < 1e-12: break v = [q / u for q in w] lam = sum(v[i] * multiply(A, v)[i] for i in range(n)) COMPONENTS.append(v) SHARE.append(max(0.0, lam)) A = [[A[i][j] - lam * v[i] * v[j] for j in range(n)] for i in range(n)] TOTAL = sum(SHARE) for x in DATA: for i, v in enumerate(COMPONENTS): x["c" + str(i)] = round(sum(x["_z"][j] * v[j] for j in range(n)), 4) RAW_RESULT = measure(RAW_FIELDS) print(f"{'representation':<22} {'columns':>7} {'share':>7} {'train':>9} {'held_out':>9} {'diff':>8}") print(f"{'raw features':<22} {n:>7} {'':>7} {RAW_RESULT[0]:>9.4f} {RAW_RESULT[1]:>9.4f}") for k in (1, 2, 3, 5, 8, 11, 13): e, s = measure(["c" + str(i) for i in range(k)]) print(f"{'component ' + str(k):<22} {k:>7} {sum(SHARE[:k]) / TOTAL:>7.3f} {e:>9.4f} " f"{s:>9.4f} {s - RAW_RESULT[1]:>+8.4f}") print(f"\nbaseline {baseline(TRAIN, TEST):.4f}") ``` ``` representation columns share train held_out diff raw features 13 0.8876 0.8254 component 1 1 0.177 0.8003 0.7381 -0.0873 component 2 2 0.323 0.8651 0.8175 -0.0079 component 3 3 0.432 0.8677 0.8056 -0.0198 component 5 5 0.630 0.9074 0.7778 -0.0476 component 8 8 0.877 0.9074 0.7460 -0.0794 component 11 11 0.990 0.8995 0.7817 -0.0437 component 13 13 1.000 0.9048 0.7897 -0.0357 baseline 0.7579 ``` The answer to the first half of the question is in the table: **no number of components catches up with the raw features.** The closest row is two components; with 0.8175 it is 0.0079 below the raw representation, that is, two subscribers. Thirteen columns have been reduced to two columns and two subscribers have been lost. This is a defensible balance as a compression decision. The real finding of the table comes after this. As the number of components grows, the number does **not improve**: 0.8056 with three components, 0.7778 with five, 0.7460 with eight. The component share, on the other hand, grows steadily — from 0.323 to 0.877. The two columns do not go in the same direction. When all thirteen components are used there is no loss of information, the representation is a rotated version of the raw representation, and still the number stays at 0.7897; the difference is 0.0357. A decision tree makes its splits perpendicular to the axes, and building the same separation on a rotated axis takes more splits. The component representation is not independent of the model. #### Share Ranking Is Not Separation Ranking The components never see the label; their ranking looks only at which direction has more variability. Whether the label sits in those directions is a separate question and has to be measured. ```python def usage(node, rows): counts = {} def walk(node, rows): if "prediction" in node: return counts[node["field"]] = counts.get(node["field"], 0) + len(rows) walk(node["left"], [x for x in rows if x[node["field"]] < node["threshold"]]) walk(node["right"], [x for x in rows if x[node["field"]] >= node["threshold"]]) walk(node, rows) total = sum(counts.values()) return ", ".join(f"{f} {counts[f] / total:.2f}" for _, f in sorted((-counts[f], f) for f in counts)[:4]) print("most split in raw features:", usage(tree(TRAIN, 6, RAW_FIELDS), TRAIN)) print("in component features :", usage(tree(TRAIN, 6, ["c" + str(i) for i in range(13)]), TRAIN)) for i in (0, 1): y = sorted(((-abs(COMPONENTS[i][j]), RAW_FIELDS[j], COMPONENTS[i][j]) for j in range(n))) print(f"c{i} loadings (share {SHARE[i] / TOTAL:.3f}): " + ", ".join(f"{f} {w:+.2f}" for _, f, w in y[:4]) + f", ... total weight of the remaining {n - 4} columns " f"{sum(abs(w) for _, _, w in y[4:]):.2f}") print() print(f"{'single component':<16} {'share':>7} {'train':>9} {'held_out':>9}") for i in range(5): e, s = measure(["c" + str(i)]) print(f"{'c' + str(i):<16} {SHARE[i] / TOTAL:>7.3f} {e:>9.4f} {s:>9.4f}") ``` ``` most split in raw features: mean_consumption 0.35, volatility 0.19, kw_seal 0.17, per_person 0.14 in component features : c0 0.29, c1 0.25, c8 0.12, c2 0.11 c0 loadings (share 0.177): per_person +0.54, mean_consumption +0.53, area_west -0.34, area_east +0.30, ... total weight of the remaining 9 columns 1.16 c1 loadings (share 0.146): volatility +0.62, zero_readings +0.51, period +0.42, kw_seal +0.29, ... total weight of the remaining 9 columns 0.86 single component share train held_out c0 0.177 0.8003 0.7381 c1 0.146 0.8267 0.7421 c2 0.109 0.8188 0.7579 c3 0.105 0.7937 0.7579 c4 0.093 0.8148 0.7659 ``` The last table shows with numbers that the share ranking is not the separation ranking. The component with the largest share, `c0`, gives 0.7381 on its own — **below** the baseline by 0.0198. `c4`, with about half of its share, gives 0.7659 and stays above the baseline. "The direction that carries the most variability" and "the direction that separates the most" are two different things, and since the components are ordered only by the first, taking the first k components is an arbitrary cut as far as separation is concerned. It is also understood from here why two components together give 0.8175: `c0` carries the consumption magnitude, while `c1` carries the volatility and the zero readings. Two separate causes of the label have fallen onto two separate components, and the model can combine the two. #### What Is Lost from Interpretability The second half of the loss is not in the number but in the names. In the raw representation, 0.35 of the splits are read from `mean_consumption`, 0.19 from `volatility` and 0.17 from the seal indicator. These three rows can be given to a field crew: high consumption, volatile readings and a seal record. In the component representation the same list comes out as `c0` 0.29, `c1` 0.25, `c8` 0.12 and says nothing. What a component is can be understood only by reading the loadings, and the loadings do not point to a single column. `c0` carries mainly consumption per person and mean consumption, but the total weight of the remaining nine columns is 1.16 — that is, more than half of the component comes from columns outside the four large loadings. The answer to "why was this subscriber flagged as suspicious" is a column name and a threshold in the raw representation; in the component representation it is a weighted sum of thirteen numbers. The decision is a comparison of two losses. Going down to two components buys 0.0079 of number and a saving of eleven columns of storage; in return the ability to write the justification of every decision the model gives into a table is lost. In a field such as illicit-use suspicion the second loss is usually more expensive, because the justification of an action taken against a subscriber can be asked for. The feature role of the component representation is therefore a narrow role: it is valuable if the column count is truly too large to manage or if the columns are copies of each other, and not when there are thirteen columns measured one by one. #### Summary - The thirteen raw columns give 0.8254 on the held-out set; no number of components catches up with this, the closest row being two components with 0.8175 (difference 0.0079, two subscribers). - While the component share grows, the held-out number does not grow: 0.8056 with three components, 0.7778 with five, 0.7460 with eight; with thirteen components, with no loss of information, the number still stays at 0.7897. - The component with the largest share falls below the baseline on its own with 0.7381, while the fifth component with a share of 0.093 gives 0.7659; share ranking is not separation ranking. - Interpretability is a measurable loss: in the raw representation 0.35 of the splits are read from the mean consumption, in the component representation the same share falls on a mixture with no name. - The scale and the component directions are learned from the training set only; if they were learned from the whole set, the measurement would inflate. #### Next Step In four lessons the number of preparation steps grew, and in each of them the same sentence was repeated: this step is learned from the training set only. The scale mean, the region median, the selected column set, the component directions — all four are trained objects, and all four have to be applied by hand, separately, in the right order. When this chain is applied by hand, one has to count how many steps were learned from the wrong set. The next lesson combines the preparation and the model in a single object and measures how many steps leak without a pipeline. ### Pipelines Source: https://academia.sh/en/courses/feature-engineering/pipelines Binding the preparation chain to one object and counting how many steps leak without a pipeline: when the four-step chain is applied by hand, all four are learned from the whole set and the test number rises from 0.7817 to 0.8294, an inflation of 0.0476. When imputation and scaling leak, the number does not change at all; when target encoding leaks it inflates by 0.0397, and when selection leaks as well it inflates by 0.0476. The pipeline learns 131 numbers, and all of them come from the training set; the contribution of the scaling step in the tree model is exactly zero, while the contribution of the target encoding step is minus 0.0317. One sentence was repeated over four lessons: this step is learned from the training set only. The region median, the scale mean, the selected column set, the component directions — all four are numbers read from the data, and all four have to be read from the right set. As the sentence was repeated, another thing became visible: all of these steps have the same shape. Each has a **fit** face — it extracts a number or a mapping from the data — and a **transform** face — it works that number into the rows it is given. This shape is an object definition. When the steps are called one by one by hand, the order can get mixed up, a step can learn from the wrong set, a step can be forgotten at test time. The structure that gathers the steps in a single object is called a **pipeline**: the pipeline too has a fit face and sees only the training set, and it has a transform face and applies the same learned numbers to every set. This lesson writes the pipeline and asks a single question: when the same chain is applied by hand, how many steps leak, and how much does the number inflate. - **FS30.** The chain is four steps: missing value imputation, scaling, target encoding, feature selection. The model comes fifth and does not change: a decision tree of depth 6. - **FS31.** The missing satisfaction values and the `tariff_code` column are **fictional**, generated from the same seed. The tariff code depends on the region and the household size, so it carries a real relation with the label. - **FS32.** Target encoding learns the suspicious rate for each level of the tariff code. An **unseen category** falls to the overall rate. - **FS33.** In the leakage measurement the first `k` steps learn from the whole set and the rest from the training set; the model is trained from the training set only, in every case. - **FS34.** The inflation is the difference from the pipeline's number on the test set. The baseline is 0.7579. - **FS35.** The Data Leakage lesson of the Introduction to Machine Learning course counted the four kinds of leakage one by one; the channel here is its first kind — preparation before the split — and the inflation numbers are not repeated. #### The Two Faces of a Step ```python # COMMON — MODEL: the FICTIONAL subscriber table of the Introduction to Machine Learning course is rebuilt with the same seed; # the model is a decision tree of depth 6. The only new column left from the previous lesson is per_person. import math SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10), ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05), ("central", 0.18, 23, 0.05)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 SUBSCRIBERS, DATA = [], [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() >= 0.046: SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household, "satisfaction": rating}) for k in SUBSCRIBERS: r, v = rng(SEED + 7000 + k["id"]), [] for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): v.append(0.0 if r() < 0.038 else math.floor( k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - n * 0.05) * 100 + 0.5) / 100) r() if not v: continue mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"]) z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3] + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30) x = {"id": k["id"], "household": k["household"], "satisfaction": k["satisfaction"], "period": len(v), "mean_consumption": round(mean, 2), "volatility": round(vol, 2), "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35), "region_name": k["region"][0]} for b in REGION: x["area_" + b[0]] = int(k["region"][0] == b[0]) DATA.append(x) BASE = ["mean_consumption", "volatility", "household", "satisfaction", "period", "zero_readings", "area_north", "area_south", "area_east", "area_west", "area_central"] BASE_FIELDS = BASE + ["per_person"] for x in DATA: x["per_person"] = round(x["mean_consumption"] / x["household"], 3) def shuffle(data, seed): r, s = rng(seed), list(range(len(data))) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return [data[i] for i in s] K = shuffle(DATA, SEED + 90000) TRAIN, VAL, TEST = K[:756], K[756:1008], K[1008:] def gini(rows): p = sum(x["suspicious"] for x in rows) / len(rows) if rows else 0.0 return 2 * p * (1 - p) def tree(rows, depth, fields, min_size=2): p, best = sum(x["suspicious"] for x in rows) / len(rows), None if depth and len(rows) >= 2 * min_size and 0.0 < p < 1.0: for f in fields: values = sorted({x[f] for x in rows}) for v in (values[1:] if len(values) < 10 else [values[int(i * len(values) / 10)] for i in range(1, 10)]): left = [x for x in rows if x[f] < v] right = [x for x in rows if x[f] >= v] if min(len(left), len(right)) < min_size: continue k = gini(rows) - (len(left) * gini(left) + len(right) * gini(right)) / len(rows) if best is None or k > best[0]: best = (k, f, v, left, right) if best is None or best[0] <= 1e-9: return {"prediction": int(p > 0.5)} return {"field": best[1], "threshold": best[2], "left": tree(best[3], depth - 1, fields, min_size), "right": tree(best[4], depth - 1, fields, min_size)} def predict(node, x): while "prediction" not in node: node = node["left"] if x[node["field"]] < node["threshold"] else node["right"] return node["prediction"] def accuracy(node, rows): return sum(predict(node, x) == x["suspicious"] for x in rows) / len(rows) def baseline(train, rows): c = int(sum(x["suspicious"] for x in train) * 2 > len(train)) return sum(x["suspicious"] == c for x in rows) / len(rows) ``` All four steps carry the same two methods. `fit` looks only at the rows given to it and writes what it has learned to the `state` field; `apply` works that state into the rows it is given and learns nothing. The pipeline carries this pair to the whole chain. ```python for x in DATA: r = rng(SEED + 61000 + x["id"]) r(), r() x["kw_seal"] = int(r() < (0.42 if x["suspicious"] else 0.10)) r = rng(SEED + 21000 + x["id"]) x["tariff_code"] = (24 * [b[0] for b in REGION].index(x["region_name"]) + 4 * (x["household"] - 1) + int(r() * 4)) x["satisfaction"] = None if r() < 0.08 else x["satisfaction"] POOL = BASE_FIELDS + ["kw_seal", "tariff_target"] class Imputer: name = "imputation" def fit(self, rows): v = sorted(x["satisfaction"] for x in rows if x["satisfaction"] is not None) self.state = {"median": v[len(v) // 2]} def apply(self, rows): for x in rows: if x["satisfaction"] is None: x["satisfaction"] = self.state["median"] class Scaler: name = "scaling" def fit(self, rows): self.state = {} for f in ("mean_consumption", "volatility", "per_person"): m = sum(x[f] for x in rows) / len(rows) self.state[f] = (m, max(1e-9, (sum((x[f] - m) ** 2 for x in rows) / len(rows)) ** 0.5)) def apply(self, rows): for x in rows: for f, (m, d) in self.state.items(): x[f] = round((x[f] - m) / d, 4) class TargetEncoder: name = "target encoding" def fit(self, rows): g, self.overall = {}, sum(x["suspicious"] for x in rows) / len(rows) for x in rows: g.setdefault(x["tariff_code"], []).append(x["suspicious"]) self.state = {k: sum(v) / len(v) for k, v in g.items()} def apply(self, rows): for x in rows: x["tariff_target"] = round(self.state.get(x["tariff_code"], self.overall), 4) class Selector: name = "selection" def __init__(self, pool): self.pool = pool def fit(self, rows): g0, self.state = gini(rows), [] for f in self.pool: values, top = sorted({x[f] for x in rows}), 0.0 for v in (values[1:] if len(values) < 10 else [values[int(i * len(values) / 10)] for i in range(1, 10)]): left = [x for x in rows if x[f] < v] right = [x for x in rows if x[f] >= v] if min(len(left), len(right)) >= 20: top = max(top, g0 - (len(left) * gini(left) + len(right) * gini(right)) / len(rows)) if top >= 0.004: self.state.append(f) def apply(self, rows): pass class Pipeline: def __init__(self, steps, depth=6): self.steps, self.depth = steps, depth def fit(self, train, leakage_source=None, leaked=0): train = [dict(x) for x in train] dirty = [dict(x) for x in (leakage_source or [])] for i, step in enumerate(self.steps): if i < leaked: step.fit(dirty) step.apply(dirty) else: step.fit(train) step.apply(train) self.model = tree(train, self.depth, self.steps[-1].state) self.train_score = accuracy(self.model, train) return self def measure(self, rows): rows = [dict(x) for x in rows] for step in self.steps: step.apply(rows) return accuracy(self.model, rows) def chain(pool=None): return [Imputer(), Scaler(), TargetEncoder(), Selector(pool or POOL)] PL = Pipeline(chain()).fit(TRAIN) print(f"{'setup':<26} {'leaked':>6} {'columns':>7} {'train':>9} {'held_out':>9} " f"{'inflation':>9}") print(f"{'pipeline':<26} {0:>6} {len(PL.steps[-1].state):>7} {PL.train_score:>9.4f} " f"{PL.measure(TEST):>9.4f}") for k, label in ((1, "manual: imputation"), (2, "manual: + scaling"), (3, "manual: + target encoding"), (4, "manual: + selection")): b = Pipeline(chain()).fit(TRAIN, K, k) print(f"{label:<26} {k:>6} {len(b.steps[-1].state):>7} {b.train_score:>9.4f} " f"{b.measure(TEST):>9.4f} {b.measure(TEST) - PL.measure(TEST):>+9.4f}") print(f"\nbaseline {baseline(TRAIN, TEST):.4f}; tariff code " f"{len({x['tariff_code'] for x in DATA})} levels, missing satisfaction " f"{sum(1 for x in DATA if x['satisfaction'] is None)} rows") ``` ``` setup leaked columns train held_out inflation pipeline 0 8 0.8955 0.7817 manual: imputation 1 8 0.8955 0.7817 +0.0000 manual: + scaling 2 8 0.8955 0.7817 +0.0000 manual: + target encoding 3 8 0.9021 0.8214 +0.0397 manual: + selection 4 9 0.9008 0.8294 +0.0476 baseline 0.7579; tariff code 119 levels, missing satisfaction 103 rows ``` When the chain is applied by hand, **all four steps** can be learned from the whole set, because all four are a `fit` call, and which list is given to that call is written somewhere else in the code. In the pipeline this decision sits in a single place: the `fit` method takes the training set and the whole chain learns from that set. Adding a step does not increase the risk of leakage. The inflation column has to be read step by step. The first two rows are **zero**: when the imputation value is taken from the whole set the number does not change, and when the scale is taken from the whole set it does not change either. This does not mean those two steps did not leak — they did; the measured number did not change in this set and with this model. The decision tree makes its splits by ranking, and standardizing does not change the ranking; the median imputation affects 103 rows, and the two medians are close to each other. The non-appearance of a leak is not proof of its absence. The third row is target encoding, and the number inflates by 0.0397: ten subscribers. The reason is clear — this step uses the label directly. The tariff code has 119 levels and on average ten rows fall to each level; when the encoding is learned from the whole set, the own label of a test row makes up one tenth of the number that row will see. In the fourth row the selection is also made from the whole set, the number of selected columns goes from eight to nine, and the total inflation reaches 0.0476, twelve subscribers. The second rule of the course is completed in this table: **every preparation step is learned from the training set only and applied to the held-out set.** The pipeline turns this rule into a procedure; the rule stops being a written warning and becomes the interface of the object. #### What the Pipeline Measures The second benefit of the pipeline is not preventing leakage but making the steps **measurable**. When a step is removed from the chain and the same measurement is repeated, the honest contribution of that step can be read. ```python NARROW = BASE_FIELDS + ["kw_seal"] print("objects learned by the pipeline: " + ", ".join( f"{s.name} {len(s.state)}" for s in PL.steps)) seen = set(PL.steps[2].state) print(f"{len(seen)} of the {len({x['tariff_code'] for x in DATA})} levels of the tariff code " f"were seen in training; rows in the test set falling into an unseen category: " f"{sum(x['tariff_code'] not in seen for x in TEST)}\n") print(f"{'chain':<26} {'steps':>5} {'columns':>7} {'train':>9} {'held_out':>9} " f"{'contrib':>8}") print(f"{'four steps':<26} {4:>5} {len(PL.steps[-1].state):>7} {PL.train_score:>9.4f} " f"{PL.measure(TEST):>9.4f}") for label, steps in (("scaling removed", [Imputer(), TargetEncoder(), Selector(POOL)]), ("target encoding removed", [Imputer(), Scaler(), Selector(NARROW)]), ("only imputation+selection", [Imputer(), Selector(NARROW)])): b = Pipeline(steps).fit(TRAIN) print(f"{label:<26} {len(steps):>5} {len(b.steps[-1].state):>7} {b.train_score:>9.4f} " f"{b.measure(TEST):>9.4f} {b.measure(TEST) - PL.measure(TEST):>+8.4f}") ``` ``` objects learned by the pipeline: imputation 1, scaling 3, target encoding 116, selection 8 116 of the 119 levels of the tariff code were seen in training; rows in the test set falling into an unseen category: 1 chain steps columns train held_out contrib four steps 4 8 0.8955 0.7817 scaling removed 3 8 0.8955 0.7817 +0.0000 target encoding removed 3 7 0.8730 0.8135 +0.0317 only imputation+selection 2 7 0.8730 0.8135 +0.0317 ``` The pipeline learns 131 numbers in total: one median, two numbers each for three columns, one rate each for 116 tariff levels, and a list of eight columns. All of these 131 numbers come from the 756 rows of the training set and none of them looks at the test set. The leakage discussion is a discussion of how much of this count comes from which set. Of the 119 levels of the tariff code, 116 were seen in the training set; one of the remaining three levels meets us in the test set with a single row and falls to the overall rate as an **unseen category**. In a hand-written preparation chain this row typically turns into an error or a silent empty value; in the pipeline the transform face of the step makes the decision once and makes the same decision in every set. The lower table reads the honest contribution of two steps. When scaling is removed the number does **not** change at all: the contribution is exactly zero, because the decision tree looks at the ranking and a linear scale transformation does not change the ranking. When the step is done wrong it changes nothing either, so it can stay silently wrong. When target encoding is removed, on the other hand, the number **rises** from 0.7817 to 0.8135: the honest contribution of the step is minus 0.0317. Encoding a high-cardinality column with the target does harm in this set even when it is learned from the right set; when it was learned from the whole set it seemed to bring 0.0397. The same step is the most profitable step when it is leaked and the most harmful when it is applied honestly. For this reason the before-the-decision and after-the-decision numbers of this lesson are read in opposite directions. The chain applied by hand gives 0.8294, the pipeline 0.7817; the contribution is minus 0.0476. The minus sign here does not show a loss but that **the lost number was never learned**. What the pipeline adds is not a number but the correctness of the number. #### Summary - The preparation step is an object with two faces: `fit` sees only the training set and stores what it has learned, `apply` works those numbers into every set without learning anything. - When the chain is applied by hand, all four steps can be learned from the whole set; in the pipeline the decision sits in a single place and adding a step does not increase the risk of leakage. - The inflation comes out as 0.0000, 0.0000, 0.0397 and 0.0476 step by step; the leakage of the first two steps does not change the number, and the non-appearance of a leak is not proof of its absence. - The pipeline learns 131 numbers and all of them come from the 756 rows of the training set; 116 of the 119 levels of the tariff code were seen, and one row in the test set falls into an unseen category. - The same step brings 0.0397 when it is leaked and loses 0.0317 when it is applied honestly; the contribution of the pipeline is minus 0.0476, and this minus shows that the lost number was never learned. #### Next Step The pipeline bound the whole chain to learning from a single set, and the measurement was read from a single split: 756 rows of training, 252 rows of testing. Those 252 rows were used once and the number they gave is a single number; another split would have given another number. Taking the measurement over several splits is called cross-validation, and when the preparation chain enters here a new question arises: is the chain learned once outside the folds, or again inside each fold. The two procedures give two different numbers, and the next lesson measures which one comes out closer to the real number on the held-out set. ### Fitting with Cross-Validation Source: https://academia.sh/en/courses/feature-engineering/fitting-with-cross-validation Comparing the two cross-validation numbers that come from learning the preparation chain outside the folds and inside the folds with the actual number on the held-out set: while the actual number on the test set is 0.7817, preparation inside the folds gives 0.7966 and preparation outside the folds gives 0.8105, and the deviation of preparation inside the folds is 0.0149 against 0.0288 for the one outside. The moment the second step of the chain, target encoding, moves outside the folds, the number jumps from 0.7966 to 0.8115. The 0.8254 given by the single split equals the highest of the four folds and is 0.0437 away from the actual number. In the course wrap-up, the before-decision and after-decision numbers of the twelve lessons are gathered in a single table. The previous lesson bound the preparation chain to an object and guaranteed that the whole chain was learned from the training set only. The measurement was still read from a single split: 756 rows of training, 252 rows of testing. Those 252 rows were used once and gave a single number. Another split would have given another number, and there was no way of saying which one was right. **Cross-validation** answers this problem: the data left in hand is divided into **fold**s, each fold becomes the measurement set in turn, the rest becomes the training set, and in the end several numbers and their spread are obtained. When the preparation chain enters the work a new question arises, and the second rule of the course takes its final form here: is the chain learned once outside the folds, or again inside each fold. The two procedures give two separate numbers; this lesson measures which one comes out close to the **actual number on the held-out set**. - **FS36.** The cross-validation pool is the training and validation sets of the Introduction to Machine Learning course: 1,008 rows. The 252 rows of the test set enter no fold, no preparation step and no selection. - **FS37.** The number of folds is **four**; in each fold 252 rows become the measurement set, that is, the same size as the test set. The number of folds was not chosen by looking at the result. - **FS38.** The chain is the chain of the previous lesson with the scaling step removed: imputation, target encoding, feature selection. The contribution of scaling in the decision tree was measured there as exactly zero. - **FS39.** The number of steps learned outside the folds is varied from 0 to 3. Zero means the whole chain is learned inside the folds. - **FS40.** The actual number is the number for which the chain is learned from 1,008 rows and measured on the test set: it is the single number of this course that is read once. - **FS41.** The spread is the mean distance of the four fold numbers from their own average and is the measure of dispersion of the Data Analytics curriculum. - **FS42.** The tuning budget calculation is not the subject of this lesson and is left to the Supervised Learning course; no model tuning is chosen here, and the model is the decision tree of depth 6 that stays fixed throughout the course. #### Outside the Folds and Inside the Folds ```python # COMMON — MODEL: the FICTIONAL subscriber table of the Introduction to Machine Learning course is rebuilt with the same seed; # the model is a decision tree of depth 6. The only new column left from the previous lesson is per_person. import math SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10), ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05), ("central", 0.18, 23, 0.05)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 SUBSCRIBERS, DATA = [], [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() >= 0.046: SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household, "satisfaction": rating}) for k in SUBSCRIBERS: r, v = rng(SEED + 7000 + k["id"]), [] for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): v.append(0.0 if r() < 0.038 else math.floor( k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - n * 0.05) * 100 + 0.5) / 100) r() if not v: continue mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"]) z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3] + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30) x = {"id": k["id"], "household": k["household"], "satisfaction": k["satisfaction"], "period": len(v), "mean_consumption": round(mean, 2), "volatility": round(vol, 2), "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35), "region_name": k["region"][0]} for b in REGION: x["area_" + b[0]] = int(k["region"][0] == b[0]) DATA.append(x) BASE = ["mean_consumption", "volatility", "household", "satisfaction", "period", "zero_readings", "area_north", "area_south", "area_east", "area_west", "area_central"] BASE_FIELDS = BASE + ["per_person"] for x in DATA: x["per_person"] = round(x["mean_consumption"] / x["household"], 3) def shuffle(data, seed): r, s = rng(seed), list(range(len(data))) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return [data[i] for i in s] K = shuffle(DATA, SEED + 90000) TRAIN, VAL, TEST = K[:756], K[756:1008], K[1008:] def gini(rows): p = sum(x["suspicious"] for x in rows) / len(rows) if rows else 0.0 return 2 * p * (1 - p) def tree(rows, depth, fields, min_size=2): p, best = sum(x["suspicious"] for x in rows) / len(rows), None if depth and len(rows) >= 2 * min_size and 0.0 < p < 1.0: for f in fields: values = sorted({x[f] for x in rows}) for v in (values[1:] if len(values) < 10 else [values[int(i * len(values) / 10)] for i in range(1, 10)]): left = [x for x in rows if x[f] < v] right = [x for x in rows if x[f] >= v] if min(len(left), len(right)) < min_size: continue k = gini(rows) - (len(left) * gini(left) + len(right) * gini(right)) / len(rows) if best is None or k > best[0]: best = (k, f, v, left, right) if best is None or best[0] <= 1e-9: return {"prediction": int(p > 0.5)} return {"field": best[1], "threshold": best[2], "left": tree(best[3], depth - 1, fields, min_size), "right": tree(best[4], depth - 1, fields, min_size)} def predict(node, x): while "prediction" not in node: node = node["left"] if x[node["field"]] < node["threshold"] else node["right"] return node["prediction"] def accuracy(node, rows): return sum(predict(node, x) == x["suspicious"] for x in rows) / len(rows) def baseline(train, rows): c = int(sum(x["suspicious"] for x in train) * 2 > len(train)) return sum(x["suspicious"] == c for x in rows) / len(rows) ``` The pipeline is the object of the previous lesson. This time the `fit` method takes two extra values: the source outside the folds and the number of steps that will learn from that source. When zero is given, the whole chain learns from the inner fold set it is given. ```python for x in DATA: r = rng(SEED + 61000 + x["id"]) r(), r() x["kw_seal"] = int(r() < (0.42 if x["suspicious"] else 0.10)) r = rng(SEED + 21000 + x["id"]) x["tariff_code"] = (24 * [b[0] for b in REGION].index(x["region_name"]) + 4 * (x["household"] - 1) + int(r() * 4)) x["satisfaction"] = None if r() < 0.08 else x["satisfaction"] POOL = BASE_FIELDS + ["kw_seal", "tariff_target"] class Imputer: name = "imputation" def fit(self, rows): v = sorted(x["satisfaction"] for x in rows if x["satisfaction"] is not None) self.state = {"median": v[len(v) // 2]} def apply(self, rows): for x in rows: if x["satisfaction"] is None: x["satisfaction"] = self.state["median"] class TargetEncoder: name = "target encoding" def fit(self, rows): g, self.overall = {}, sum(x["suspicious"] for x in rows) / len(rows) for x in rows: g.setdefault(x["tariff_code"], []).append(x["suspicious"]) self.state = {k: sum(v) / len(v) for k, v in g.items()} def apply(self, rows): for x in rows: x["tariff_target"] = round(self.state.get(x["tariff_code"], self.overall), 4) class Selector: name = "selection" def __init__(self, pool): self.pool = pool def fit(self, rows): g0, self.state = gini(rows), [] for f in self.pool: values, top = sorted({x[f] for x in rows}), 0.0 for v in (values[1:] if len(values) < 10 else [values[int(i * len(values) / 10)] for i in range(1, 10)]): left = [x for x in rows if x[f] < v] right = [x for x in rows if x[f] >= v] if min(len(left), len(right)) >= 20: top = max(top, g0 - (len(left) * gini(left) + len(right) * gini(right)) / len(rows)) if top >= 0.004: self.state.append(f) def apply(self, rows): pass class Pipeline: def __init__(self, steps, depth=6): self.steps, self.depth = steps, depth def fit(self, train, outside_source=None, outside_count=0): train = [dict(x) for x in train] outside = [dict(x) for x in (outside_source or [])] for i, step in enumerate(self.steps): if i < outside_count: step.fit(outside) step.apply(outside) else: step.fit(train) step.apply(train) self.model = tree(train, self.depth, self.steps[-1].state) self.train_score = accuracy(self.model, train) return self def measure(self, rows): rows = [dict(x) for x in rows] for step in self.steps: step.apply(rows) return accuracy(self.model, rows) def chain(pool=None): return [Imputer(), TargetEncoder(), Selector(pool or POOL)] REST, FOLDS = K[:1008], 4 def split_fold(data, i, k=FOLDS): n = len(data) // k return ([x for j, x in enumerate(data) if not i * n <= j < (i + 1) * n], data[i * n:(i + 1) * n]) def cross_validate(outside_count): scores = [] for i in range(FOLDS): inner, outer = split_fold(REST, i) scores.append(Pipeline(chain()).fit(inner, REST, outside_count).measure(outer)) return scores def average(scores): return sum(scores) / len(scores) def spread(scores): m = average(scores) return (sum((q - m) ** 2 for q in scores) / len(scores)) ** 0.5 ACTUAL = Pipeline(chain()).fit(REST).measure(TEST) SINGLE = Pipeline(chain()).fit(K[:756]).measure(K[756:1008]) print(f"{'outside folds':<14} " + " ".join(f"{'fold ' + str(i + 1):>7}" for i in range(FOLDS)) + f" {'average':>9} {'spread':>8} {'diff':>8}") for d in range(4): scores = cross_validate(d) print(f"{'steps: ' + str(d):<14} " + " ".join(f"{q:>7.4f}" for q in scores) + f" {average(scores):>9.4f} {spread(scores):>8.4f} {average(scores) - ACTUAL:>+8.4f}") print(f"\nsingle split 756/252 {SINGLE:.4f}, diff {SINGLE - ACTUAL:+.4f}") print(f"actual number on the test set {ACTUAL:.4f}, baseline {baseline(REST, TEST):.4f}") ``` ``` outside folds fold 1 fold 2 fold 3 fold 4 average spread diff steps: 0 0.7976 0.7738 0.7897 0.8254 0.7966 0.0187 +0.0149 steps: 1 0.7976 0.7738 0.7897 0.8254 0.7966 0.0187 +0.0149 steps: 2 0.8016 0.8135 0.8333 0.7976 0.8115 0.0139 +0.0298 steps: 3 0.8016 0.8056 0.8452 0.7897 0.8105 0.0209 +0.0288 single split 756/252 0.8254, diff +0.0437 actual number on the test set 0.7817, baseline 0.7579 ``` The last column is the answer of the lesson. When the whole chain is learned inside the folds, cross-validation gives 0.7966 and stays 0.0149 above the actual number. When the whole chain is learned outside the folds it gives 0.8105 and the deviation rises to 0.0288, that is, almost doubles. **Preparation inside the folds is the procedure that comes out close to the actual number.** Where the jump comes from can be seen in the middle rows. When imputation moves outside the folds nothing changes; the moment target encoding moves outside the folds the average jumps from 0.7966 to 0.8115. Selection adds to this a drop below the measurement resolution. The whole cost of preparing outside the folds comes from a single step, and that step is the one that uses the label. This has the same structure as the table in the previous lesson and arises for the same reason. Target encoding learned outside the folds mixes the labels of the measurement rows of each fold into the number those rows will see. As the number of folds grows the share of the measurement rows falls and the inflation shrinks; as it falls the inflation grows. For this reason **a cross-validation number prepared outside the folds cannot be read independently of its own number of folds.** The sign of the deviation also deserves attention. Preparation inside the folds exceeds the actual number by 0.0149, that is, it is still optimistic; the reason is not leakage but the chain looking at the data once more in every fold. Cross-validation does not measure the actual number, it measures an interval around the actual number. #### The Single Number of a Single Split The single split gives 0.8254 and is 0.0437 away from the actual number. This number equals the **highest** of the four fold numbers prepared inside the folds. The single split did not make a wrong calculation; it happened to land on one end of this interval, and since it is a single number it could not say which end it landed on. The spread of the four folds is 0.0187. A number read from one split carries a fluctuation of this size, and separating a contribution of 0.0119 from this fluctuation is impossible with a single split. For this reason the contributions of 0.0119 measured in the first three lessons of the course were written as "measured and could not be told apart from zero". What cross-validation adds is not a bigger number but an **interval** next to the number. And that interval is the true interval only when the preparation chain is learned inside the folds. #### Summary - The actual number on the test set is 0.7817; preparation inside the folds gives 0.7966 (deviation 0.0149), preparation outside the folds gives 0.8105 (deviation 0.0288). - The whole deviation comes from a single step: when imputation moves outside the folds the number does not change, when target encoding moves outside the folds the average jumps from 0.7966 to 0.8115. - A cross-validation number prepared outside the folds cannot be read independently of the number of folds, because the size of the inflation depends on the share of the measurement rows. - The 0.8254 given by the single split equals the highest of the four folds and is 0.0437 away from the actual number; a single number cannot say which end it landed on. - The spread of the folds is 0.0187; with a fluctuation of this size, a contribution of 0.0119 cannot be told apart from zero with a single split. #### Course Wrap-Up The accounts of twelve lessons stand in a single table, and every number was read from its own lesson. The before-decision and after-decision columns were always measured on the **held-out set**; the baseline is 0.7579 in all the lessons. | Lesson | Before decision | After decision | Contribution | |---|---|---|---| | Preparation 01 Data Cleaning | 0.7659 raw records | 0.7778 three corrections | +0.0119 | | Preparation 02 Feature Scaling | 0.7579 unscaled | 0.8016 standard score | +0.0437 | | Preparation 03 Categorical Encoding | 0.7302 field not used | 0.7778 one-hot encoding | +0.0476 | | Preparation 04 Missing Value Strategies | 0.7619 median imputation | 0.8333 missingness indicator | +0.0714 | | Preparation 05 Imbalanced Classes | 0.7778 raw set | 0.7619 two-fold oversampling | −0.0159 | | Preparation 06 Outlier Management | 0.7698 no clipping | 0.8016 clipped at the fence | +0.0317 | | Selection 01 Feature Generation | 0.7778 eleven columns | 0.7897 ratio feature | +0.0119 | | Selection 02 Time and Text Features | 0.7897 twelve columns | 0.8214 five numbers from the note | +0.0317 | | Selection 03 Feature Selection | 0.8135 no selection | 0.8254 embedded method | +0.0119 | | Selection 04 Dimensionality Reduction as a Feature Tool | 0.8254 raw thirteen columns | 0.8175 two components | −0.0079 | | Selection 05 Pipelines | 0.8294 manual chain | 0.7817 pipeline | −0.0476 | | Selection 06 Fitting with Cross-Validation | 0.8105 outside the folds | 0.7966 inside the folds | −0.0139 | The rows of the second and sixth preparation lessons are read from the gradient-based linear model, the remaining ten rows from the fixed decision tree of the course; the row of the fifth preparation lesson is the accuracy column, and the same decision raises the number of suspicious subscribers caught from 21 to 33. The first rule of the course is the table itself: **a preparation step exists to the extent of its contribution measured on the held-out set.** In the contribution column **four** rows are zero or negative. If these four had not been measured they would have stayed in the set, and all four would have been defended with a reasonable justification. The second rule of the course is read in the last two rows: **every preparation step is learned from the training set only and applied to the held-out set.** The minus signs in those two rows are not a loss but an inflation taken back; the pipeline lowered the number by 0.0476, preparation inside the folds by 0.0139, and none of the numbers that fell came from learning. Before looking at the sign of a measured contribution, one has to look at where it comes from. One account opened at the start of the course remains. The first lesson wrote that every look at the test set accumulates a **selection share** and left the accounting of the share to the end of the course. The table itself gives that account: each of the twelve rows carries two numbers, that is, the test set was looked at **at least 24 times**, and with the sweeps inside the lessons this number is much higher. For this reason no number in the table is an estimate of generalization; all of them are a comparison. The actual number can only be read on a set that was never touched during this course, and that set no longer exists. In twelve lessons the features were built and the contribution of each step was measured on the held-out set. But throughout all these measurements the **model** was always chosen from the same two or three families: a decision tree of depth 6, beside it a gradient-based linear model and a distance-based method. What the family itself adds was never asked; why the depth is six, why the number of neighbors is fifteen and from which set these numbers should be read was never measured. The next course, Supervised Learning, takes up these questions.