--- title: 'Feature Generation' source: 'https://academia.sh/en/courses/feature-engineering/feature-generation' course: 'Data Preparation and Feature Engineering' language: en updated: '2026-10-07T04:31:19+00:00' license: 'CC BY-SA 4.0' objectives: - 'Generates ratio, difference, interaction, threshold indicator and learned indicator features and measures the contribution of each on the held-out set separately' - 'Writes the features with zero and negative contribution into the table as well and tells apart two separate causes of zero: a near-constant column and a threshold the model has already found' - 'Shows with a cumulative addition table that contributions measured one by one cannot be added up' --- # Feature Generation Counting one by one what new columns derived from existing columns add on the held-out set: the ratio feature carries the test number from 0.7778 to 0.7897 while the difference and the indicator learned from the training set add 0.0040, the hand-set 25 m³ threshold loses 0.0159 and the product of two columns loses 0.0238, and the 40 m³ threshold produces exactly zero contribution. When the six columns are added cumulatively the training number rises from 0.8466 to 0.8677 while the held-out number peaks at 0.7897 and falls to 0.7698; the three columns that each add a positive contribution on their own add a total contribution of 0.0000 when added together. The previous six lessons worked on existing columns: missing values were imputed, the scale was learned only from the training set, categories were encoded, imbalance was handled, outliers were assessed according to the model. In all of them the set of columns stayed the same. No **new column** was generated and no column was dropped; in every lesson the model looked at the same eleven numbers. This lesson increases the number of columns for the first time. Its question is this: whether a new number derived from existing columns by arithmetic **adds information** to the model or only **widens the search space**. The rule of the course holds here too — a preparation step exists to the extent of its contribution measured on the held-out set — and in this lesson six features are generated one by one and measured one by one. Those whose contribution comes out zero or negative are also written into the table. - **FS1.** The dataset is the **fictional** subscriber table of the Introduction to Machine Learning course: the same seed (20260218), the same split (756 training, 252 validation, 252 test), the same label. The held-out set is the test set. - **FS2.** The model is held **fixed** throughout the course: a decision tree of depth 6. The only thing that changes is the set of columns. The baseline is telling the most frequent class and it is 0.7579 on the test set. - **FS3.** The line before the decision is the line of the eleven columns the preparation steps left behind: 0.8466 on training, 0.7778 on test. - **FS4.** The raw readings are still kept. `mean_consumption` and `volatility` are a summary of those readings, and that summarizing was itself a feature generation decision; the lesson reopens this decision. - **FS5.** Features computed at the row level (ratio, difference, product, fixed threshold) learn nothing from the dataset and are independent of the split. - **FS6.** A feature that learns a number from the dataset — being above the region median — reads that number **only from the training set** and applies it to all three sets with the same number. - **FS7.** Contribution is the difference of the same model's number on the held-out set from the line before the decision. In a 252-row set, one row corresponds to 0.0040. ## Six New Numbers from the Same Columns The setup is the same as in the Introduction to Machine Learning course and produces the same set with the same seed. This time the list of raw readings is also kept, because one of the features to be generated depends on it. ```python # generation.py — MODEL: the FICTIONAL subscriber table of the Introduction to Machine Learning course is rebuilt with the same seed; # the model is a decision tree of depth 6 and stays fixed throughout the course. import math SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10), ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05), ("central", 0.18, 23, 0.05)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 SUBSCRIBERS, DATA = [], [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) if r() >= 0.046: SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household, "satisfaction": rating}) for k in SUBSCRIBERS: r, v = rng(SEED + 7000 + k["id"]), [] for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): v.append(0.0 if r() < 0.038 else math.floor( k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - n * 0.05) * 100 + 0.5) / 100) r() if not v: continue mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"]) z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3] + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30) x = {"id": k["id"], "household": k["household"], "satisfaction": k["satisfaction"], "period": len(v), "mean_consumption": round(mean, 2), "volatility": round(vol, 2), "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35), "readings": [round(q, 2) for q in v], "region_name": k["region"][0]} for b in REGION: x["area_" + b[0]] = int(k["region"][0] == b[0]) DATA.append(x) BASE_FIELDS = ["mean_consumption", "volatility", "household", "satisfaction", "period", "zero_readings", "area_north", "area_south", "area_east", "area_west", "area_central"] def shuffle(data, seed): r, s = rng(seed), list(range(len(data))) for i in range(len(s) - 1, 0, -1): j = int(r() * (i + 1)) s[i], s[j] = s[j], s[i] return [data[i] for i in s] K = shuffle(DATA, SEED + 90000) TRAIN, VAL, TEST = K[:756], K[756:1008], K[1008:] def gini(rows): p = sum(x["suspicious"] for x in rows) / len(rows) if rows else 0.0 return 2 * p * (1 - p) def tree(rows, depth, fields, min_size=2): p, best = sum(x["suspicious"] for x in rows) / len(rows), None if depth and len(rows) >= 2 * min_size and 0.0 < p < 1.0: for f in fields: values = sorted({x[f] for x in rows}) for v in (values[1:] if len(values) < 10 else [values[int(i * len(values) / 10)] for i in range(1, 10)]): left = [x for x in rows if x[f] < v] right = [x for x in rows if x[f] >= v] if min(len(left), len(right)) < min_size: continue k = gini(rows) - (len(left) * gini(left) + len(right) * gini(right)) / len(rows) if best is None or k > best[0]: best = (k, f, v, left, right) if best is None or best[0] <= 1e-9: return {"prediction": int(p > 0.5)} return {"field": best[1], "threshold": best[2], "left": tree(best[3], depth - 1, fields, min_size), "right": tree(best[4], depth - 1, fields, min_size)} def predict(node, x): while "prediction" not in node: node = node["left"] if x[node["field"]] < node["threshold"] else node["right"] return node["prediction"] def accuracy(node, rows): return sum(predict(node, x) == x["suspicious"] for x in rows) / len(rows) def baseline(train, rows): c = int(sum(x["suspicious"] for x in train) * 2 > len(train)) return sum(x["suspicious"] == c for x in rows) / len(rows) def measure(fields, depth=6): m = tree(TRAIN, depth, fields) return accuracy(m, TRAIN), accuracy(m, TEST) ``` The six features to be generated represent five types. **Ratio** is the quotient of two columns: consumption per person separates the size of the household from consumption. **Difference** is the change between the last and the first of the raw readings; `volatility` carries only the span, not the direction. The **interaction feature** is the product of two columns. The **threshold indicator** comes from domain knowledge: the tariff tiers stop at 25 and 40 m³. The last one **learns** a number from the dataset — whether the subscriber is above the median of their own region — and that median is read from the training set only. ```python for x in DATA: o = x["readings"] x["per_person"] = round(x["mean_consumption"] / x["household"], 3) x["last_first_diff"] = round(o[-1] - o[0], 2) x["household_satisfaction"] = x["household"] * x["satisfaction"] x["threshold_25"] = int(x["mean_consumption"] > 25) x["threshold_40"] = int(x["mean_consumption"] > 40) MEDIAN = {} for b in REGION: v = sorted(x["mean_consumption"] for x in TRAIN if x["region_name"] == b[0]) MEDIAN[b[0]] = v[len(v) // 2] for x in DATA: x["region_above"] = int(x["mean_consumption"] > MEDIAN[x["region_name"]]) BEFORE = measure(BASE_FIELDS) NEW = ["per_person", "last_first_diff", "region_above", "threshold_25", "household_satisfaction", "threshold_40"] print(f"{'feature':<22} {'type':<12} {'train':>9} {'held_out':>9} {'contrib':>8}") print(f"{'-- none --':<22} {'11 columns':<12} {BEFORE[0]:>9.4f} {BEFORE[1]:>9.4f}") for a, t in zip(NEW, ["ratio", "difference", "learned", "threshold", "interaction", "threshold"]): e, s = measure(BASE_FIELDS + [a]) print(f"{a:<22} {t:<12} {e:>9.4f} {s:>9.4f} {s - BEFORE[1]:>+8.4f}") print(f"\nbaseline {baseline(TRAIN, TEST):.4f}; above 25 m3 {sum(x['threshold_25'] for x in DATA)} " f"rows, above 40 m3 {sum(x['threshold_40'] for x in DATA)} rows") ``` ``` feature type train held_out contrib -- none -- 11 columns 0.8466 0.7778 per_person ratio 0.8519 0.7897 +0.0119 last_first_diff difference 0.8571 0.7817 +0.0040 region_above learned 0.8492 0.7817 +0.0040 threshold_25 threshold 0.8479 0.7619 -0.0159 household_satisfaction interaction 0.8492 0.7540 -0.0238 threshold_40 threshold 0.8466 0.7778 +0.0000 baseline 0.7579; above 25 m3 243 rows, above 40 m3 2 rows ``` The contribution column carries three signs at once. The largest gain comes from the ratio feature: 0.0119, that is, three subscribers in a 252-row set. This is a column that arithmetically carries no new information — `mean_consumption` and `household` are already in the set — but the tree makes its splits **on a single column** and cannot build the quotient of two columns with one split. What the ratio adds is not information but the **splittable form** of that information. The difference feature and the indicator learned from the training set earn one row each: 0.0040. That is the correct classification of a single subscriber and is too small to be told apart from chance. A number like this cannot be reported as "there is a contribution"; at most it can be reported as "measured and could not be told apart from zero". There is a difference of kind between these two columns, and the second rule of the course passes exactly through it. `last_first_diff` is computed inside each row on its own; it sees no other row in the dataset and gives the same number before and after the split. `region_above`, on the other hand, rests on a number — the region median — and that number is **read** from the data. In the code this distinction sits on a single line: the median dictionary is built over `TRAIN`, not over `DATA`. Had the source been `DATA`, the values of the column would not have changed at all, the 252 rows in the test set would have contributed to determining their own thresholds, and a share of the measured contribution would have come from leakage rather than from learning. A generated feature is also **a learned object**, and writing where it learned is as obligatory as writing the number it contributed. This distinction will become the founding principle of the pipeline in the fifth lesson. ## Two Separate Causes of Zero The most instructive row of the table is the 40 m³ threshold: the contribution is **exactly zero**, and not even the training number has moved by a hair. The cause shows in the last row — there are only 2 rows above 40 m³ in the set. A near-constant column cannot produce two sides of at least two rows each in any split, so the tree never selects it. The column was added, it was measured, and the tree the model sees is exactly the same. The 25 m³ threshold is a different kind of zero and in fact is not even zero, it is negative: a loss of 0.0159, four subscribers. This column separates 243 rows, so it is not constant. But the information it carries is already inside `mean_consumption`, and the tree finds its own threshold by **searching** the data. The hand-set threshold says nothing new; in return it competes as a split candidate at every node, and gets a split that looks slightly better on the training set selected, producing a worse tree on the held-out set. The largest loss comes from the interaction feature: 0.0238, six subscribers. The product of `household` and `satisfaction` squeezes the two columns into one number, and its relationship with the label is weaker than that of the two columns themselves. When the tree selects the product it makes the separation coarser. This is the typical behavior of interactions that do not come from domain knowledge and are generated on a "might work" basis. Three separate causes of zero-or-below must be told apart: the column **cannot be split at all** (40 m³), its information is **already there** (25 m³), or it carries the information **by distorting it** (the product). If all three are summarized as "no contribution", what is lost in the table is which fix will work. ## Contributions Do Not Add Up The contributions measured one by one gave three positive values: 0.0119, 0.0040, 0.0040. Their sum is 0.0199. One needs to look at what happens when the columns are added cumulatively. ```python cumulative = list(BASE_FIELDS) print(f"{'added':<22} {'columns':>7} {'train':>9} {'held_out':>9} {'contrib':>8}") for a in NEW: cumulative.append(a) e, s = measure(cumulative) print(f"{a:<22} {len(cumulative):>7} {e:>9.4f} {s:>9.4f} {s - BEFORE[1]:>+8.4f}") POSITIVE = ["per_person", "last_first_diff", "region_above"] e, s = measure(BASE_FIELDS + POSITIVE[:2]) print(f"\nonly two contributing columns {e:.4f} {s:.4f} {s - BEFORE[1]:+.4f}") e, s = measure(BASE_FIELDS + POSITIVE) print(f"only three contributing columns {e:.4f} {s:.4f} {s - BEFORE[1]:+.4f}") ``` ``` added columns train held_out contrib per_person 12 0.8519 0.7897 +0.0119 last_first_diff 13 0.8598 0.7897 +0.0119 region_above 14 0.8611 0.7778 +0.0000 threshold_25 15 0.8664 0.7698 -0.0079 household_satisfaction 16 0.8677 0.7698 -0.0079 threshold_40 17 0.8677 0.7698 -0.0079 only two contributing columns 0.8598 0.7897 +0.0119 only three contributing columns 0.8611 0.7778 +0.0000 ``` If the two columns were independent of each other, their contributions would add up. They do not. Two columns that add 0.0119 and 0.0040 one by one add 0.0119 together; when the third is added the total contribution falls to **0.0000**. Three features that measured positive one by one return, together, to the line before the decision. The cause is that the tree is greedy: at each node it selects the split that looks best at that moment, and the selection is not undone. A new column can offer a better-looking split at the root node and send the rest of the tree down an entirely different path. The contribution is not the column's own information but **the effect of the column on the model's search path**, and this effect is not an additive quantity. The two numbers must be read separately. The training number rises in one direction: from 0.8466 to 0.8677, each of the six columns pulls it a bit higher. The held-out number, on the other hand, peaks at twelve columns and falls, and at seventeen columns it stays below the line before the decision. The Overfitting and Underfitting lesson of the Introduction to Machine Learning course produced this divergence by increasing capacity; here the model is fixed and the only thing growing is the **number of columns**. Adding features is also increasing capacity. The rule of the course takes concrete form in this table. Six columns were generated, all six with a reasonable justification, and when measured on the held-out set only one of them remained. Had it not been measured, all six would have stayed in the set and the model would have worked with seventeen columns at 0.7698, 0.0080 below the line before the decision — and moreover the table would have looked better because the training number rose to 0.8677. ## Summary - The ratio feature carries the held-out number from 0.7778 to 0.7897 (0.0119, three subscribers); the difference feature and the region indicator learned from the training set add 0.0040, that is, one subscriber each. - Those with zero and negative contribution are written as well: the 40 m³ threshold is exactly 0.0000, the 25 m³ threshold loses 0.0159, the product of two columns loses 0.0238. - The cause of the zero is separated: the column cannot be split because it is near-constant, its information is already in another column, or it carries the information by distorting it. - Contributions do not add up: three columns that add 0.0119, 0.0040 and 0.0040 one by one add 0.0000 together, because the contribution is not the column's information but its effect on the model's search path. - While the number of columns goes from 11 to 17, the training number rises in one direction from 0.8466 to 0.8677 and the held-out number peaks at 0.7897 and falls to 0.7698; adding features is also increasing capacity. ## Next Step The six features of this lesson were derived from numerical columns by arithmetic. In the subscriber table, however, there are two more non-numerical fields: the date the contract started and the short note left by the field team. Neither can be given to a model as it is; multiple numbers can be derived from each. How many numbers come out of a date, how many numbers come out of a short text, and what the **total** contribution of these numbers is — the next lesson measures it.