Feature Generation

The previous six lessons worked on existing columns: missing values were imputed, the scale was learned only from the training set, categories were encoded, imbalance was handled, outliers were assessed according to the model. In all of them the set of columns stayed the same. No new column was generated and no column was dropped; in every lesson the model looked at the same eleven numbers.

This lesson increases the number of columns for the first time. Its question is this: whether a new number derived from existing columns by arithmetic adds information to the model or only widens the search space. The rule of the course holds here too — a preparation step exists to the extent of its contribution measured on the held-out set — and in this lesson six features are generated one by one and measured one by one. Those whose contribution comes out zero or negative are also written into the table.

• FS1. The dataset is the fictional subscriber table of the Introduction to Machine Learning course: the same seed (20260218), the same split (756 training, 252 validation, 252 test), the same label. The held-out set is the test set.
• FS2. The model is held fixed throughout the course: a decision tree of depth 6. The only thing that changes is the set of columns. The baseline is telling the most frequent class and it is 0.7579 on the test set.
• FS3. The line before the decision is the line of the eleven columns the preparation steps left behind: 0.8466 on training, 0.7778 on test.
• FS4. The raw readings are still kept. mean_consumption and volatility are a summary of those readings, and that summarizing was itself a feature generation decision; the lesson reopens this decision.
• FS5. Features computed at the row level (ratio, difference, product, fixed threshold) learn nothing from the dataset and are independent of the split.
• FS6. A feature that learns a number from the dataset — being above the region median — reads that number only from the training set and applies it to all three sets with the same number.
• FS7. Contribution is the difference of the same model’s number on the held-out set from the line before the decision. In a 252-row set, one row corresponds to 0.0040.

Six New Numbers from the Same Columns

The setup is the same as in the Introduction to Machine Learning course and produces the same set with the same seed. This time the list of raw readings is also kept, because one of the features to be generated depends on it.

# generation.py — MODEL: the FICTIONAL subscriber table of the Introduction to Machine Learning course is rebuilt with the same seed;
# the model is a decision tree of depth 6 and stays fixed throughout the course.
import math

SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF
REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10),
          ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05),
          ("central", 0.18, 23, 0.05)]


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


SUBSCRIBERS, DATA = [], []
for i in range(RAW):
    r = rng(SEED + i)
    b = REGION[pick(r(), [x[1] for x in REGION])]
    household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05])
    rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17])
    if r() >= 0.046:
        SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household,
                            "satisfaction": rating})
for k in SUBSCRIBERS:
    r, v = rng(SEED + 7000 + k["id"]), []
    for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])):
        v.append(0.0 if r() < 0.038 else math.floor(
            k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62)
            * (1 - n * 0.05) * 100 + 0.5) / 100)
        r()
    if not v:
        continue
    mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"])
    z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol
         + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3]
         + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30)
    x = {"id": k["id"], "household": k["household"], "satisfaction": k["satisfaction"],
         "period": len(v), "mean_consumption": round(mean, 2), "volatility": round(vol, 2),
         "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35),
         "readings": [round(q, 2) for q in v], "region_name": k["region"][0]}
    for b in REGION:
        x["area_" + b[0]] = int(k["region"][0] == b[0])
    DATA.append(x)

BASE_FIELDS = ["mean_consumption", "volatility", "household", "satisfaction", "period",
               "zero_readings", "area_north", "area_south", "area_east", "area_west",
               "area_central"]


def shuffle(data, seed):
    r, s = rng(seed), list(range(len(data)))
    for i in range(len(s) - 1, 0, -1):
        j = int(r() * (i + 1))
        s[i], s[j] = s[j], s[i]
    return [data[i] for i in s]


K = shuffle(DATA, SEED + 90000)
TRAIN, VAL, TEST = K[:756], K[756:1008], K[1008:]


def gini(rows):
    p = sum(x["suspicious"] for x in rows) / len(rows) if rows else 0.0
    return 2 * p * (1 - p)


def tree(rows, depth, fields, min_size=2):
    p, best = sum(x["suspicious"] for x in rows) / len(rows), None
    if depth and len(rows) >= 2 * min_size and 0.0 < p < 1.0:
        for f in fields:
            values = sorted({x[f] for x in rows})
            for v in (values[1:] if len(values) < 10 else
                      [values[int(i * len(values) / 10)] for i in range(1, 10)]):
                left = [x for x in rows if x[f] < v]
                right = [x for x in rows if x[f] >= v]
                if min(len(left), len(right)) < min_size:
                    continue
                k = gini(rows) - (len(left) * gini(left) + len(right) * gini(right)) / len(rows)
                if best is None or k > best[0]:
                    best = (k, f, v, left, right)
    if best is None or best[0] <= 1e-9:
        return {"prediction": int(p > 0.5)}
    return {"field": best[1], "threshold": best[2],
            "left": tree(best[3], depth - 1, fields, min_size),
            "right": tree(best[4], depth - 1, fields, min_size)}


def predict(node, x):
    while "prediction" not in node:
        node = node["left"] if x[node["field"]] < node["threshold"] else node["right"]
    return node["prediction"]


def accuracy(node, rows):
    return sum(predict(node, x) == x["suspicious"] for x in rows) / len(rows)


def baseline(train, rows):
    c = int(sum(x["suspicious"] for x in train) * 2 > len(train))
    return sum(x["suspicious"] == c for x in rows) / len(rows)


def measure(fields, depth=6):
    m = tree(TRAIN, depth, fields)
    return accuracy(m, TRAIN), accuracy(m, TEST)

The six features to be generated represent five types. Ratio is the quotient of two columns: consumption per person separates the size of the household from consumption. Difference is the change between the last and the first of the raw readings; volatility carries only the span, not the direction. The interaction feature is the product of two columns. The threshold indicator comes from domain knowledge: the tariff tiers stop at 25 and 40 m³. The last one learns a number from the dataset — whether the subscriber is above the median of their own region — and that median is read from the training set only.

for x in DATA:
    o = x["readings"]
    x["per_person"] = round(x["mean_consumption"] / x["household"], 3)
    x["last_first_diff"] = round(o[-1] - o[0], 2)
    x["household_satisfaction"] = x["household"] * x["satisfaction"]
    x["threshold_25"] = int(x["mean_consumption"] > 25)
    x["threshold_40"] = int(x["mean_consumption"] > 40)

MEDIAN = {}
for b in REGION:
    v = sorted(x["mean_consumption"] for x in TRAIN if x["region_name"] == b[0])
    MEDIAN[b[0]] = v[len(v) // 2]
for x in DATA:
    x["region_above"] = int(x["mean_consumption"] > MEDIAN[x["region_name"]])

BEFORE = measure(BASE_FIELDS)
NEW = ["per_person", "last_first_diff", "region_above", "threshold_25",
       "household_satisfaction", "threshold_40"]
print(f"{'feature':<22} {'type':<12} {'train':>9} {'held_out':>9} {'contrib':>8}")
print(f"{'-- none --':<22} {'11 columns':<12} {BEFORE[0]:>9.4f} {BEFORE[1]:>9.4f}")
for a, t in zip(NEW, ["ratio", "difference", "learned", "threshold", "interaction",
                      "threshold"]):
    e, s = measure(BASE_FIELDS + [a])
    print(f"{a:<22} {t:<12} {e:>9.4f} {s:>9.4f} {s - BEFORE[1]:>+8.4f}")
print(f"\nbaseline {baseline(TRAIN, TEST):.4f}; above 25 m3 {sum(x['threshold_25'] for x in DATA)} "
      f"rows, above 40 m3 {sum(x['threshold_40'] for x in DATA)} rows")

feature                type             train  held_out  contrib
-- none --             11 columns      0.8466    0.7778
per_person             ratio           0.8519    0.7897  +0.0119
last_first_diff        difference      0.8571    0.7817  +0.0040
region_above           learned         0.8492    0.7817  +0.0040
threshold_25           threshold       0.8479    0.7619  -0.0159
household_satisfaction interaction     0.8492    0.7540  -0.0238
threshold_40           threshold       0.8466    0.7778  +0.0000

baseline 0.7579; above 25 m3 243 rows, above 40 m3 2 rows

The contribution column carries three signs at once. The largest gain comes from the ratio feature: 0.0119, that is, three subscribers in a 252-row set. This is a column that arithmetically carries no new information — mean_consumption and household are already in the set — but the tree makes its splits on a single column and cannot build the quotient of two columns with one split. What the ratio adds is not information but the splittable form of that information.

The difference feature and the indicator learned from the training set earn one row each: 0.0040. That is the correct classification of a single subscriber and is too small to be told apart from chance. A number like this cannot be reported as “there is a contribution”; at most it can be reported as “measured and could not be told apart from zero”.

There is a difference of kind between these two columns, and the second rule of the course passes exactly through it. last_first_diff is computed inside each row on its own; it sees no other row in the dataset and gives the same number before and after the split. region_above, on the other hand, rests on a number — the region median — and that number is read from the data. In the code this distinction sits on a single line: the median dictionary is built over TRAIN, not over DATA. Had the source been DATA, the values of the column would not have changed at all, the 252 rows in the test set would have contributed to determining their own thresholds, and a share of the measured contribution would have come from leakage rather than from learning. A generated feature is also a learned object, and writing where it learned is as obligatory as writing the number it contributed. This distinction will become the founding principle of the pipeline in the fifth lesson.

Two Separate Causes of Zero

The most instructive row of the table is the 40 m³ threshold: the contribution is exactly zero, and not even the training number has moved by a hair. The cause shows in the last row — there are only 2 rows above 40 m³ in the set. A near-constant column cannot produce two sides of at least two rows each in any split, so the tree never selects it. The column was added, it was measured, and the tree the model sees is exactly the same.

The 25 m³ threshold is a different kind of zero and in fact is not even zero, it is negative: a loss of 0.0159, four subscribers. This column separates 243 rows, so it is not constant. But the information it carries is already inside mean_consumption, and the tree finds its own threshold by searching the data. The hand-set threshold says nothing new; in return it competes as a split candidate at every node, and gets a split that looks slightly better on the training set selected, producing a worse tree on the held-out set.

The largest loss comes from the interaction feature: 0.0238, six subscribers. The product of household and satisfaction squeezes the two columns into one number, and its relationship with the label is weaker than that of the two columns themselves. When the tree selects the product it makes the separation coarser. This is the typical behavior of interactions that do not come from domain knowledge and are generated on a “might work” basis.

Three separate causes of zero-or-below must be told apart: the column cannot be split at all (40 m³), its information is already there (25 m³), or it carries the information by distorting it (the product). If all three are summarized as “no contribution”, what is lost in the table is which fix will work.

Contributions Do Not Add Up

The contributions measured one by one gave three positive values: 0.0119, 0.0040, 0.0040. Their sum is 0.0199. One needs to look at what happens when the columns are added cumulatively.

cumulative = list(BASE_FIELDS)
print(f"{'added':<22} {'columns':>7} {'train':>9} {'held_out':>9} {'contrib':>8}")
for a in NEW:
    cumulative.append(a)
    e, s = measure(cumulative)
    print(f"{a:<22} {len(cumulative):>7} {e:>9.4f} {s:>9.4f} {s - BEFORE[1]:>+8.4f}")
POSITIVE = ["per_person", "last_first_diff", "region_above"]
e, s = measure(BASE_FIELDS + POSITIVE[:2])
print(f"\nonly two contributing columns   {e:.4f} {s:.4f} {s - BEFORE[1]:+.4f}")
e, s = measure(BASE_FIELDS + POSITIVE)
print(f"only three contributing columns {e:.4f} {s:.4f} {s - BEFORE[1]:+.4f}")

added                  columns     train  held_out  contrib
per_person                  12    0.8519    0.7897  +0.0119
last_first_diff             13    0.8598    0.7897  +0.0119
region_above                14    0.8611    0.7778  +0.0000
threshold_25                15    0.8664    0.7698  -0.0079
household_satisfaction      16    0.8677    0.7698  -0.0079
threshold_40                17    0.8677    0.7698  -0.0079

only two contributing columns   0.8598 0.7897 +0.0119
only three contributing columns 0.8611 0.7778 +0.0000

If the two columns were independent of each other, their contributions would add up. They do not. Two columns that add 0.0119 and 0.0040 one by one add 0.0119 together; when the third is added the total contribution falls to 0.0000. Three features that measured positive one by one return, together, to the line before the decision.

The cause is that the tree is greedy: at each node it selects the split that looks best at that moment, and the selection is not undone. A new column can offer a better-looking split at the root node and send the rest of the tree down an entirely different path. The contribution is not the column’s own information but the effect of the column on the model’s search path, and this effect is not an additive quantity.

The two numbers must be read separately. The training number rises in one direction: from 0.8466 to 0.8677, each of the six columns pulls it a bit higher. The held-out number, on the other hand, peaks at twelve columns and falls, and at seventeen columns it stays below the line before the decision. The Overfitting and Underfitting lesson of the Introduction to Machine Learning course produced this divergence by increasing capacity; here the model is fixed and the only thing growing is the number of columns. Adding features is also increasing capacity.

The rule of the course takes concrete form in this table. Six columns were generated, all six with a reasonable justification, and when measured on the held-out set only one of them remained. Had it not been measured, all six would have stayed in the set and the model would have worked with seventeen columns at 0.7698, 0.0080 below the line before the decision — and moreover the table would have looked better because the training number rose to 0.8677.

Summary

• The ratio feature carries the held-out number from 0.7778 to 0.7897 (0.0119, three subscribers); the difference feature and the region indicator learned from the training set add 0.0040, that is, one subscriber each.
• Those with zero and negative contribution are written as well: the 40 m³ threshold is exactly 0.0000, the 25 m³ threshold loses 0.0159, the product of two columns loses 0.0238.
• The cause of the zero is separated: the column cannot be split because it is near-constant, its information is already in another column, or it carries the information by distorting it.
• Contributions do not add up: three columns that add 0.0119, 0.0040 and 0.0040 one by one add 0.0000 together, because the contribution is not the column’s information but its effect on the model’s search path.
• While the number of columns goes from 11 to 17, the training number rises in one direction from 0.8466 to 0.8677 and the held-out number peaks at 0.7897 and falls to 0.7698; adding features is also increasing capacity.

Next Step

The six features of this lesson were derived from numerical columns by arithmetic. In the subscriber table, however, there are two more non-numerical fields: the date the contract started and the short note left by the field team. Neither can be given to a model as it is; multiple numbers can be derived from each. How many numbers come out of a date, how many numbers come out of a short text, and what the total contribution of these numbers is — the next lesson measures it.