Feature Scaling

The previous lesson fixed the three defects in the raw record and took the distance-based method from 0.7540 to 0.7857. That number exceeds the baseline by only 0.0278 — seven subscribers. The cause is not something left uncleaned but something never asked: the columns were converted to the same unit but not to the same magnitude. Mean consumption varies between zero and forty-three, while household size varies between one and six and the region columns between zero and one. For a method that sums the squares of the differences, this silently gives the columns weights.

This lesson measures two scaling decisions. Standardization subtracts the column mean from each value and divides by its dispersion; the result is the standard score, and it was defined in the Data Collection and Preparation course of the Data Analytics curriculum. Normalization squeezes each column into a range so that its smallest value is zero and its largest is one; it shares a name with the text normalization of the Data Collection and Preparation course but does a different job: one unifies spelling, the other fits a number into a range. Neither is new as a procedure; what is measured is in which model family each one is worth how many subscribers.

• PS9. The starting table of this lesson is the output of the previous lesson: the fictional table of 1,260 subscribers with the three defects fixed, the same 756/252/252 split, a baseline of 0.7579 on the test set.
• PS10. Three model families are compared, and all three are written within the lesson: a distance-based method that looks at the nearest fifteen neighbors, a linear model that updates its weights in the direction of the gradient, and a depth 6 decision tree. Their settings are fixed for the whole course.
• PS11. The output of the gradient-based model is squeezed between zero and one by an S-shaped function; a subscriber above the threshold of 0.5 counts as suspicious. The threshold is a choice and does not change in this course.
• PS12. Scaling is applied only to the five numeric columns: mean consumption, volatility, household size, satisfaction, number of periods. Scaling the binary columns is measured at the end of the lesson as a separate decision.
• PS13. The scale is a fitted object and has two faces: fit, extracting the scale numbers from the training set; transform, applying those numbers to a set. The second rule of the course turns into a procedure here for the first time: every preparation step is fitted only on the training set and applied to the held-out set.
• PS14. A scale fitted on the whole set is a leak. The Introduction to Machine Learning course counted this as one of the four kinds of leakage; the job here is not to repeat those counts but to build the form that does not leak as a procedure.
• PS15. The measure is accuracy and the test set is 252 subscribers; one subscriber is worth 0.0040.
• PS16. Floating-point comparison is not done with ==. The out-of-range count uses 0.0 <= v <= 1.0; the boundary value itself counts as inside the range.

Three Models, Two Scaling Decisions

The setup produces the table and prints the raw ranges of the five numeric columns.

# scaling.py — MODEL. The fictional subscriber table of the Introduction to Machine Learning course is regenerated in the
# form cleaned in the previous lesson.
import math
import statistics

SEED, CANDIDATES, M32 = 20260218, 1400, 0xFFFFFFFF
REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10),
          ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05),
          ("central", 0.18, 23, 0.05)]


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


SUBSCRIBERS, READINGS = [], {}
for i in range(CANDIDATES):
    r = rng(SEED + i)
    b = REGION[pick(r(), [x[1] for x in REGION])]
    household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05])
    satisfaction = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17])
    if r() >= 0.046:
        SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household,
                            "satisfaction": satisfaction})
for k in SUBSCRIBERS:
    r = rng(SEED + 7000 + k["id"])
    for d in range(pick(r(), [0.05, 0.12, 0.21, 0.62])):
        READINGS.setdefault(k["id"], []).append(0.0 if r() < 0.038 else math.floor(
            k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62)
            * (1 - d * 0.05) * 100 + 0.5) / 100)
        r()

DATA = []
for k in SUBSCRIBERS:
    v = READINGS.get(k["id"])
    if v is None:
        continue
    mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"])
    z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol
         + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3]
         + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30)
    x = {"household": k["household"], "satisfaction": k["satisfaction"], "period": len(v),
         "mean_consumption": round(mean, 2), "volatility": round(vol, 2),
         "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35)}
    for b in REGION:
        x["b_" + b[0]] = int(k["region"][0] == b[0])
    DATA.append(x)


def shuffle(data, seed):
    r, s = rng(seed), list(range(len(data)))
    for i in range(len(s) - 1, 0, -1):
        j = int(r() * (i + 1))
        s[i], s[j] = s[j], s[i]
    return [data[i] for i in s]


K = shuffle(DATA, SEED + 90000)
TRAIN, TEST = K[:756], K[1008:]
BASELINE = sum(x["suspicious"] == 0 for x in TEST) / len(TEST)
NUMERIC = ["mean_consumption", "volatility", "household", "satisfaction", "period"]
FIELDS = NUMERIC + ["zero_readings", "b_north", "b_south", "b_east", "b_west", "b_central"]
print(f"train {len(TRAIN)}, test {len(TEST)}, baseline {BASELINE:.4f}")
print(f"{'column':<16} {'minimum':>9} {'maximum':>9} {'mean':>9}")
for a in NUMERIC:
    v = [x[a] for x in TRAIN]
    print(f"{a:<16} {min(v):>9.2f} {max(v):>9.2f} {statistics.fmean(v):>9.2f}")

train 756, test 252, baseline 0.7579
column             minimum   maximum      mean
mean_consumption      0.00     42.90     19.77
volatility            0.00     40.57      9.62
household             1.00      6.00      3.22
satisfaction          1.00      5.00      3.37
period                1.00      3.00      2.53

The ratio between the ranges reaches ninety times. Now the three models and the two scaling decisions are written. Splitting the scaling functions into fit and transform is not a matter of style; it is the rule of the course carried into code.

def gini(s):
    p = sum(x["suspicious"] for x in s) / len(s) if s else 0.0
    return 2 * p * (1 - p)


def tree(s, depth, fields, min_size=2):     # MODEL: decision tree, depth 6
    p = sum(x["suspicious"] for x in s) / len(s)
    best = None
    if depth and len(s) >= 2 * min_size and 0.0 < p < 1.0:
        for a in fields:
            d = sorted({x[a] for x in s})
            for v in (d[1:] if len(d) < 10 else
                      [d[int(i * len(d) / 10)] for i in range(1, 10)]):
                left = [x for x in s if x[a] < v]
                right = [x for x in s if x[a] >= v]
                if min(len(left), len(right)) < min_size:
                    continue
                k = gini(s) - (len(left) * gini(left) + len(right) * gini(right)) / len(s)
                if best is None or k > best[0]:
                    best = (k, a, v, left, right)
    if best is None or best[0] <= 1e-9:
        return {"prediction": int(p > 0.5)}
    return {"field": best[1], "threshold": best[2],
            "left": tree(best[3], depth - 1, fields, min_size),
            "right": tree(best[4], depth - 1, fields, min_size)}


def tree_predict(d, x):
    while "prediction" not in d:
        d = d["left"] if x[d["field"]] < d["threshold"] else d["right"]
    return d["prediction"]


def neighbor_predict(train, x, fields, k=15):  # MODEL: distance-based method
    u = sorted((sum((x[a] - y[a]) ** 2 for a in fields), y["suspicious"]) for y in train)[:k]
    return int(sum(t[1] for t in u) * 2 > k)


def squash(z):                              # S-shaped function with overflow guard
    if z >= 0:
        return 1.0 / (1.0 + math.exp(-z)) if z < 700 else 1.0
    e = math.exp(z) if z > -700 else 0.0
    return e / (1.0 + e)


def linear_model(train, fields, step=0.5, rounds=400):  # MODEL: gradient-based linear model
    w, b, n = [0.0] * len(fields), 0.0, len(train)
    for _ in range(rounds):
        gw, gb = [0.0] * len(fields), 0.0
        for x in train:
            h = squash(sum(w[i] * x[a] for i, a in enumerate(fields)) + b) - x["suspicious"]
            for i, a in enumerate(fields):
                gw[i] += h * x[a]
            gb += h
        for i in range(len(fields)):
            w[i] -= step * gw[i] / n
        b -= step * gb / n
    return w, b


def linear_predict(m, x, fields):
    return int(squash(sum(m[0][i] * x[a] for i, a in enumerate(fields)) + m[1]) > 0.5)


def three_models(train, test, fields=None):
    fields = fields or FIELDS
    m1, m3 = tree(train, 6, fields), linear_model(train, fields)
    return (sum(neighbor_predict(train, x, fields) == x["suspicious"] for x in test) / len(test),
            sum(linear_predict(m3, x, fields) == x["suspicious"] for x in test) / len(test),
            sum(tree_predict(m1, x) == x["suspicious"] for x in test) / len(test))


def fit_standard(train, columns):           # FIT: mean and dispersion
    return {a: (statistics.fmean(x[a] for x in train),
                statistics.pstdev([x[a] for x in train])) for a in columns}


def apply_standard(p, s):                   # APPLY
    return [dict(x, **{a: (x[a] - p[a][0]) / p[a][1] for a in p}) for x in s]


def fit_normal(train, columns):             # FIT: minimum and maximum
    return {a: (min(x[a] for x in train), max(x[a] for x in train)) for a in columns}


def apply_normal(p, s):                     # APPLY
    return [dict(x, **{a: (x[a] - p[a][0]) / (p[a][1] - p[a][0]) for a in p}) for x in s]


PS, PN = fit_standard(TRAIN, NUMERIC), fit_normal(TRAIN, NUMERIC)
CASES = [("unscaled (raw column)", TRAIN, TEST),
         ("standardization", apply_standard(PS, TRAIN), apply_standard(PS, TEST)),
         ("normalization", apply_normal(PN, TRAIN), apply_normal(PN, TEST))]
S = {}
print(f"{'decision':<30} {'distance':>8} {'linear':>13} {'tree':>8}")
print(f"{'baseline':<30} {BASELINE:>8.4f} {BASELINE:>13.4f} {BASELINE:>8.4f}")
for name, e, s in CASES:
    S[name] = three_models(e, s)
    print(f"{name:<30} {S[name][0]:>8.4f} {S[name][1]:>13.4f} {S[name][2]:>8.4f}")
raw = S["unscaled (raw column)"]
for name in ("standardization", "normalization"):
    k = [S[name][i] - raw[i] for i in range(3)]
    print(f"{name + ' contribution':<30} {k[0]:>+8.4f} {k[1]:>+13.4f} {k[2]:>+8.4f}")

decision                       distance        linear     tree
baseline                         0.7579        0.7579   0.7579
unscaled (raw column)            0.7857        0.7579   0.7778
standardization                  0.7698        0.8016   0.7778
normalization                    0.7619        0.7817   0.7778
standardization contribution    -0.0159       +0.0437  +0.0000
normalization contribution      -0.0238       +0.0238  +0.0000

The tree column gives the same number in all three cases, and this is not a small difference that has fallen below the measurement noise but an exact zero. The reason can be shown: in each column the tree looks only at the order of the values, picks its candidates from among the sorted distinct values and splits with a single threshold. Standardization and normalization are increasing transformations; they do not change the order, so the splits the tree can choose stay exactly the same. Putting a scaling step in front of a tree-based method does not change the result; its contribution is zero, and that is the definition of a step that is a habit.

The gradient-based linear model is at the opposite end. On the unscaled table it reads 0.7579 — exactly the baseline. This is not a good number but a symptom of collapse: the model gives every subscriber the same answer. While mean consumption goes up to forty-three, household size does not exceed six, so the same step size is too large for one column and too small for the other, and in four hundred rounds the weights cannot reach a usable place. Standardization adds 0.0437 to this model, that is, eleven subscribers.

The row of the distance-based method, on the other hand, breaks a common expectation. Standardization makes it lose 0.0159, normalization 0.0238. Scaling is not a step that helps distance-based methods; it is a step that gives the columns equal weight. In this table the raw scale had already given the widest ranges to the two most informative columns, mean consumption and volatility. Equalization takes that weight and distributes it to household size and the number of periods. The sign of the decision depends on the information the scale itself happens to carry, and this can be known only by measuring.

The Scale Is a Fitted Object

What the scaling step fits is a handful of numbers, and those numbers come only from the training set.

print(f"{'column':<16} {'mean':>9} {'dispersion':>12} "
      f"{'minimum':>9} {'maximum':>9}")
for a in NUMERIC:
    print(f"{a:<16} {PS[a][0]:>9.3f} {PS[a][1]:>12.3f} "
          f"{PN[a][0]:>9.2f} {PN[a][1]:>9.2f}")
sn = apply_normal(PN, TEST)
print(f"\nvalues outside the [0, 1] range in test "
      f"{sum(1 for x in sn for a in NUMERIC if not 0.0 <= x[a] <= 1.0)}, "
      f"subscribers {sum(1 for x in sn if any(not 0.0 <= x[a] <= 1.0 for a in NUMERIC))}")
ss = apply_standard(PS, TEST)
print(f"{'column':<16} {'lowest in test':>18} {'highest':>10} (standard score)")
for a in NUMERIC:
    v = [x[a] for x in ss]
    print(f"{a:<16} {min(v):>18.2f} {max(v):>10.2f}")
PT = fit_standard(TRAIN, FIELDS)
t = three_models(apply_standard(PT, TRAIN), apply_standard(PT, TEST))
print(f"\nall eleven columns standardized: distance {t[0]:.4f}, "
      f"linear {t[1]:.4f}, tree {t[2]:.4f}")

column                mean   dispersion   minimum   maximum
mean_consumption    19.767        6.200      0.00     42.90
volatility           9.622        8.120      0.00     40.57
household            3.224        1.270      1.00      6.00
satisfaction         3.372        1.147      1.00      5.00
period               2.532        0.715      1.00      3.00

values outside the [0, 1] range in test 2, subscribers 2
column               lowest in test    highest (standard score)
mean_consumption              -2.63       3.83
volatility                    -1.19       5.11
household                     -1.75       2.19
satisfaction                  -2.07       1.42
period                        -2.14       0.66

all eleven columns standardized: distance 0.7659, linear 0.8016, tree 0.7778

The fitted object is ten numbers, and all of them come from the training set of 756 subscribers. At the point of application there is a visible consequence: normalization is defined as “fit every value into the range between zero and one”, yet in the test set two values fall outside that range. This is not a bug and it is not fixed. The only way to fix it would be to choose the minimum and the maximum by looking at the test set as well; that would be fitting the scale on the held-out set, that is, leaking. Before the split, the Introduction to Machine Learning course had counted scaling as one of the four kinds of leakage; the job here is not to repeat those counts but to build the form that does not leak as an object. The standard score table shows the same thing from the other direction: in the volatility column the highest value in the test set is 5.11 standard scores, and the scale fitted on the training set lets it through without clipping.

The last line measures a third decision: which columns are scaled. When all eleven columns, binary ones included, are standardized instead of the five numeric columns, the gradient-based model stays at 0.8016, the tree stays at 0.7778, and the distance-based method drops from 0.7698 to 0.7659. A difference of one subscriber is not read on its own, but its direction can be explained: when a binary column is standardized, the rare category is carried to a large standard score, and since the region difference is in the distance sum, it weighs heavily. The scaling decision is not a single switch; which column it is applied to is also a decision, and that too is measured on the held-out set.

Summary

• Standardization adds 0.0437 to the gradient-based linear model, that is, eleven subscribers; the same decision gives −0.0159 on the distance-based method and exactly 0.0000 on the decision tree.
• The zero in the tree is not measurement noise but a provable result: standardization and normalization are increasing transformations, they do not change the order within a column, and the tree looks only at the order.
• On the unscaled table the gradient-based model reads exactly the baseline, 0.7579; this is not a good number but a symptom that a single step size is not enough for ranges ninety times apart.
• The scale is ten numbers fitted on the training set; applied to the test set, two values fall outside the [0, 1] range, and this is not fixed, because the only way to fix it would be to fit the scale on the held-out set.
• The choice of which columns to scale is a separate decision: when the binary columns are scaled too, the distance-based method drops from 0.7698 to 0.7659 and the other two models do not change.

Next Step

In this lesson the region information was used as five ready-made binary columns, and where those five columns came from was not asked. Yet the region is a text field, and turning it into a number is a decision: one column can be opened for each category, the categories can be given sequence numbers, or each category can be replaced with the suspicious rate in its own training set. The three have different costs; the first grows the number of columns by the number of categories, the third settles for a single column but puts the label inside the encoding. The next lesson measures these three encodings on a reading-route field of two hundred forty codes, counts the categories never seen in the held-out set, and prints how much of the inflation that target encoding produces on the training set recedes when it is computed inside the fold.