Application Areas

In three lessons a single question was asked: is this subscriber suspicious. Because the question is binary, the baseline also stayed the same throughout — telling the most frequent class. The same table can carry other questions too. How many cubic meters will this subscriber consume next period. Into how many natural groups do the subscribers split. Which hundred subscribers should the team going out to the field today visit, and in what order.

This lesson asks the same data four questions. The only thing that does not change during the lesson is the table; what changes is the question, the measure of the question and the baseline of the question. All four are measured on the same held-out set.

• FC20. A dataset does not define a single problem. What determines the problem is not the data but the question asked; which column is the label in the same table changes together with the question.
• FC21. Classification: the output is a value from a finite set of labels. Its baseline is the most frequent class, because it is the best single decision that can be given without looking at anything.
• FC22. Regression: the output is a numerical value. Its baseline is the overall mean; if the mean absolute error is measured, the median is a better baseline, and the choice is written down.
• FC23. Clustering: there is no label, the output is a grouping. Its baseline is random grouping, because it produces the same number of groups and uses no structure.
• FC24. Ranking: the output is not a single decision but an order, and the measure looks at the budget at the head of the order. Its baseline is the random order; a non-learning heuristic order is put next to it too, because that is the procedure actually used in the field.
• FC25. Of the four measures, a large number is good in two and a small number is good in two. The numbers of the models can be compared not with one another but only with their own baselines.

Rebuilding the Fictional Dataset

# setup.py — the fictional dataset of lesson 01 is rebuilt with the same seed: the readings
# table, the subscriber summary, the fictional label, the train / held-out split and the tree learner.
import math

SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF
REGION = [("north", 0.28, 21), ("south", 0.22, 17), ("east", 0.18, 26),
          ("west", 0.14, 14), ("central", 0.18, 23)]
FEATURES = ["mean", "per_person", "drop", "zero_readings", "reading_count",
            "satisfaction", "household_size", "highest", "lowest"]


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


_subscribers, _dropped, _series = [], [], {}
for i in range(RAW):
    r = rng(SEED + i)
    b = REGION[pick(r(), [x[1] for x in REGION])]
    k = {"subscriber_id": 10001 + i, "region": b[0], "base": b[2],
         "household_size": pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]),
         "satisfaction": pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) + 1}
    (_dropped if r() < 0.046 else _subscribers).append(k)
_valid = {a["subscriber_id"] for a in _subscribers}
for k in _subscribers + _dropped:
    r = rng(SEED + 7000 + k["subscriber_id"])
    for d in range(pick(r(), [0.05, 0.12, 0.21, 0.62])):
        t = 0.0 if r() < 0.038 else math.floor(k["base"] * math.exp(
            (r() + r() + r() - 1.5) * 0.62) * (1 - d * 0.05) * 100 + 0.5) / 100
        if k["subscriber_id"] in _valid:
            _series.setdefault(k["subscriber_id"], []).append(t)
        r()

SUBSCRIBERS = []
for k in _subscribers:
    t = _series.get(k["subscriber_id"])
    if not t:
        continue
    mean = sum(t) / len(t)
    a = {"subscriber_id": k["subscriber_id"], "region": k["region"], "series": t,
         "household_size": k["household_size"], "satisfaction": k["satisfaction"],
         "reading_count": len(t), "mean": mean, "highest": max(t),
         "lowest": min(t), "zero_readings": sum(1 for v in t if v == 0),
         "per_person": mean / max(1, k["household_size"]),
         "drop": (t[0] - t[-1]) / t[0] if len(t) > 1 and t[0] > 0 else 0.0}
    r = rng(SEED + 31000 + a["subscriber_id"])
    a["suspicion"] = 1 if (1.30 * a["zero_readings"] + 1.25 * (a["drop"] > 0.45)
                           + 1.05 * (a["per_person"] > 12.0) + 0.65 * (a["reading_count"] < 3)
                           + 0.45 * (a["satisfaction"] <= 2) - 1.55
                           + (r() + r() - 1.0) * 1.1) > 0 else 0
    SUBSCRIBERS.append(a)

r = rng(SEED + 90001)
TRAIN, HELD_OUT = [], []
for a in SUBSCRIBERS:
    (TRAIN if r() < 0.70 else HELD_OUT).append(a)
positives = lambda k: sum(a["suspicion"] for a in k)
accuracy = lambda f, k: sum(1 for a in k if f(a) == a["suspicion"]) / len(k)
_impurity = lambda k: 2 * (positives(k) / len(k)) * (1 - positives(k) / len(k)) if k else 0.0


def fit(k, depth=3, min_size=40):
    """The greedy decision tree of lesson 01: the split criterion and stopping rule are there."""
    best, choice = _impurity(k) * len(k), None
    if depth and len(k) >= min_size:
        for feat in FEATURES:
            for threshold in sorted({a[feat] for a in k})[1:]:
                left = [a for a in k if a[feat] < threshold]
                right = [a for a in k if a[feat] >= threshold]
                m = _impurity(left) * len(left) + _impurity(right) * len(right)
                if m < best - 1e-9:
                    best, choice = m, (feat, threshold, left, right)
    if choice is None:
        return 1 if 2 * positives(k) > len(k) else 0
    lt, rt = fit(choice[2], depth - 1, min_size), fit(choice[3], depth - 1, min_size)
    return lt if lt == rt else (choice[0], choice[1], lt, rt)


def apply(d, a):
    while isinstance(d, tuple):
        d = d[2] if a[d[0]] < d[1] else d[3]
    return d


print(f"fictional dataset (seed {SEED}): {len(SUBSCRIBERS)} subscribers, {len(TRAIN)} train, "
      f"{len(HELD_OUT)} held-out, {positives(SUBSCRIBERS)} suspicious")

fictional dataset (seed 20260218): 1260 subscribers, 883 train, 377 held-out, 327 suspicious

A Binary Decision and a Numerical Value

The first two questions are labeled and differ from each other only in the type of the output. The first asks for a class: is this subscriber suspicious. The second asks for a number: how many cubic meters does this subscriber consume on average. The second question is not for the field team but for network planning, and it must be answerable before the reading arrives; this is why only the columns that come from the subscriber record, together with the region, are given to the regression model.

The baselines are chosen separately because the measures are separate. For accuracy, the most frequent class; for the mean absolute error, the overall mean. The second choice is debatable: if the mean absolute error is measured, the best single non-learning number is not the mean but the median. Here the mean was chosen, because squared error will also be used in later courses and a single baseline will be kept across the two measures; because the choice is debatable, it is written down.

# two_problems.py — two questions to the same table: a binary decision and a numerical value.
# The baseline of each question is chosen separately; the most frequent class for the binary
# question, the overall mean for the numerical one, because the measures are different.
import statistics
from collections import Counter

RESULTS = []
HEADER = ["problem", "measure", "base train", "base held-out", "model train", "model held-out"]
WIDTHS = [16, 40, 15, 15, 15, 15]
fmt = lambda h: str(h[0]).ljust(WIDTHS[0]) + str(h[1]).ljust(WIDTHS[1]) + "".join(
    str(v).rjust(WIDTHS[i + 2]) for i, v in enumerate(h[2:]))


def record(name, measure, bt, bh, mt, mh, form="{:.4f}"):
    RESULTS.append([name, measure] + [form.format(v) for v in (bt, bh, mt, mh)])
    print(fmt(RESULTS[-1]))


most_frequent = Counter(a["suspicion"] for a in TRAIN).most_common(1)[0][0]
TREE = fit(TRAIN)
REGION_NAMES = [b[0] for b in REGION]


def regression_inputs(a):
    return ([1.0, float(a["household_size"]), float(a["satisfaction"]),
             float(a["reading_count"])]
            + [1.0 if a["region"] == b else 0.0 for b in REGION_NAMES[1:]])


def solve(A, b):
    n = len(b)
    M = [A[i][:] + [b[i]] for i in range(n)]
    for i in range(n):
        p = max(range(i, n), key=lambda s: abs(M[s][i]))
        M[i], M[p] = M[p], M[i]
        if abs(M[i][i]) < 1e-12:
            continue
        for s in range(n):
            if s != i and M[s][i]:
                f = M[s][i] / M[i][i]
                for c in range(i, n + 1):
                    M[s][c] -= f * M[i][c]
    return [M[i][n] / M[i][i] if abs(M[i][i]) > 1e-12 else 0.0 for i in range(n)]


k = len(regression_inputs(TRAIN[0]))
X, t = [regression_inputs(a) for a in TRAIN], [a["mean"] for a in TRAIN]
WEIGHTS = solve([[sum(x[i] * x[j] for x in X) for j in range(k)] for i in range(k)],
                [sum(x[i] * v for x, v in zip(X, t)) for i in range(k)])
OVERALL = statistics.fmean(t)
predict_value = lambda a: sum(c * v for c, v in zip(WEIGHTS, regression_inputs(a)))
error = lambda f, s: statistics.fmean(abs(f(a) - a["mean"]) for a in s)
print(fmt(HEADER))
record("classification", "accuracy (high is good)",
       accuracy(lambda a: most_frequent, TRAIN), accuracy(lambda a: most_frequent, HELD_OUT),
       accuracy(lambda a: apply(TREE, a), TRAIN),
       accuracy(lambda a: apply(TREE, a), HELD_OUT))
record("regression", "mean absolute error m3 (low is good)", error(lambda a: OVERALL, TRAIN),
       error(lambda a: OVERALL, HELD_OUT), error(predict_value, TRAIN), error(predict_value, HELD_OUT))

problem         measure                                      base train  base held-out    model train model held-out
classification  accuracy (high is good)                          0.7350         0.7533         0.8709         0.8594
regression      mean absolute error m3 (low is good)             4.8767         5.2255         3.7226         4.1275

The two rows are read in two different directions. In classification the model is above the baseline by 0.1061; in regression it is below it by 1.0980 m³, and that is what is good. The magnitudes of the numbers cannot be compared either: 0.8594 is unitless, 4.1275 is cubic meters.

Unlabeled Grouping and an Order

The third question uses no label at all: into how many natural groups do the subscribers split. The measure is the mean distance of each subscriber to the center of its own group; a small value shows that the groups are tight. The non-learning procedure here distributes the subscribers into four random groups and computes the center of each group from its own members; in this way the same number of groups is produced but no structure is used.

The fourth question asks for an order. The daily capacity of the field team is a hundred visits, and the measure is how many illicit cases are found in the first hundred of the order. Two baselines are written here. The random order is the non-learning procedure itself; the order starting from the highest mean consumption is the heuristic procedure actually used in the field, and the model has to beat that too.

# two_more_problems.py — the third question is unlabeled, the fourth asks for an order, not
# a number. The baseline of the third is random grouping, that of the fourth a random order.
MEANS = {o: statistics.fmean(a[o] for a in TRAIN) for o in FEATURES}
STDS = {o: statistics.pstdev(a[o] for a in TRAIN) or 1.0 for o in FEATURES}
vec = lambda a: [(a[o] - MEANS[o]) / STDS[o] for o in FEATURES]
dist = lambda v, m: math.sqrt(sum((x - y) ** 2 for x, y in zip(v, m)))
within_dist = lambda s, M: statistics.fmean(min(dist(vec(a), m) for m in M) for a in s)

K = 4
CENTERS = [vec(a) for a in TRAIN[:K]]
for _ in range(25):
    groups = [[] for _ in range(K)]
    for a in TRAIN:
        groups[min(range(K), key=lambda i: dist(vec(a), CENTERS[i]))].append(vec(a))
    for i in range(K):
        if groups[i]:
            CENTERS[i] = [sum(c) / len(c) for c in zip(*groups[i])]
r = rng(SEED + 6104)
random_groups = [[] for _ in range(K)]
for a in TRAIN:
    random_groups[int(r() * K)].append(vec(a))
RANDOM_CENTERS = [[sum(c) / len(c) for c in zip(*g)] for g in random_groups]

TOP = 100
r = rng(SEED + 6205)
ENSEMBLE = [fit([TRAIN[int(r() * len(TRAIN))] for _ in TRAIN]) for _ in range(9)]
score = lambda a: sum(apply(d, a) for d in ENSEMBLE)
r = rng(SEED + 6206)
lots = {a["subscriber_id"]: r() for a in SUBSCRIBERS}
found_in_top = lambda f, s: positives(sorted(s, key=f)[:TOP])

print(fmt(HEADER))
record("clustering", "within-cluster distance (low is good)", within_dist(TRAIN, RANDOM_CENTERS),
       within_dist(HELD_OUT, RANDOM_CENTERS), within_dist(TRAIN, CENTERS),
       within_dist(HELD_OUT, CENTERS))
record("ranking", f"illicit cases in top {TOP} (high is good)",
       found_in_top(lambda a: lots[a["subscriber_id"]], TRAIN),
       found_in_top(lambda a: lots[a["subscriber_id"]], HELD_OUT),
       found_in_top(lambda a: (-score(a), a["subscriber_id"]), TRAIN),
       found_in_top(lambda a: (-score(a), a["subscriber_id"]), HELD_OUT), "{:.0f}")
print(f"\nsecond baseline in ranking — start from the highest mean consumption: "
      f"train {found_in_top(lambda a: (-a['mean'], a['subscriber_id']), TRAIN)}, "
      f"held-out {found_in_top(lambda a: (-a['mean'], a['subscriber_id']), HELD_OUT)}")
print(f"for comparison: train {positives(TRAIN)} suspicious / {len(TRAIN)} subscribers, "
      f"held-out {positives(HELD_OUT)} suspicious / {len(HELD_OUT)} subscribers")

problem         measure                                      base train  base held-out    model train model held-out
clustering      within-cluster distance (low is good)            2.7237         2.7683         2.1678         2.1551
ranking         illicit cases in top 100 (high is good)              29             20             86             71

second baseline in ranking — start from the highest mean consumption: train 38, held-out 30
for comparison: train 234 suspicious / 883 subscribers, held-out 93 suspicious / 377 subscribers

In clustering the four learned centers bring the distance of 2.7683 of the random four groups down to 2.1551; the gain is 0.6132. A warning stands beside this number: the measure of clustering never looks at the label, which means a “good” clustering is not a clustering that finds illicit cases. In the previous lesson it was measured separately how close a six-cluster split came to the baseline on the illicit-use question; these two numbers are not two measures of the same thing but two separate measures of two separate problems.

In ranking three numbers stand side by side. The random order finds 20 illicit cases in the first hundred visits of the held-out set — which is what is expected, given that 93 of the 377 subscribers are suspicious. The heuristic order starting from the highest consumption finds 30; domain knowledge beats random by 10 illicit cases. The order of the ensemble of nine trees finds 71. The model is 41 illicit cases above the heuristic and 51 above random, and depending on which baseline is written, the gain of the model is 41 or 51. Choosing the baseline is a measurement decision.

Four Problems, Four Baselines

# four_baselines.py — four problems, four separate baselines, four separate measures and two
# separate directions.
print(fmt(HEADER))
for s in RESULTS:
    print(fmt(s))

problem         measure                                      base train  base held-out    model train model held-out
classification  accuracy (high is good)                          0.7350         0.7533         0.8709         0.8594
regression      mean absolute error m3 (low is good)             4.8767         5.2255         3.7226         4.1275
clustering      within-cluster distance (low is good)            2.7237         2.7683         2.1678         2.1551
ranking         illicit cases in top 100 (high is good)              29             20             86             71

The table came out of a single dataset and gave four separate rows. The model columns cannot be compared with one another: putting 0.8594 and 4.1275 side by side and counting one of them as larger means nothing, because one is a ratio and the other is cubic meters, and in one a large number is good while in the other it is bad. The only comparable thing is the difference between the two columns within each row.

The baseline columns carry information too. In classification the baseline is already 0.7533; the largest share the model could gain is 0.2467, and 0.1061 is 43.0 percent of that. In ranking the baseline is only 20 and the ceiling is 93; with 71 the model has closed 69.9 percent of the gap. The same model, the same data, the same day — but two separate success stories on two questions.

Summary

• A dataset does not define a single problem; the same fictional subscriber table carries all four of the classification, regression, clustering and ranking questions.
• The four questions have four separate baselines: the most frequent class 0.7533, the overall mean 5.2255 m³, a random four-way grouping 2.7683 within-cluster distance, a random order 20 illicit cases in the first 100 visits.
• The held-out numbers of the models are 0.8594, 4.1275 m³, 2.1551 and 71 illicit cases respectively; a large number is good in two and a small number is good in two.
• A second baseline was written in ranking: the heuristic order starting from the highest consumption finds 30 illicit cases, so the gain of the model is 41 or 51 illicit cases depending on the baseline written.
• The numbers of the models cannot be compared with one another; the only comparable thing is the distance of each row to its own baseline.

Next Step

All four problems were read from a single point: a ready table, a ready feature set, a ready split and a single measurement. These lessons showed how a model is built, not how a job runs. In reality the problem definition changes after measurement, the feature set widens after evaluation, the threshold is moved after deployment, and the flow never runs once from start to finish. The next lesson sets up this flow as seven steps and what it measures is not the steps themselves but the returns: which step returns to which step is counted, and how much each return moved the held-out figure is printed.