Baseline Model

The previous lesson turned the baseline into a question of its own for the second time. When the cost matrix changed, the baseline changed too: in one, flagging nobody, in the other, flagging everybody. Across the subgroups every group turned out to have its own baseline, which was 0.0000 in one group and 0.7882 in another. Over seven courses the baseline was a tool — a fixed mark that makes the model’s number readable and is not argued over again once it is chosen.

This lesson measures that it is not fixed. How many baselines a problem has, how one is chosen among them, in which two separate ways the choice lets the same model be read. Here the baseline is not a tool but a design object: a procedure with a capacity of its own, an overfitting of its own and a subgroup behavior of its own. At the end of the lesson the baselines actually chosen in the seven courses of the Machine Learning and Data Science curriculum are collected in a single table.

• ED16. The setup, the split (756/252/252), the seed, the metric (accuracy) and the subgroup definitions carry over unchanged from the previous two lessons: four axes, 13 subgroups, a group whose n is below 25 does not enter the gap.
• ED17. The model being compared is the depth-6 decision tree, and it gives 0.7778 on the test set. This number never changes during the lesson; the only thing that changes is the baseline placed opposite it.
• ED18. A baseline candidate is a procedure that, even if learned from the training set, does not look at all of the features the model uses and does not itself run a tuning search. The boundary is a decision.
• ED19. Eight candidates are tested: calling everyone suspicious, random assignment by class proportion, stating the most frequent class, a single-split rule, and four lookup tables — region; region and tier; those plus household; all four axes. A lookup table states the training majority in a cell, and the most frequent class in a cell it has not seen; the number of cells is written, because that is what is being measured.
• ED20. Random assignment is the mean of twenty runs; if a single run were printed the number would swing from run to run.
• ED21. No choice among the candidates is made in this lesson. What is measured is not the result of the choice but how far the choice can move the same model.
• ED22. Every number in the course table of the last section was read from its source lesson; the units are not shared across courses and the rows are not added to one another.

How Many Baselines a Problem Has

The first block sets up the scenario, trains the model and measures the eight baseline candidates on the same test set.

# MODEL: the measurement network of the Data Analytics curriculum is a FICTIONAL setup generated with the
# same seed; readings are summarized at subscriber level and a FICTIONAL illicit-use/fault suspicion label is added on top.
import math

SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF
REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10),
          ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05),
          ("central", 0.18, 23, 0.05)]


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


SUBSCRIBERS, READINGS, DATA = [], {}, []
for i in range(RAW):
    r = rng(SEED + i)
    b = REGION[pick(r(), [x[1] for x in REGION])]
    household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05])
    rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17])
    if r() >= 0.046:
        SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household,
                            "satisfaction": rating})
for k in SUBSCRIBERS:
    r = rng(SEED + 7000 + k["id"])
    for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])):
        READINGS.setdefault(k["id"], []).append(0.0 if r() < 0.038 else math.floor(
            k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62)
            * (1 - n * 0.05) * 100 + 0.5) / 100)
        r()
    v = READINGS.get(k["id"])
    if v is None:
        continue
    mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"])
    z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol
         + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3]
         + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30)
    x = {"id": k["id"], "household": k["household"], "satisfaction": k["satisfaction"],
         "period": len(v), "mean_consumption": round(mean, 2), "volatility": round(vol, 2),
         "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35),
         "region_name": k["region"][0]}
    for b in REGION:
        x["area_" + b[0]] = int(k["region"][0] == b[0])
    DATA.append(x)

FIELDS = ["mean_consumption", "volatility", "household", "satisfaction", "period",
          "zero_readings", "area_north", "area_south", "area_east", "area_west", "area_central"]


def shuffle(data, seed):
    r, s = rng(seed), list(range(len(data)))
    for i in range(len(s) - 1, 0, -1):
        j = int(r() * (i + 1))
        s[i], s[j] = s[j], s[i]
    return [data[i] for i in s]


def three_way_split(rows):
    a, b = int(len(rows) * 0.6), int(len(rows) * 0.8)
    return rows[:a], rows[a:b], rows[b:]


def impurity(rows):
    p = sum(x["suspicious"] for x in rows) / len(rows) if rows else 0.0
    return 2 * p * (1 - p)


def tree(rows, depth, fields, min_size=2):
    p, best = sum(x["suspicious"] for x in rows) / len(rows), None
    if depth and len(rows) >= 2 * min_size and 0.0 < p < 1.0:
        for f in fields:
            values = sorted({x[f] for x in rows})
            for v in (values[1:] if len(values) < 10 else
                      [values[int(i * len(values) / 10)] for i in range(1, 10)]):
                left = [x for x in rows if x[f] < v]
                right = [x for x in rows if x[f] >= v]
                if min(len(left), len(right)) < min_size:
                    continue
                k = impurity(rows) - (len(left) * impurity(left) + len(right) * impurity(right)) / len(rows)
                if best is None or k > best[0]:
                    best = (k, f, v, left, right)
    if best is None or best[0] <= 1e-9:
        return {"prediction": int(p > 0.5)}
    return {"field": best[1], "threshold": best[2], "left": tree(best[3], depth - 1, fields),
            "right": tree(best[4], depth - 1, fields)}


def predict(node, x):
    while "prediction" not in node:
        node = node["left"] if x[node["field"]] < node["threshold"] else node["right"]
    return node["prediction"]


def accuracy(node, rows):
    return sum(predict(node, x) == x["suspicious"] for x in rows) / len(rows)


def subgroups(x):                  # four axes, thirteen subgroups; carried over unchanged from lesson 01
    return [("region", x["region_name"]),
            ("tariff", "10" if x["mean_consumption"] <= 10 else
             ("25" if x["mean_consumption"] <= 25 else "40")),
            ("household", "1-2" if x["household"] <= 2 else ("3-4" if x["household"] <= 4 else "5+")),
            ("reading", "few" if x["period"] <= 2 else "full")]


SUBGROUPS = sorted({g for x in DATA for g in subgroups(x)})
TRAIN, VAL, TEST = three_way_split(shuffle(DATA, SEED + 90000))
M, STUMP = tree(TRAIN, 6, FIELDS), tree(TRAIN, 1, FIELDS)
RATE = sum(x["suspicious"] for x in TRAIN) / len(TRAIN)
MAJORITY = int(RATE > 0.5)


def groups_of(x):
    return dict(subgroups(x))


def lookup(key_fn):                # lookup-table baseline: the training majority in the cell
    counts = {}
    for x in TRAIN:
        counts.setdefault(key_fn(x), [0, 0])[x["suspicious"]] += 1
    votes = {c: int(v[1] > v[0]) for c, v in counts.items()}
    return lambda x: votes.get(key_fn(x), MAJORITY)


def random_assign(rows, n=20):     # assignment by class proportion, the mean of twenty runs
    t = 0.0
    for j in range(n):
        r = rng(SEED + 61000 + j)
        t += sum((1 if r() < RATE else 0) == x["suspicious"] for x in rows) / len(rows)
    return t / n


def hit_rate(f, rows):
    return sum(f(x) == x["suspicious"] for x in rows) / len(rows)


KEYS = (("region", lambda x: x["region_name"]),
        ("region+tier", lambda x: (x["region_name"], groups_of(x)["tariff"])),
        ("+household", lambda x: (x["region_name"], groups_of(x)["tariff"],
                                  groups_of(x)["household"])),
        ("four axes", lambda x: (x["region_name"], groups_of(x)["tariff"],
                                 groups_of(x)["household"], groups_of(x)["reading"])))
CANDIDATES = [("all suspicious", 1, lambda x: 1), ("most frequent class", 1, lambda x: MAJORITY),
              ("single rule", 2, lambda x: predict(STUMP, x))]
CANDIDATES += [(name, len({k(x) for x in TRAIN}), lookup(k)) for name, k in KEYS]

model_acc = accuracy(M, TEST)
print(f"model, depth-6 decision tree: train {accuracy(M, TRAIN):.4f} test {model_acc:.4f}\n\n"
      f"{'baseline candidate':<19} {'cells':>5} {'train':>7} {'test':>7} {'model diff':>12}")
print(f"{'random assignment':<19} {'--':>5} {random_assign(TRAIN):>7.4f} {random_assign(TEST):>7.4f} "
      f"{model_acc - random_assign(TEST):>+12.4f}")
diffs = [model_acc - random_assign(TEST)]
for name, n, f in CANDIDATES:
    diffs.append(model_acc - hit_rate(f, TEST))
    print(f"{name:<19} {n:>5} {hit_rate(f, TRAIN):>7.4f} {hit_rate(f, TEST):>7.4f} {diffs[-1]:>+12.4f}")
print(f"\nthe baseline choice makes the same model read between {min(diffs):+.4f} and {max(diffs):+.4f}:"
      f" range {max(diffs) - min(diffs):.4f}")

model, depth-6 decision tree: train 0.8466 test 0.7778

baseline candidate  cells   train    test   model diff
random assignment      --  0.6522  0.6450      +0.1327
all suspicious          1  0.2222  0.2421      +0.5357
most frequent class     1  0.7778  0.7579      +0.0198
single rule             2  0.7778  0.7579      +0.0198
region                  5  0.7778  0.7579      +0.0198
region+tier            15  0.7897  0.7897      -0.0119
+household             39  0.7976  0.7500      +0.0278
four axes              66  0.8095  0.7738      +0.0040

the baseline choice makes the same model read between -0.0119 and +0.5357: range 0.5476

The last column is eight separate readings of a single model. The model did not change at all: 0.7778 on the test set. But against the “call everyone suspicious” baseline the model’s contribution is +0.5357, against the “region and tier lookup table” baseline it is −0.0119. The 0.5476 between them is a number produced only by the choice of baseline, not by the data or the model. The same piece of work can be told in two separate reports, and both are computed correctly.

The three middle rows give the second result. The most frequent class, a single-split rule and a lookup table that looks at the region produce the same number: 0.7579. Because the suspicious share does not exceed half in any region, all five cells of the region table say the same answer. Making a baseline more complicated does not make it stronger; it gets stronger only if the majority changes in the cells.

The main finding is two rows further down. When region and tariff tier are taken together, the majority flips in some of the fifteen cells and the table reads 0.7897 — above the depth-6 decision tree. A tree trained on eleven features, which tried hundreds of candidate splits, loses to a lookup table with two columns. When household is added the cell count rises to 39: the train number climbs to 0.7976 while the test number falls to 0.7500. With all four axes there are 66 cells, train 0.8095, test 0.7738. A baseline has a capacity too, and when that capacity is exceeded the baseline overfits as well. This is what lifts the baseline out of being a tool and turns it into a design object: when it is chosen, the questions asked when choosing a model are asked.

The Choice of Baseline Changes Direction in the Subgroups

The overall difference between the two baselines is only 0.0318. Once we go down to the subgroups, the two do not say the same thing.

REGION_TIER = lookup(lambda x: (x["region_name"], groups_of(x)["tariff"]))
print(f"{'subgroup':<14} {'n':>4} {'model':>7} {'A baseline':>10} {'A diff':>8} "
      f"{'B baseline':>10} {'B diff':>8}")
diffs_a, diffs_b = [], []
for g in SUBGROUPS:
    rows = [x for x in TEST if g in subgroups(x)]
    base_a, base_b, acc_g = hit_rate(lambda x: MAJORITY, rows), hit_rate(REGION_TIER, rows), accuracy(M, rows)
    if len(rows) >= 25:
        diffs_a.append(acc_g - base_a)
        diffs_b.append(acc_g - base_b)
    print(f"{' '.join(g):<14} {len(rows):>4} {acc_g:>7.4f} {base_a:>10.4f} {acc_g - base_a:>+8.4f} "
          f"{base_b:>10.4f} {acc_g - base_b:>+8.4f}{'  n<25' if len(rows) < 25 else ''}")
print(f"\nA most frequent class: diff {min(diffs_a):+.4f} .. {max(diffs_a):+.4f}, gap "
      f"{max(diffs_a) - min(diffs_a):.4f}, below baseline {sum(1 for v in diffs_a if v < 0)}/{len(diffs_a)}")
print(f"B region+tier        : diff {min(diffs_b):+.4f} .. {max(diffs_b):+.4f}, gap "
      f"{max(diffs_b) - min(diffs_b):.4f}, below baseline {sum(1 for v in diffs_b if v < 0)}/{len(diffs_b)}")

subgroup          n   model A baseline   A diff B baseline   B diff
household 1-2    73  0.7945     0.7534  +0.0411     0.8219  -0.0274
household 3-4   143  0.7762     0.7692  +0.0070     0.7832  -0.0070
household 5+     36  0.7500     0.7222  +0.0278     0.7500  +0.0000
reading few      85  0.7412     0.7882  -0.0471     0.7647  -0.0235
reading full    167  0.7964     0.7425  +0.0539     0.8024  -0.0060
region central   58  0.6724     0.6552  +0.0172     0.6552  +0.0172
region east      36  0.6111     0.4722  +0.1389     0.6944  -0.0833
region north     68  0.7794     0.8235  -0.0441     0.8235  -0.0441
region south     58  0.8793     0.8276  +0.0517     0.8276  +0.0517
region west      32  0.9688     1.0000  -0.0312     1.0000  -0.0312
tariff 10        12  0.5833     0.6667  -0.0833     0.6667  -0.0833  n<25
tariff 25       193  0.8290     0.8394  -0.0104     0.8394  -0.0104
tariff 40        47  0.6170     0.4468  +0.1702     0.6170  +0.0000

A most frequent class: diff -0.0471 .. +0.1702, gap 0.2173, below baseline 4/12
B region+tier        : diff -0.0833 .. +0.0517, gap 0.1351, below baseline 8/12

The east region is where the two baselines part. Against the most frequent class the model beats the baseline there by 0.1389 and gives one of the best contributions of the course; against the region and tier table it stays 0.0833 below the baseline. The same happens in the tariff 40 tier: 0.0000 instead of +0.1702. Both numbers are correct. The difference is that the first baseline is unaware of the east region’s high suspicion rate, while the second already knows that rate. A model can say what it adds in a subgroup only when it is compared with a baseline that already knows that subgroup.

The two summary lines give the result. Against the most-frequent-class baseline the model is below the baseline in four of the twelve subgroups; against the lookup-table baseline, in eight. The difference gap also falls from 0.2173 to 0.1351. A strong baseline both shrinks the model’s contribution and flattens it across subgroups; this is not a flaw but the consequence of the baseline already taking the subgroup into account. Which baseline is reported is therefore the decision that determines what the study claims.

Baselines Chosen in the Seven Courses of Machine Learning and Data Science

Over seven courses every lesson chose a baseline and wrote down its reason. The table below collects those choices from the source lessons. The units are not shared and the rows are not added to one another.

Two things can be read from the table. First, the baseline changes type from course to course: stating a class, stating a mean, behaving randomly, recommending the most chosen, or putting in the state where a decision was not applied. Second, the rows of the Deep Learning course and of the Natural Language Processing and Computer Vision course show that even within one course there is not a single baseline: four separate datasets, four separate baselines. Whenever the sentence “it beat the baseline” was written, what was beaten was a different procedure each time, and the table of this lesson gives the size of that difference.

Summary

• The baseline is not a tool but a design object. The same depth-6 decision tree reads between −0.0119 and +0.5357 against eight baseline candidates; the choice alone moves the number by 0.5476.
• Making a baseline more complicated does not make it stronger: the most frequent class, the single-split rule and the region lookup table give the same 0.7579, because the majority flips in no region.
• The baseline has a capacity too. The lookup table beats the model with 0.7897 in 15 cells, and in 39 cells it climbs to 0.7976 on train while falling to 0.7500 on test: the baseline overfits as well.
• The choice of baseline changes direction in the subgroups. In the east region the model is +0.1389 against one baseline and −0.0833 against the other. The model that is below the baseline in 4 of 12 subgroups against the most frequent class is below it in 8 against the lookup table; the difference gap falls from 0.2173 to 0.1351.
• In the seven courses of the Machine Learning and Data Science curriculum the baseline changed type, and even within a single course more than one baseline was used; in the sentence “it beat the baseline” what is beaten is a different procedure each time.

Next Step

Over three lessons fourteen separate number pairs were compared, and in all of them the difference was read directly: 0.0198, −0.0119, +0.1702. What was never asked in any of them is whether these differences can be told apart from the noise of a 252-subscriber set. A contribution of 0.1389 read from 36 subscribers in one subgroup could vanish if the class of a few subscribers flipped. Hypothesis testing, permutation testing and the bootstrap were measured in the Statistical Analysis and Exploratory Data Analysis course and will not be repeated. The next lesson measures only this: how large the margin of the difference between two models is, how much narrower a margin the paired comparison measured on the same samples gives than the independent comparison, and how the size of the held-out set determines that margin.