Metric Selection

The previous lesson took the metric to be accuracy and never questioned it. Accuracy counts the two error types with the same weight: sending a subscriber who is not suspicious to inspection and missing a suspicious subscriber are equally wrong. In operations they are not equal. An inspection crew’s travel, its time and the mark a wasted visit leaves on the subscriber are one cost; an illicit use that slipped through or a fault left unrepaired is another cost, and the ratio of the two is a business decision.

This lesson writes that ratio as a cost matrix and measures one thing: the same model, the same scores, two separate cost matrices — how many separate operating points, how many separate decisions. Metric definitions, the threshold and the curves were set up in the Evaluation and Tuning topic of the Supervised Learning course and are not repeated here. The only thing added is reading the threshold from the business objective. The subgroups and the triple carry over unchanged from the previous lesson.

• ED8. The setup, the split (756/252/252), the seed and the subgroup definitions carry over unchanged from the previous lesson: four axes, 13 subgroups, an n written next to every number, and a group whose n is below 25 does not enter the gap.
• ED9. The model is the same depth-6 decision tree, with one difference: the leaf holds not the class but the suspicious share. This share is the model’s score; a subscriber above the threshold is sent to inspection.
• ED10. The metric is the cost per subscriber and small is good. A false alarm (a flagged subscriber who is not suspicious) costs $a$ units, a miss (an unflagged subscriber who is suspicious) costs $m$ units; correct decisions cost zero. The unit is abstract and is not money.
• ED11. Two cost matrices are tested. Inspection costly: $a = 2$, $m = 1$. Illicit use costly: $a = 1$, $m = 4$. Both are fictional and stand for a business decision.
• ED12. The threshold is swept over 10 candidates (from 0.05 to 0.95 in steps of one tenth) and is selected on the validation set; the test set is read only for the report.
• ED13. The baseline is computed separately for each cost matrix and is the cheaper one of two procedures that learn nothing: flagging nobody, or flagging everybody.
• ED14. A subgroup-specific threshold is selected on that subgroup’s validation rows and reported on that subgroup’s test rows. Subgroup validation sets are small, and the selection share comes from there.
• ED15. Accuracy is never printed in this lesson. The 0.7778 of the previous lesson and the costs of this lesson are not on the same axis and are not compared.

Two Cost Structures, Two Thresholds

The first block sets up the scenario, trains the tree so that it leaves a share at the leaf, and sweeps ten thresholds under the two cost matrices.

# MODEL: the measurement network of the Data Analytics curriculum is a FICTIONAL setup generated with the
# same seed; readings are summarized at subscriber level and a FICTIONAL illicit-use/fault suspicion label is added on top.
import math

SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF
REGION = [("north", 0.28, 21, 0.00), ("south", 0.22, 17, -0.10),
          ("east", 0.18, 26, 0.30), ("west", 0.14, 14, -0.05),
          ("central", 0.18, 23, 0.05)]


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


SUBSCRIBERS, READINGS, DATA = [], {}, []
for i in range(RAW):
    r = rng(SEED + i)
    b = REGION[pick(r(), [x[1] for x in REGION])]
    household = 1 + pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05])
    rating = 1 + pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17])
    if r() >= 0.046:
        SUBSCRIBERS.append({"id": 10001 + i, "region": b, "household": household,
                            "satisfaction": rating})
for k in SUBSCRIBERS:
    r = rng(SEED + 7000 + k["id"])
    for n in range(pick(r(), [0.05, 0.12, 0.21, 0.62])):
        READINGS.setdefault(k["id"], []).append(0.0 if r() < 0.038 else math.floor(
            k["region"][2] * math.exp((r() + r() + r() - 1.5) * 0.62)
            * (1 - n * 0.05) * 100 + 0.5) / 100)
        r()
    v = READINGS.get(k["id"])
    if v is None:
        continue
    mean, vol, r = sum(v) / len(v), max(v) - min(v), rng(SEED + 51000 + k["id"])
    z = (0.052 * (mean - 20) + 0.85 * (min(v) == 0.0) + 0.026 * vol
         + 0.24 * (k["household"] >= 5) - 0.20 * (k["satisfaction"] >= 4) + k["region"][3]
         + 0.9 * (k["id"] - 10001) / 1399 + (r() + r() + r() - 1.5) * 1.30)
    x = {"id": k["id"], "household": k["household"], "satisfaction": k["satisfaction"],
         "period": len(v), "mean_consumption": round(mean, 2), "volatility": round(vol, 2),
         "zero_readings": int(min(v) == 0.0), "suspicious": int(z > 1.35),
         "region_name": k["region"][0]}
    for b in REGION:
        x["area_" + b[0]] = int(k["region"][0] == b[0])
    DATA.append(x)

FIELDS = ["mean_consumption", "volatility", "household", "satisfaction", "period",
          "zero_readings", "area_north", "area_south", "area_east", "area_west", "area_central"]


def shuffle(data, seed):
    r, s = rng(seed), list(range(len(data)))
    for i in range(len(s) - 1, 0, -1):
        j = int(r() * (i + 1))
        s[i], s[j] = s[j], s[i]
    return [data[i] for i in s]


def three_way_split(rows):
    a, b = int(len(rows) * 0.6), int(len(rows) * 0.8)
    return rows[:a], rows[a:b], rows[b:]


def impurity(rows):
    p = sum(x["suspicious"] for x in rows) / len(rows) if rows else 0.0
    return 2 * p * (1 - p)


def tree(rows, depth, fields, min_size=2):     # the leaf holds the SHARE, not the class
    p, best = sum(x["suspicious"] for x in rows) / len(rows), None
    if depth and len(rows) >= 2 * min_size and 0.0 < p < 1.0:
        for f in fields:
            values = sorted({x[f] for x in rows})
            for v in (values[1:] if len(values) < 10 else
                      [values[int(i * len(values) / 10)] for i in range(1, 10)]):
                left = [x for x in rows if x[f] < v]
                right = [x for x in rows if x[f] >= v]
                if min(len(left), len(right)) < min_size:
                    continue
                k = impurity(rows) - (len(left) * impurity(left) + len(right) * impurity(right)) / len(rows)
                if best is None or k > best[0]:
                    best = (k, f, v, left, right)
    if best is None or best[0] <= 1e-9:
        return {"p": p}
    return {"field": best[1], "threshold": best[2], "left": tree(best[3], depth - 1, fields),
            "right": tree(best[4], depth - 1, fields)}


def score(node, x):
    while "p" not in node:
        node = node["left"] if x[node["field"]] < node["threshold"] else node["right"]
    return node["p"]


def subgroups(x):                  # four axes, thirteen subgroups; carried over unchanged from lesson 01
    return [("region", x["region_name"]),
            ("tariff", "10" if x["mean_consumption"] <= 10 else
             ("25" if x["mean_consumption"] <= 25 else "40")),
            ("household", "1-2" if x["household"] <= 2 else ("3-4" if x["household"] <= 4 else "5+")),
            ("reading", "few" if x["period"] <= 2 else "full")]


def cost(rows, t, fa, miss):       # cost per subscriber; small is good
    total = 0
    for x in rows:
        total += fa * (1 - x["suspicious"]) if score(M, x) >= t else miss * x["suspicious"]
    return total / len(rows)


def flagged(rows, t):
    return sum(score(M, x) >= t for x in rows) / len(rows)


SUBGROUPS = sorted({g for x in DATA for g in subgroups(x)})
TRAIN, VAL, TEST = three_way_split(shuffle(DATA, SEED + 90000))
M = tree(TRAIN, 6, FIELDS)
THRESHOLDS = [0.05 + 0.10 * i for i in range(10)]
STRUCTURES = (("inspection costly", 2, 1), ("illicit use costly", 1, 4))

print(f"{'threshold':>9} | {'A val':>7} {'A test':>7} | {'B val':>7} {'B test':>7} "
      f"| {'flagged':>7}")
for t in THRESHOLDS:
    print(f"{t:>9.2f} | {cost(VAL, t, 2, 1):>7.4f} {cost(TEST, t, 2, 1):>7.4f} "
          f"| {cost(VAL, t, 1, 4):>7.4f} {cost(TEST, t, 1, 4):>7.4f} "
          f"| {flagged(TEST, t):>7.4f}")
CHOSEN = {}
for name, fa, miss in STRUCTURES:
    t = CHOSEN[name] = min(THRESHOLDS, key=lambda c: (cost(VAL, c, fa, miss), c))
    flag_none, flag_all = cost(TEST, 1.1, fa, miss), cost(TEST, -0.1, fa, miss)
    print(f"\n{name:<18} threshold {t:.2f}  test cost {cost(TEST, t, fa, miss):.4f}  baseline "
          f"{min(flag_none, flag_all):.4f} ({'none' if flag_none < flag_all else 'all'})  gain "
          f"{min(flag_none, flag_all) - cost(TEST, t, fa, miss):+.4f}  flagged "
          f"{sum(score(M, x) >= t for x in TEST)}/{len(TEST)}")

threshold |   A val  A test |   B val  B test | flagged
     0.05 |  1.0952  0.9762 |  0.6032  0.5437 |  0.7063
     0.15 |  0.7659  0.7659 |  0.4802  0.5913 |  0.5357
     0.25 |  0.4802  0.4246 |  0.4762  0.5595 |  0.3056
     0.35 |  0.3889  0.3214 |  0.5278  0.5913 |  0.2183
     0.45 |  0.3492  0.2976 |  0.5079  0.6349 |  0.1825
     0.55 |  0.2778  0.2579 |  0.5833  0.7262 |  0.1151
     0.65 |  0.2540  0.2460 |  0.6548  0.7341 |  0.1032
     0.75 |  0.2222  0.2302 |  0.6667  0.7540 |  0.0833
     0.85 |  0.2222  0.2302 |  0.6667  0.7540 |  0.0833
     0.95 |  0.2183  0.2222 |  0.6786  0.7778 |  0.0675

inspection costly  threshold 0.95  test cost 0.2222  baseline 0.2421 (none)  gain +0.0198  flagged 17/252

illicit use costly threshold 0.25  test cost 0.5595  baseline 0.7579 (all)  gain +0.1984  flagged 77/252

The two cost columns fall in opposite directions. In the structure where inspection is costly, the cost drops as the threshold rises and the cheapest point is 0.95; in the structure where illicit use is costly the cheapest point is 0.25, after which the cost starts to climb. The model is a single model, the scores are a single set of scores, and the two structures pick two separate rows from the same table. In one case 17 of 252 subscribers go to inspection, in the other 77.

The baseline moves with the cost as well. When inspection is costly, the cheapest procedure that learns nothing is flagging nobody, and it costs 0.2421; when illicit use is costly it is flagging everybody, and it costs 0.7579. Both are computed on the same set with the same labels. The model’s gain changes accordingly: in the first structure it can drop only 0.0198 below the baseline, in the second 0.1984. The same model, the same test set — and one is reported as “helps little”, the other as “helps clearly”.

Who the Decision Changes For

The difference between the two operating points is two rows in a table, but on the subscriber side it is something concrete: some subscribers go to inspection under one structure and not under the other.

tA, tB = CHOSEN["inspection costly"], CHOSEN["illicit use costly"]
changed = [x for x in TEST if (score(M, x) >= tA) != (score(M, x) >= tB)]
shares = sorted(((sum(g in subgroups(x) for x in changed)
                  / sum(g in subgroups(x) for x in TEST), g) for g in SUBGROUPS), reverse=True)
print(f"decision changed for {len(changed)}/{len(TEST)} subscribers, {sum(x['suspicious'] for x in changed)}"
      f" of them suspicious; share by subgroup:")
for o, g in shares[:2] + shares[-1:]:
    print(f"  {' '.join(g):<14} {o:.4f}")

decision changed for 60/252 subscribers, 23 of them suspicious; share by subgroup:
  tariff 10      0.7500
  region east    0.5556
  region west    0.0938

The decision of sixty subscribers flips, and 23 of them are truly suspicious. Choosing the cost structure is not a metric preference; it is whether sixty households get a knock on the door or not. The distribution is not even either: 0.5556 of the subscribers in the east region move between the two structures, while in the west the share is 0.0938. Once a cost matrix is chosen, the subgroups do not carry its load equally.

The Cost Is Not the Same Across Subgroups

The single threshold selected in the structure where illicit use is costly is 0.25, and it is applied to all thirteen subgroups in the same way. The block below prints, side by side, the cost of that single threshold in each subgroup, that subgroup’s own baseline, and what would happen if the subgroup selected its own threshold.

fa, miss = 1, 4
print(f"{'subgroup':<14} {'n':>4} {'flagged':>7} {'cost':>7} {'baseline':>8} {'gain':>8}"
      f" | {'own':>5} {'cost':>7} {'extra':>8}")
gains, extras = [], []
for g in SUBGROUPS:
    val_g = [x for x in VAL if g in subgroups(x)]
    test_g = [x for x in TEST if g in subgroups(x)]
    c, base = cost(test_g, tB, fa, miss), min(cost(test_g, 1.1, fa, miss), cost(test_g, -0.1, fa, miss))
    own_t = min(THRESHOLDS, key=lambda t: (cost(val_g, t, fa, miss), t))
    own_cost = cost(test_g, own_t, fa, miss)
    if len(test_g) >= 25:
        gains.append(base - c)
        extras.append(c - own_cost)
    print(f"{' '.join(g):<14} {len(test_g):>4} {flagged(test_g, tB):>7.4f} {c:>7.4f} {base:>8.4f} "
          f"{base - c:>+8.4f} | {own_t:>5.2f} {own_cost:>7.4f} {c - own_cost:>+8.4f}"
          f"{'  n<25' if len(test_g) < 25 else ''}")
print(f"\ngain gap ({len(gains)} subgroups): best {max(gains):+.4f}, worst "
      f"{min(gains):+.4f}, gap {max(gains) - min(gains):.4f}; below baseline "
      f"{sum(1 for v in gains if v < 0)}/{len(gains)}")
print(f"subgroups whose own threshold lowers the test cost: {sum(1 for v in extras if v > 0)}"
      f"/{len(extras)}, unchanged {sum(1 for v in extras if v == 0)}, raised "
      f"{sum(1 for v in extras if v < 0)}")

subgroup          n flagged    cost baseline     gain |   own    cost    extra
household 1-2    73  0.3562  0.5205   0.7534  +0.2329 |  0.15  0.6301  -0.1096
household 3-4   143  0.2657  0.5944   0.7692  +0.1748 |  0.25  0.5944  +0.0000
household 5+     36  0.3611  0.5000   0.7222  +0.2222 |  0.55  0.7500  -0.2500
reading few      85  0.2706  0.7059   0.7882  +0.0824 |  0.45  0.8471  -0.1412
reading full    167  0.3234  0.4850   0.7425  +0.2575 |  0.15  0.5509  -0.0659
region central   58  0.3276  0.8448   0.6552  -0.1897 |  0.25  0.8448  +0.0000
region east      36  0.7500  0.7778   0.4722  -0.3056 |  0.45  0.7222  +0.0556
region north     68  0.2500  0.6618   0.7059  +0.0441 |  0.15  0.6765  -0.0147
region south     58  0.1897  0.2759   0.6897  +0.4138 |  0.15  0.5517  -0.2759
region west      32  0.0938  0.0938   0.0000  -0.0938 |  0.35  0.0625  +0.0312
tariff 10        12  0.7500  0.4167   0.6667  +0.2500 |  0.05  0.4167  +0.0000  n<25
tariff 25       193  0.1813  0.4870   0.6425  +0.1554 |  0.45  0.4560  +0.0311
tariff 40        47  0.7021  0.8936   0.4468  -0.4468 |  0.05  0.5957  +0.2979

gain gap (12 subgroups): best +0.4138, worst -0.4468, gap 0.8606; below baseline 4/12
subgroups whose own threshold lowers the test cost: 4/12, unchanged 2, raised 6

The gain, 0.1984 overall, spreads across the subgroups between +0.4138 and −0.4468; the gap is 0.8606. In four subgroups the model is below its own baseline: west, east, central and the tariff 40 tier. The baseline of the tariff 40 tier is 0.4468 and the model’s cost is 0.8936 — in that tier the model is twice as expensive as saying “flag everybody”. The reason is in the flagged column: the 0.25 threshold flags 0.7021 of the subscribers in that tier, but because the suspicious share of the tier is 0.553, the suspicious subscribers left among the unflagged pay the four-unit cost of a miss.

The last three columns suggest a correction and show the cost of the correction too. When each subgroup selects its own threshold from its own validation rows, the tariff 40 tier drops to 0.05 and its cost falls from 0.8936 to 0.5957. But the same procedure lowers the test cost in only four of the twelve subgroups; in two it changes nothing, and in six it raises it — by 0.2759 units in the south and 0.2500 in household size 5 and above. The reason is the smallness of the subgroup validation sets: a threshold selected from forty to seventy rows selects the noise of those rows. Tuning a threshold per subgroup requires a validation set per subgroup, and those sets come from the design, not from the split.

Summary

• A cost matrix turns the business objective into a number: the ratio between a false alarm and a miss is a business decision, and the threshold is read from that ratio. The metric is the cost per subscriber and small is good.
• The same model, the same scores, two cost matrices: when inspection is costly the selected threshold is 0.95 and 17 of 252 subscribers are flagged; when illicit use is costly the threshold is 0.25 and 77 are flagged. The decision of 60 subscribers flips, and 23 of them are truly suspicious.
• The baseline moves with the cost too: in the first structure flagging nobody (0.2421), in the second flagging everybody (0.7579) is the baseline. The model’s gain is accordingly +0.0198 or +0.1984.
• A single threshold does not give a single result across subgroups: the gain spreads between +0.4138 and −0.4468, the gap is 0.8606, and in 4 of 12 subgroups the model stays below its own baseline.
• Selecting a subgroup-specific threshold lowers the test cost in only 4 of 12 subgroups and raises it in six; the smaller the subgroup validation set, the larger the selection share.

Next Step

In this lesson the baseline turned into a question of its own for the second time. When the cost structure changed, the baseline changed too: in one, flagging nobody, in the other, flagging everybody. Across the subgroups every group turned out to have its own baseline, which was 0.0000 in one group and 0.7882 in another. Over seven courses the baseline was a tool — a fixed mark that makes the model’s number readable. Here it was seen not to be fixed. The next lesson takes the baseline up directly: how many baselines a problem has, which one is chosen, which two separate ways the choice lets the same model be read, and what shows when the baselines chosen in the seven courses of this curriculum are collected in a single table.