Roles

The previous lesson ran the flow from a single desk: one person wrote the question, chose the features, built the model, set the threshold and monitored the field. In reality these steps are divided among different job definitions, and the division not only shares out the work, it also changes what is delivered.

The Introduction to Data Analytics course had separated three analyst roles by their acceptance criteria: the measuring role delivered a number, the role that binds the decision a threshold, the generalizing role a generalization together with its assumption. This lesson makes the same separation on the machine learning side, and what it measures is not the delivery itself but the baseline each role chooses.

• FC32. Role names here are not a title but a job definition: what is delivered and by which measure it is accepted. No company, title standard or document is mentioned.
• FC33. The data scientist delivers a finding: whether the problem is predictable and how far. Its baseline is the field’s current procedure, because what it has to prove is that something better than the present state of the job is possible.
• FC34. The machine learning engineer delivers a procedure that runs every day. Its baseline is the model running today, because its job is not to prove something from scratch but to keep something standing.
• FC35. The AI engineer delivers an end-to-end system. Its baseline is the same system built without a model, and its measure is not the average but the worst case, because a system is as reliable as what it gives on its worst day.
• FC36. The three roles use the same measure and all three deliver 30, but they produce three separate gain figures. What determines the number is not the model but the chosen baseline.
• FC37. A fallback path is not counted as an improvement without being measured. In this run the system that falls back finds fewer illicit cases on the drifted data than the one that never falls.

Rebuilding the Fictional Dataset

# setup.py — the fictional dataset of lesson 01 is rebuilt with the same seed: the readings
# table, the subscriber summary, the fictional label, the train / held-out split and the tree learner.
import math

SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF
REGION = [("north", 0.28, 21), ("south", 0.22, 17), ("east", 0.18, 26),
          ("west", 0.14, 14), ("central", 0.18, 23)]
FEATURES = ["mean", "per_person", "drop", "zero_readings", "reading_count",
            "satisfaction", "household_size", "highest", "lowest"]


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


_subscribers, _dropped, _series = [], [], {}
for i in range(RAW):
    r = rng(SEED + i)
    b = REGION[pick(r(), [x[1] for x in REGION])]
    k = {"subscriber_id": 10001 + i, "region": b[0], "base": b[2],
         "household_size": pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]),
         "satisfaction": pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) + 1}
    (_dropped if r() < 0.046 else _subscribers).append(k)
_valid = {a["subscriber_id"] for a in _subscribers}
for k in _subscribers + _dropped:
    r = rng(SEED + 7000 + k["subscriber_id"])
    for d in range(pick(r(), [0.05, 0.12, 0.21, 0.62])):
        t = 0.0 if r() < 0.038 else math.floor(k["base"] * math.exp(
            (r() + r() + r() - 1.5) * 0.62) * (1 - d * 0.05) * 100 + 0.5) / 100
        if k["subscriber_id"] in _valid:
            _series.setdefault(k["subscriber_id"], []).append(t)
        r()

SUBSCRIBERS = []
for k in _subscribers:
    t = _series.get(k["subscriber_id"])
    if not t:
        continue
    mean = sum(t) / len(t)
    a = {"subscriber_id": k["subscriber_id"], "region": k["region"], "series": t,
         "household_size": k["household_size"], "satisfaction": k["satisfaction"],
         "reading_count": len(t), "mean": mean, "highest": max(t),
         "lowest": min(t), "zero_readings": sum(1 for v in t if v == 0),
         "per_person": mean / max(1, k["household_size"]),
         "drop": (t[0] - t[-1]) / t[0] if len(t) > 1 and t[0] > 0 else 0.0}
    r = rng(SEED + 31000 + a["subscriber_id"])
    a["suspicion"] = 1 if (1.30 * a["zero_readings"] + 1.25 * (a["drop"] > 0.45)
                           + 1.05 * (a["per_person"] > 12.0) + 0.65 * (a["reading_count"] < 3)
                           + 0.45 * (a["satisfaction"] <= 2) - 1.55
                           + (r() + r() - 1.0) * 1.1) > 0 else 0
    SUBSCRIBERS.append(a)

r = rng(SEED + 90001)
TRAIN, HELD_OUT = [], []
for a in SUBSCRIBERS:
    (TRAIN if r() < 0.70 else HELD_OUT).append(a)
positives = lambda k: sum(a["suspicion"] for a in k)
accuracy = lambda f, k: sum(1 for a in k if f(a) == a["suspicion"]) / len(k)
_impurity = lambda k: 2 * (positives(k) / len(k)) * (1 - positives(k) / len(k)) if k else 0.0


def fit(k, depth=3, min_size=40):
    """The greedy decision tree of lesson 01: the split criterion and stopping rule are there."""
    best, choice = _impurity(k) * len(k), None
    if depth and len(k) >= min_size:
        for feat in FEATURES:
            for threshold in sorted({a[feat] for a in k})[1:]:
                left = [a for a in k if a[feat] < threshold]
                right = [a for a in k if a[feat] >= threshold]
                m = _impurity(left) * len(left) + _impurity(right) * len(right)
                if m < best - 1e-9:
                    best, choice = m, (feat, threshold, left, right)
    if choice is None:
        return 1 if 2 * positives(k) > len(k) else 0
    lt, rt = fit(choice[2], depth - 1, min_size), fit(choice[3], depth - 1, min_size)
    return lt if lt == rt else (choice[0], choice[1], lt, rt)


def apply(d, a):
    while isinstance(d, tuple):
        d = d[2] if a[d[0]] < d[1] else d[3]
    return d


print(f"fictional dataset (seed {SEED}): {len(SUBSCRIBERS)} subscribers, {len(TRAIN)} train, "
      f"{len(HELD_OUT)} held-out, {positives(SUBSCRIBERS)} suspicious")

fictional dataset (seed 20260218): 1260 subscribers, 883 train, 377 held-out, 327 suspicious

A Common Measure, Five Rankers

For the three roles to be comparable the measure must be common. The common measure comes from the previous lesson: the daily capacity is 40 visits and what is counted is the illicit cases found in the first 40 visits. What is not common is the baseline against which each role reads this measure.

The five rankers are measured in two data layouts. The first layout is the table at the beginning of the lesson; the second is the drift built in the previous lesson — the measurement policy has changed, zero readings have been filled with an estimate, missing periods have been completed. The label is the same in both layouts.

# rankers.py — the common measure of the three roles: illicit cases found in as many visits as
# the daily capacity. Five separate rankers and two separate data layouts are measured side by side.
import statistics
from collections import Counter

CAPACITY = 40
r = rng(SEED + 7205)
DEPLOYED = [fit([TRAIN[int(r() * len(TRAIN))] for _ in TRAIN]) for _ in range(9)]
r = rng(SEED + 7206)
lots = {a["subscriber_id"]: r() for a in SUBSCRIBERS}
found_in_top = lambda f, s: positives(sorted(s, key=f)[:CAPACITY])
ranking = lambda T: (lambda a: (-sum(apply(d, a) for d in T), a["subscriber_id"]))
field = lambda a: (-a["mean"], a["subscriber_id"])
hand = lambda a: 1 if (a["zero_readings"] >= 1 or a["drop"] > 0.50
                       or a["per_person"] > 15.0) else 0
rule = lambda a: (-hand(a), -a["mean"], a["subscriber_id"])


def drifted(a):
    """The measurement policy change of lesson 05: zero readings are filled with an
    estimate, missing periods are completed. The label does not change."""
    filled = [v for v in a["series"] if v > 0]
    est = round(statistics.fmean(filled), 2) if filled else 0.0
    t = [v if v > 0 else est for v in a["series"]] + [est] * (3 - len(a["series"]))
    avg = sum(t) / len(t)
    return {**a, "series": t, "reading_count": 3, "mean": avg, "highest": max(t),
            "lowest": min(t), "zero_readings": sum(1 for v in t if v == 0),
            "per_person": avg / max(1, a["household_size"]),
            "drop": (t[0] - t[-1]) / t[0] if t[0] > 0 else 0.0}


DRIFT_TRAIN, DRIFT_HELD = [drifted(a) for a in TRAIN], [drifted(a) for a in HELD_OUT]
r = rng(SEED + 7305)
CURRENT = [fit([DRIFT_TRAIN[int(r() * len(DRIFT_TRAIN))] for _ in DRIFT_TRAIN]) for _ in range(9)]
signal = lambda s: sum(1 for a in s if a["zero_readings"] > 0 or a["reading_count"] < 3) / len(s)

W = [33, 12, 12, 12, 12]
row = lambda h: str(h[0]).ljust(W[0]) + "".join(str(v).rjust(W[i + 1])
                                                for i, v in enumerate(h[1:]))
RANKERS = [("random order", lambda a: lots[a["subscriber_id"]]),
           ("field procedure: top consumer", field),
           ("hand-written rule order", rule),
           ("deployed model order", ranking(DEPLOYED)),
           ("model built on drifted data", ranking(CURRENT))]
print(f"illicit cases found in the first {CAPACITY} visits")
print(row(["ranker", "old tr.", "old held.", "drift tr.", "drift held."]))
COUNTS = {}
for name, f in RANKERS:
    COUNTS[name] = [found_in_top(f, TRAIN), found_in_top(f, HELD_OUT),
                    found_in_top(f, DRIFT_TRAIN), found_in_top(f, DRIFT_HELD)]
    print(row([name] + COUNTS[name]))
print(f"\ndata quality signal (share of subscribers with a zero reading or a missing period): "
      f"old {signal(HELD_OUT):.4f}, drifted {signal(DRIFT_HELD):.4f}")
print(f"the held-out set has {positives(HELD_OUT)} suspicious; the ceiling of {CAPACITY} visits is {CAPACITY}")

illicit cases found in the first 40 visits
ranker                                old tr.   old held.   drift tr. drift held.
random order                               10          14          10          14
field procedure: top consumer              17          13          18          14
hand-written rule order                    25          21          26          21
deployed model order                       30          30          33          23
model built on drifted data                40          30          39          30

data quality signal (share of subscribers with a zero reading or a missing period): old 0.3740, drifted 0.0027
the held-out set has 93 suspicious; the ceiling of 40 visits is 40

The table shows five separate procedures in five rows. The random order finds 14 illicit cases and is fixed in both layouts; the difficulty of the question has not changed. The field’s current procedure — starting from the highest consumer — finds 13 on the held-out set, that is, one illicit case behind even the random order. The hand-written rule order finds 21. The deployed model finds 30 and falls to 23 on the drifted data. The model rebuilt on the drifted data gives 30 in both layouts.

The data quality signal in the last row tells where the drift comes from: the share of subscribers with a zero reading or a missing period has gone from 0.3740 to 0.0027. The two columns the model relied on have been erased from the table, and what measures this is not the model but a counter that looks at the data itself.

Three Roles, Three Baselines

# roles.py — the same model, three job definitions, three separate baselines. Each role chooses its
# own baseline and what it delivers is compared with that baseline.
WORST = lambda s: min(s[1], s[3])  # the held-out counts of the two data layouts
DEPLOYED_NAME = "deployed model order"
CURRENT_NAME = "model built on drifted data"
FALLBACK = [COUNTS[DEPLOYED_NAME][0], COUNTS[DEPLOYED_NAME][1],
            COUNTS["hand-written rule order"][2], COUNTS["hand-written rule order"][3]]

ROLES = [
    ("data scientist", "field's current procedure",
     COUNTS["field procedure: top consumer"][1], "model order",
     COUNTS[DEPLOYED_NAME][1]),
    ("machine learning eng.", "deployed model, on drifted data",
     COUNTS[DEPLOYED_NAME][3], "rebuilt procedure", COUNTS[CURRENT_NAME][3]),
    ("AI engineer", "system without a model, worst case",
     WORST(COUNTS["hand-written rule order"]), "system with signal, worst case",
     WORST(COUNTS[CURRENT_NAME])),
]
W2 = [24, 36, 10, 32, 10, 8]
row2 = lambda h: str(h[0]).ljust(W2[0]) + str(h[1]).ljust(W2[1]) + str(h[2]).rjust(W2[2]) \
    + "  " + str(h[3]).ljust(W2[3]) + str(h[4]).rjust(W2[4]) + str(h[5]).rjust(W2[5])
print(row2(["role", "chosen baseline", "baseline", "delivered", "count", "gap"]))
for name, base_name, base, delivered_name, delivered in ROLES:
    print(row2([name, base_name, base, delivered_name, delivered, f"{delivered - base:+d}"]))
print(f"\nfallback system (switches to the rule order when the signal falls): "
      f"old data {FALLBACK[1]}, drifted data {FALLBACK[3]}, worst {min(FALLBACK[1], FALLBACK[3])}")
print(f"system that never falls back: old {COUNTS[DEPLOYED_NAME][1]}, drifted {COUNTS[DEPLOYED_NAME][3]}, "
      f"worst {WORST(COUNTS[DEPLOYED_NAME])}")

role                    chosen baseline                       baseline  delivered                            count     gap
data scientist          field's current procedure                   13  model order                             30     +17
machine learning eng.   deployed model, on drifted data             23  rebuilt procedure                       30      +7
AI engineer             system without a model, worst case          21  system with signal, worst case          30      +9

fallback system (switches to the rule order when the signal falls): old data 30, drifted data 21, worst 21
system that never falls back: old 30, drifted 23, worst 23

The same measure, the same delivered count — and three separate gains: 17, 7 and 9 illicit cases. All three numbers are correct and none of them can be written in place of another.

The 17 of the data scientist is a proof: if the field’s current procedure finds 13 illicit cases in 40 visits and a model finds 30, this problem is a learnable problem. This number has a short life; once it is proven, nobody writes the same baseline a second time.

The 7 of the machine learning engineer is a maintenance number. A model is already running in the field and it gives 23 on the drifted data; the procedure delivered gives 30. Seven illicit cases look small, but the baseline is not 13 but 23 — the job of the engineer is not to produce a gain from zero but not to lose the gain already produced.

The 9 of the AI engineer is a resilience number, and it is not a single number but the worst of two layouts. The system without a model gives 21 in both layouts; the system with a signal gives 30 in the worst case. The 9 in between is not an average gain but the difference of the lowest guaranteed value.

Measuring the Fallback Path

The last two rows refute a design decision. A system that drops the model and switches to the hand-written rule order when the signal falls finds 21 illicit cases on the drifted data. One that never falls back, that is, keeps working with the corrupted feature, finds 23. Falling back makes the number two illicit cases worse.

This does not mean the fallback path is unnecessary. The value of the signal lies not in switching to the fallback but in giving notice: a counter that falls from 0.3740 to 0.0027 is what triggers the return from the seventh step to the second in the previous lesson, and the model built as a result of that return gives 30. Without the signal the list in the field would go on producing seven illicit cases too few, and in the system that falls back nine illicit cases too few.

What the three roles have in common shows here. All three deliver a number, all three write their own baseline next to that number, and the baseline of all three is a choice. When the baseline is not written, all three roles can utter the same sentence: “the system finds 30 illicit cases in 40 visits.” This sentence is three times true, three times unmeasured.

Summary

• Role names here are job definitions: what is delivered and by which measure it is accepted. The measure is the same in all three — illicit cases found in the first 40 visits — the baselines differ.
• The baseline of the data scientist is the field’s current procedure and finds 13 illicit cases; the model order delivered finds 30, the gap is 17.
• The baseline of the machine learning engineer is the model running today and finds 23 on the drifted data; the rebuilt procedure delivered finds 30, the gap is 7.
• The baseline of the AI engineer is the system without a model and finds 21 in the worst case; the worst case of the system with a signal is 30, the gap is 9.
• The system that falls back finds 21 on the drifted data, the one that never falls 23: a fallback path is not counted as an improvement without being measured, and the value of the signal lies not in the fallback but in giving notice.
• The same delivered count produces 17, 7 and 9 in three roles; what determines the number is not the model but the chosen baseline.

Next Step

Across six lessons every table had three columns: the baseline, the one measured on training and the one measured on the held-out set. The third column was printed every time and never questioned in any lesson. Why the split is made, at what ratio it is made, how many times the same held-out set may be looked at, why the two numbers diverge was not asked even once. The number was printed, and passed by.

Yet in this topic that column tried to speak several times. The hand-written rule passed the baseline on training and stayed below it on the held-out set. The model built with thirty labels gave 0.9400 on training and 0.7745 on the held-out set, and as labels increased the two numbers drew closer. The model built on the drifted data filled all 40 of 40 visits with illicit cases on the training set and stayed at 30 on the held-out set. The same thing happened in three cases: the number measured on the data the model saw came out higher than the one measured on the data it did not see, and the gap between them was never explained.

The next topic takes up this gap. The reason for the split is written, the training, validation and test sets are separated from one another, the point at which the two numbers begin to diverge as the model complexity is raised step by step is counted, how much the same procedure moves across different samples is measured, and four separate ways that make the number measured on the held-out set look larger than it really is are applied one by one.