Online Experiments

The previous lesson showed that the shadow placement measured the candidate on all two hundred rows and gave 0.9650 on 2026-12; the old model was 0.9800 in the same period. The 0.0150 between them looked sufficient for a decision. An unanswered question sits underneath this number: how small can the difference of the two models become before this set can no longer read it.

The question is one of experiment design. Comparing two arms, sample size and confidence interval were built in the User Experience and Behavior Design course of the Interface Design and Design Systems curriculum, and distribution assumptions and the interval procedure were measured in the Statistical Analysis and Exploratory Data Analysis course of the Data Analytics curriculum; neither is repeated here. This lesson calculates a single number: the minimum detectable effect of this set size. Every comparison below the number found is inconclusive, whatever procedure it is made with, and the lesson shows the single example of this together with its mark.

• RG11. The source, the pipeline and the model are fictional; no experiment platform, deployment tool or monitoring product is called. Seed 20260218.
• RG12. The label is tied to the rule subscriber_mean + 6 x commercial > 30, the decision threshold is a separate setting and the training criterion is balanced accuracy. In the monitored periods the base class ratio is 0.5342.
• RG13. The experiment horizon is the six monitored periods: 200 rows per period, 1200 in total. The step of a period’s accuracy is 0.0050, the step of the six periods is 0.0008.
• RG14. The two arms being compared see the same rows and are measured against the same label. There is no selection bias coming from the split of the arms; this is an assumption that must be shown separately in a real experiment, and here it holds by the fiction.
• RG15. The production data has a data drift and the rule has not changed.
• RG16. The candidates are the first four ranks of the previous lesson’s retraining search; the grid is 135 candidates from 27 weights and 5 decision thresholds.

The Same Candidate Moving From Period to Period

The first block sets up the setup, takes the four candidates and measures the difference of each from the baseline model period by period.

# FICTIONAL. The meter readings of the measurement network, the consumption stage of the
# pipeline, the feature path and the three-weight threshold model are generated inside
# the lesson; no learning library, experiment platform or monitoring product is
# called. Seed 20260218.
SEED, M32 = 20260218, 0xFFFFFFFF


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


REGION = ["east", "west", "north", "south", "central"]
TYPE = ["residential", "commercial"]
PERIODS = [f"2026-{a:02d}" for a in range(1, 13)]
SUBSCRIBERS = {}
for i in range(200):
    r = rng(SEED + 37 * i)
    SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)],
                                    "start": 10000 + int(r() * 60000)}


def rows():
    """Source readings and consumption from the period difference: the first three stages of the pipeline."""
    records = []
    for i, (sub, a) in enumerate(SUBSCRIBERS.items()):
        e = a["start"]
        for p in PERIODS:
            r = rng(SEED + 101 * i + 7 * PERIODS.index(p))
            e += 4 + int(r() * 46)
            records.append({"subscriber": sub, "period": p, "region": a["region"], "type": a["type"],
                            "index": e})
    t = {(k["subscriber"], k["period"]): k["index"] for k in records}
    return [dict(k, m3=k["index"] - t[(k["subscriber"], PERIODS[PERIODS.index(k["period"]) - 1])])
            for k in records if PERIODS.index(k["period"])]


def features(records):
    """Independent of the weights and computed ONCE; it sees the full history."""
    sub, reg = {}, {}
    for k in sorted(records, key=lambda k: PERIODS.index(k["period"])):
        sub.setdefault(k["subscriber"], []).append((PERIODS.index(k["period"]), k["m3"]))
        reg.setdefault(k["region"], []).append((PERIODS.index(k["period"]), k["m3"]))

    def mean(series, i):
        s = [v for d, v in series if d < i]
        return round(sum(s) / len(s), 4) if s else 0.0

    return {(k["subscriber"], k["period"]): {
        "subscriber_mean": mean(sub[k["subscriber"]], PERIODS.index(k["period"])),
        "region_mean": mean(reg[k["region"]], PERIODS.index(k["period"])),
        "commercial": 1 if k["type"] == "commercial" else 0} for k in records}


RULE_THRESHOLD, RULE_COMMERCIAL = 30, 6   # the RULE that generates the label , in other words the concept
MONITORED = PERIODS[6:]                   # six periods monitored in production
GRID = [(a, b, c) for a in (0.6, 0.8, 1.0) for b in (0.0, 0.2, 0.4) for c in (0, 4, 8)]
DECISION_GRID = (18, 22, 26, 30, 34)      # the decision threshold is A SEPARATE setting


def label(o, threshold=RULE_THRESHOLD):
    return 1 if o["subscriber_mean"] + RULE_COMMERCIAL * o["commercial"] > threshold else 0


def score(o, w):
    return w[0] * o["subscriber_mean"] + w[1] * o["region_mean"] + w[2] * o["commercial"]


def predict(o, w, decision):
    return 1 if score(o, w) > decision else 0


def balanced_accuracy(subset, feats, w, decision):
    """Training criterion: the mean of recall and specificity."""
    tp = tn = fp = fn = 0
    for k in subset:
        o = feats[(k["subscriber"], k["period"])]
        p, g = predict(o, w, decision), label(o)
        if p and g:
            tp += 1
        elif p:
            fp += 1
        elif g:
            fn += 1
        else:
            tn += 1
    recall = tp / (tp + fn) if tp + fn else 0.0
    specificity = tn / (tn + fp) if tn + fp else 0.0
    return (recall + specificity) / 2, round(recall, 4), round(specificity, 4)


def train(records, feats, periods):
    """Weights and decision threshold are searched together: 27 x 5 = 135 candidates."""
    subset = [k for k in records if k["period"] in periods]
    ranked = sorted(((balanced_accuracy(subset, feats, w, dc)[0], w, dc)
                     for w in GRID for dc in DECISION_GRID), key=lambda a: -a[0])
    best = ranked[0]
    return {"weights": best[1], "decision": best[2], "balanced": round(best[0], 4),
            "ranked": ranked, "rows": len(subset)}


def drift(records, start="2026-09", slope=0.06):
    """FICTIONAL data drift: commercial consumption grows by `slope` every period."""
    b = PERIODS.index(start)
    return [dict(k, m3=round(k["m3"] * (1.0 + slope * (PERIODS.index(k["period"]) - b + 1)), 4))
            if PERIODS.index(k["period"]) >= b and k["type"] == "commercial" else dict(k)
            for k in records]


ALL = rows()
FEATS = features(ALL)
BASELINE = train(ALL, FEATS, PERIODS[1:9])
PRODUCTION = drift(ALL)                   # the input has drifted in production
FEATS_D = features(PRODUCTION)
monitored_rows = [k for k in ALL if k["period"] in MONITORED]
labels = [label(FEATS[(k["subscriber"], k["period"])]) for k in monitored_rows]
print(f"baseline model {BASELINE['weights']}/{BASELINE['decision']}"
      f" , monitoring {len(monitored_rows)} rows , base class ratio {round(1 - sum(labels) / len(labels), 4)}")

RETRAINED = train(PRODUCTION, FEATS_D, PERIODS[5:11])   # the retraining search of the previous lesson
CANDIDATES = [{"name": name, "weights": s[1], "decision": s[2], "train": round(s[0], 4)}
              for name, s in zip("ABCD", RETRAINED["ranked"][:4])]


def diff(m, subset):
    """The accuracy difference of the candidate from the baseline model. Same rows , same label."""
    def correct(x):
        return sum(1 for k in subset
                   if predict(FEATS_D[(k["subscriber"], k["period"])], x["weights"], x["decision"])
                   == label(FEATS_D[(k["subscriber"], k["period"])]))
    return round((correct(m) - correct(BASELINE)) / len(subset), 4)


PER_PERIOD = {m["name"]: [diff(m, [k for k in PRODUCTION if k["period"] == d]) for d in MONITORED]
              for m in CANDIDATES}
print("cand  weights/decision      training balanced accuracy  prediction disagreement")
for m in CANDIDATES:
    disagree = sum(1 for k in PRODUCTION if k["period"] in MONITORED and
                   predict(FEATS_D[(k["subscriber"], k["period"])], m["weights"], m["decision"])
                   != predict(FEATS_D[(k["subscriber"], k["period"])], BASELINE["weights"],
                              BASELINE["decision"]))
    print(f"{m['name']}     {str(m['weights']) + '/' + str(m['decision']):<20}"
          f"  {m['train']:>26}  {disagree:>23}")
print("\nperiod       A        B        C        D")
for i, d in enumerate(MONITORED):
    print(f"{d}    " + "  ".join(f"{PER_PERIOD[m['name']][i]:+.4f}" for m in CANDIDATES))
overall = {m["name"]: diff(m, [k for k in PRODUCTION if k["period"] in MONITORED]) for m in CANDIDATES}
print("6 periods  " + "  ".join(f"{overall[m['name']]:+.4f}" for m in CANDIDATES))
print("band        " + "   ".join(f"{max(PER_PERIOD[m['name']]) - min(PER_PERIOD[m['name']]):.4f}"
                                  for m in CANDIDATES))

baseline model (0.6, 0.0, 4)/18 , monitoring 1200 rows , base class ratio 0.5342
cand  weights/decision      training balanced accuracy  prediction disagreement
A     (0.6, 0.0, 4)/18                            0.98                        0
B     (0.8, 0.4, 4)/34                          0.9741                       60
C     (1.0, 0.0, 8)/30                          0.9492                       33
D     (1.0, 0.2, 4)/34                          0.9434                       97

period       A        B        C        D
2026-07    +0.0000  +0.0250  -0.0200  -0.0150
2026-08    +0.0000  +0.0000  -0.0400  -0.0250
2026-09    +0.0000  -0.0150  -0.0350  -0.0500
2026-10    +0.0000  -0.0150  -0.0250  -0.0450
2026-11    +0.0000  -0.0100  -0.0250  -0.0400
2026-12    +0.0000  -0.0150  -0.0200  -0.0400
6 periods  +0.0000  -0.0050  -0.0275  -0.0358
band        0.0000   0.0400   0.0200   0.0350

Column A is entirely zero, and this is not a measurement result but a result of definition: A’s prediction disagreement is 0, that is, the two models say the same thing on every row. The difference between them cannot be measured at any set size, because there is no difference to measure. This is the first question to ask before starting the experiment.

Column B is the real finding. Over the six periods B is 0.0050 worse than the baseline model — exactly one step of a single period’s resolution. But the period values move in a band 0.0400 wide, between +0.0250 and −0.0150, and change sign. A single-period experiment run on 2026-07 declares B five resolution steps better than the baseline model. The number is not wrong; in that period it really is so. What is wrong is drawing a ranking out of that number.

For C and D the band is wide too (0.0200 and 0.0350) but the sign never turns in any period. Both are below the baseline model in every period, and a single-period experiment ranks them correctly.

The Minimum Detectable Effect

The second block does two things. First it enlarges the experiment set and follows where the difference settles; then it looks for the limit by sweeping all 135 candidates in the grid.

# This block continues the CANDIDATES, diff, MONITORED, PRODUCTION, BASELINE,
# GRID and DECISION_GRID definitions of the first block.
print("period count   rows  resolution        B        C        D")
for n in range(1, 7):
    sub = [k for k in PRODUCTION if k["period"] in MONITORED[:n]]
    print(f"{n:>12}  {len(sub):>5}  {1 / len(sub):>10.4f}  " +
          "  ".join(f"{diff(m, sub):+.4f}" for m in CANDIDATES[1:]))

# All 135 candidates: does a single-period experiment rank the 6-period result correctly
SETS = {d: [k for k in PRODUCTION if k["period"] == d] for d in MONITORED}
ALL_ROWS = [k for k in PRODUCTION if k["period"] in MONITORED]
consistent, flipping = [], []
for w in GRID:
    for dc in DECISION_GRID:
        m = {"weights": w, "decision": dc}
        h = diff(m, ALL_ROWS)
        if h == 0:
            continue
        sign = (h > 0) - (h < 0)
        ps = [diff(m, SETS[d]) for d in MONITORED]
        (consistent if all((p > 0) - (p < 0) == sign for p in ps) else flipping).append(abs(h))
print(f"\ncandidates with nonzero 6-period diff: {len(consistent) + len(flipping)}")
print(f"same sign in all six periods: candidates {len(consistent)}"
      f" , smallest diff {min(consistent):.4f}")
print(f"sign flips from period to period: candidates {len(flipping)}"
      f" , largest diff {max(flipping):.4f}")

period count   rows  resolution        B        C        D
           1    200      0.0050  +0.0250  -0.0200  -0.0150
           2    400      0.0025  +0.0125  -0.0300  -0.0200
           3    600      0.0017  +0.0033  -0.0317  -0.0300
           4    800      0.0013  -0.0013  -0.0300  -0.0338
           5   1000      0.0010  -0.0030  -0.0290  -0.0350
           6   1200      0.0008  -0.0050  -0.0275  -0.0358

candidates with nonzero 6-period diff: 134
same sign in all six periods: candidates 133 , smallest diff 0.0275
sign flips from period to period: candidates 1 , largest diff 0.0050

The table above shows the resolution improving with the set size: 0.0050 on 200 rows, 0.0008 on 1200 rows. No difference below this can even be represented; it is the smallest nonzero number a 1200-row experiment can give. B’s sign turns in the fourth period, that is at 800 rows, and even there its value is −0.0013, exactly one step of that set. C and D carry the right sign from the first period on, and enlarging only settles the value.

The sweep below gives the limit. Of the candidates in the grid, 134 have a nonzero 6-period difference. 133 of them carry the same sign in all six of six periods, that is, they are ranked correctly whichever single period they are measured in; the smallest difference of this group is 0.0275. The single candidate that moves the sign is B, and its difference is 0.0050. What comes out is an interval: the minimum difference a single-period experiment can confidently distinguish on this set lies somewhere between 0.0050 and 0.0275. It cannot be stated more narrowly because the grid contains no candidate in this interval, and it should not be stated; the uncertainty of the measure is part of the measure.

The practical counterpart is single: every comparison that declares a difference smaller than 0.0275 with a single-period experiment is inconclusive, and B is the marked example of this. Why such a large threshold is needed is written in the prediction disagreement column. B and the baseline model say the same thing on 1140 of the 1200 rows; the real sample of the experiment is not 1200 but the 60 rows on which they disagree. When two predictions disagree only one of them can be right, so the six-row total difference means that in 33 of these sixty the baseline is right and in 27 B is right. The denominator is 1200 but the information comes from 60 rows; the 1140 rows that do not disagree add nothing to the experiment and only make the difference look small. The minimum difference an experiment can distinguish is limited not by the size of the set but by the number of rows on which the two models disagree.

Summary

• The same candidate is measured at −0.0050 over six periods and between +0.0250 and −0.0150 in single periods; its band is 0.0400 and its sign turns. The base class ratio is 0.5342, the period resolution is 0.0050, the resolution of the six periods is 0.0008.
• A candidate whose prediction disagreement is 0 cannot be distinguished by any experiment, because there is no difference; this is the question to ask before the experiment.
• 133 of the grid’s 134 candidates carry the same sign in every period and the smallest difference of this group is 0.0275; the difference of the single candidate that moves the sign is 0.0050. The minimum detectable effect lies between these two numbers and the grid contains no candidate that would narrow the interval.
• A comparison that declares a difference smaller than 0.0275 with a single period is inconclusive. Enlarging the set lowers the resolution to 0.0008 but turns B’s sign only in the fourth period, at 800 rows.
• The real sample of the experiment is not 1200 rows but the 60 rows on which the two models disagree; the total difference comes from the baseline being right in 33 of these sixty and the candidate in 27, and the remaining 1140 rows only enlarge the denominator.

Next Step

The experiment showed how expensive comparing two models is: for a distinguishable difference either the disagreement must be large or many periods must be waited for. This sharpens the question behind it even more: why is the new model being trained. Because an alert fired, because the calendar came, or because accuracy really dropped. The next lesson runs three kinds of triggers in two separate degradation scenarios, counts how many models each one causes to be trained and writes the accuracy gained next to it. On the data with a data drift, all three retraining windows give the same number, and that number is the old model’s number.