Model Auditability

Over three lessons the model was measured in the shadow, given to the canary, compared by experiment and retrained. At every step a new weight, a new decision threshold and a new training window came out. The only thing not asked is to whom a single prediction given to a single subscriber in the period 2026-12 will be put months later: which model produced it, with which feature values, with which decision threshold.

This question is separate from the model’s interpretability. Which feature drives a model’s decision, how it behaves on subgroups and how it will be explained was measured in the Model Evaluation, Interpretability and Ethics course of the Machine Learning and Data Science curriculum and is not repeated here. The question here is production’s question: which model was running. Interpretability works when the model is at hand; the audit trail is the record itself that says which model it was. This lesson builds that record, counts the search made when it is missing and writes the cost of the record.

• RG31. The source, the pipeline, the model and the release are fictional; no model registry, monitoring product or deployment tool is called. Seed 20260218.
• RG32. The label is tied to the rule subscriber_mean + 6 x commercial > 30, and the decision threshold is a separate setting. In the monitored periods the base class ratio is 0.5342, there are 200 rows per period and the step of accuracy is 0.0050.
• RG33. The audited period is 2026-12, and in that period two models are open: the old model and the candidate rolled out with a canary share of 0.10.
• RG34. The decision record is a single row for every prediction and carries ten fields: period, subscriber, model, weights, decision threshold, three feature values, score and prediction.
• RG35. The search cost is counted in rows read; raw time is not written. The search without a record reads 2 rows from the release record, then the raw rows to recompute the feature.
• RG36. The storage cost is counted in fields: the number of records times the number of fields.

What the Decision Record Carries

The first block sets up the setup, runs the 2026-12 release with a canary share of 0.10 and produces a decision record for every prediction.

# FICTIONAL. The meter readings of the measurement network, the consumption stage of the
# pipeline, the feature path and the three-weight threshold model are generated inside
# the lesson; no learning library, deployment tool, experiment platform or monitoring
# product is called.
# Seed 20260218.
SEED, M32 = 20260218, 0xFFFFFFFF


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


REGION = ["east", "west", "north", "south", "central"]
TYPE = ["residential", "commercial"]
PERIODS = [f"2026-{a:02d}" for a in range(1, 13)]
SUBSCRIBERS = {}
for i in range(200):
    r = rng(SEED + 37 * i)
    SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)],
                                    "start": 10000 + int(r() * 60000)}


def rows():
    """Source readings and consumption from the period difference: the first three stages of the pipeline."""
    records = []
    for i, (sub, a) in enumerate(SUBSCRIBERS.items()):
        e = a["start"]
        for p in PERIODS:
            r = rng(SEED + 101 * i + 7 * PERIODS.index(p))
            e += 4 + int(r() * 46)
            records.append({"subscriber": sub, "period": p, "region": a["region"], "type": a["type"],
                            "index": e})
    t = {(k["subscriber"], k["period"]): k["index"] for k in records}
    return [dict(k, m3=k["index"] - t[(k["subscriber"], PERIODS[PERIODS.index(k["period"]) - 1])])
            for k in records if PERIODS.index(k["period"])]


def features(records):
    """Independent of the weights and computed ONCE; it sees the full history."""
    sub, reg = {}, {}
    for k in sorted(records, key=lambda k: PERIODS.index(k["period"])):
        sub.setdefault(k["subscriber"], []).append((PERIODS.index(k["period"]), k["m3"]))
        reg.setdefault(k["region"], []).append((PERIODS.index(k["period"]), k["m3"]))

    def mean(series, i):
        s = [v for d, v in series if d < i]
        return round(sum(s) / len(s), 4) if s else 0.0

    return {(k["subscriber"], k["period"]): {
        "subscriber_mean": mean(sub[k["subscriber"]], PERIODS.index(k["period"])),
        "region_mean": mean(reg[k["region"]], PERIODS.index(k["period"])),
        "commercial": 1 if k["type"] == "commercial" else 0} for k in records}


RULE_THRESHOLD, RULE_COMMERCIAL = 30, 6   # the RULE that generates the label , in other words the concept
MONITORED = PERIODS[6:]                   # six periods monitored in production
GRID = [(a, b, c) for a in (0.6, 0.8, 1.0) for b in (0.0, 0.2, 0.4) for c in (0, 4, 8)]
DECISION_GRID = (18, 22, 26, 30, 34)      # the decision threshold is A SEPARATE setting


def label(o, threshold=RULE_THRESHOLD):
    return 1 if o["subscriber_mean"] + RULE_COMMERCIAL * o["commercial"] > threshold else 0


def score(o, w):
    return w[0] * o["subscriber_mean"] + w[1] * o["region_mean"] + w[2] * o["commercial"]


def predict(o, w, decision):
    return 1 if score(o, w) > decision else 0


def balanced_accuracy(subset, feats, w, decision):
    """Training criterion: the mean of recall and specificity."""
    tp = tn = fp = fn = 0
    for k in subset:
        o = feats[(k["subscriber"], k["period"])]
        p, g = predict(o, w, decision), label(o)
        if p and g:
            tp += 1
        elif p:
            fp += 1
        elif g:
            fn += 1
        else:
            tn += 1
    recall = tp / (tp + fn) if tp + fn else 0.0
    spec = tn / (tn + fp) if tn + fp else 0.0
    return (recall + spec) / 2, round(recall, 4), round(spec, 4)


def train(records, feats, periods):
    """Weights and decision threshold are searched together: 27 x 5 = 135 candidates."""
    subset = [k for k in records if k["period"] in periods]
    ranked = sorted(((balanced_accuracy(subset, feats, w, dc)[0], w, dc)
                     for w in GRID for dc in DECISION_GRID), key=lambda a: -a[0])
    best = ranked[0]
    return {"weights": best[1], "decision": best[2], "balanced": round(best[0], 4),
            "ranked": ranked, "rows": len(subset)}


def drift(records, start="2026-09", slope=0.06):
    """FICTIONAL data drift: commercial consumption grows by `slope` every period."""
    b = PERIODS.index(start)
    return [dict(k, m3=round(k["m3"] * (1.0 + slope * (PERIODS.index(k["period"]) - b + 1)), 4))
            if PERIODS.index(k["period"]) >= b and k["type"] == "commercial" else dict(k)
            for k in records]


ALL = rows()
FEATS = features(ALL)
BASELINE = train(ALL, FEATS, PERIODS[1:9])
PRODUCTION = drift(ALL)                   # the input has drifted in production
FEATS_D = features(PRODUCTION)
bal, recall, spec = balanced_accuracy([k for k in ALL if k["period"] in PERIODS[1:9]], FEATS,
                                      BASELINE["weights"], BASELINE["decision"])
monitored_rows = [k for k in ALL if k["period"] in MONITORED]
labels = [label(FEATS[(k["subscriber"], k["period"])]) for k in monitored_rows]
print(f"baseline model {BASELINE['weights']} decision threshold {BASELINE['decision']}"
      f" balanced accuracy {BASELINE['balanced']} recall {recall:.4f} specificity {spec:.4f}")
print(f"monitoring {len(monitored_rows)} rows , base class ratio {round(1 - sum(labels) / len(labels), 4)}"
      f" , 200 rows per period , accuracy step 0.0050")


# This block continues the ALL, FEATS, PRODUCTION, FEATS_D, BASELINE, MONITORED, PERIODS,
# SUBSCRIBERS, label, score and predict definitions of the first block.
CANDIDATE = {"name": "candidate", "weights": (0.8, 0.4, 4), "decision": 34}
OLD = dict(BASELINE, name="old")
SHARE, LAST = 0.10, "2026-12"
last_rows = [k for k in PRODUCTION if k["period"] == LAST]
N_CANARY = int(len(last_rows) * SHARE)
FIELDS = ("period", "subscriber", "model", "weights", "decision_threshold", "subscriber_mean",
          "region_mean", "commercial", "score", "prediction")


def decision_record(subset, feats, n_canary):
    """A single row for every prediction: which model , which feature , which threshold."""
    record = {}
    for j, k in enumerate(subset):
        o = feats[(k["subscriber"], k["period"])]
        m = CANDIDATE if j < n_canary else OLD
        record[(k["subscriber"], k["period"])] = dict(
            zip(FIELDS, (k["period"], k["subscriber"], m["name"], m["weights"], m["decision"],
                         o["subscriber_mean"], o["region_mean"], o["commercial"],
                         round(score(o, m["weights"]), 4),
                         predict(o, m["weights"], m["decision"]))))
    return record


RECORD = decision_record(last_rows, FEATS_D, N_CANARY)
disagree = [k["subscriber"] for k in last_rows
            if predict(FEATS_D[(k["subscriber"], LAST)], OLD["weights"], OLD["decision"])
            != predict(FEATS_D[(k["subscriber"], LAST)], CANDIDATE["weights"], CANDIDATE["decision"])]
print(f"{LAST} release: old {OLD['weights']}/{OLD['decision']} ,"
      f" candidate {CANDIDATE['weights']}/{CANDIDATE['decision']} , canary share {SHARE}")
print(f"requests {len(last_rows)} , sent to canary {N_CANARY} ,"
      f" predictions where the two models disagree {len(disagree)}")
print("decision record fields:", " ".join(FIELDS))
print("example:", " ".join(str(v) for v in RECORD[("A001", LAST)].values()))

baseline model (0.6, 0.0, 4) decision threshold 18 balanced accuracy 0.9864 recall 1.0000 specificity 0.9728
monitoring 1200 rows , base class ratio 0.5342 , 200 rows per period , accuracy step 0.0050
2026-12 release: old (0.6, 0.0, 4)/18 , candidate (0.8, 0.4, 4)/34 , canary share 0.1
requests 200 , sent to canary 20 , predictions where the two models disagree 11
decision record fields: period subscriber model weights decision_threshold subscriber_mean region_mean commercial score prediction
example: 2026-12 A001 candidate (0.8, 0.4, 4) 34 35.502 27.6233 1 43.4509 1

The example row is an explanation on its own: the 2026-12 prediction of the subscriber A001 was produced by the candidate model, with the weights (0.8, 0.4, 4) and the decision threshold 34; the subscriber mean is 35.502, the region mean is 27.6233, the subscriber type is commercial and the score is 43.4509. Since the score exceeds the threshold, the prediction is 1. The record requires no recomputation and says which model was running.

The Search Without a Record

The second block searches the same two hundred predictions without a record: the two models open in that period are found from the release record, the features are recomputed from the raw rows, and which models agree with the observed prediction is counted.

# This block continues the RECORD, FIELDS, OLD, CANDIDATE, last_rows, LAST, N_CANARY,
# PRODUCTION, FEATS_D, SUBSCRIBERS and PERIODS definitions of the second block.
def search_without_record(subscriber, period, records, observed):
    """Without a record: the models from the release record , the features from the raw rows."""
    i, r = PERIODS.index(period), SUBSCRIBERS[subscriber]["region"]
    sub = [k["m3"] for k in records if k["subscriber"] == subscriber and PERIODS.index(k["period"]) < i]
    reg = [k["m3"] for k in records if k["region"] == r and PERIODS.index(k["period"]) < i]
    o = {"subscriber_mean": round(sum(sub) / len(sub), 4), "region_mean": round(sum(reg) / len(reg), 4),
         "commercial": 1 if SUBSCRIBERS[subscriber]["type"] == "commercial" else 0}
    matching = [m["name"] for m in (OLD, CANDIDATE) if predict(o, m["weights"], m["decision"]) == observed]
    return 2 + len(sub) + len(reg), matching    # 2 rows of release record + raw rows


read_with = read_without = single_with = single_without = 0
for k in last_rows:
    rec = RECORD[(k["subscriber"], LAST)]
    read_with, single_with = read_with + 1, single_with + 1   # search with a record: one row , one model
    n_read, matching = search_without_record(k["subscriber"], LAST, PRODUCTION, rec["prediction"])
    read_without, single_without = read_without + n_read, single_without + (len(matching) == 1)
print("search                     rows read  per prediction   model identified    unidentified")
for name, rd, one in (("with decision record", read_with, single_with),
                      ("without decision record", read_without, single_without)):
    print(f"{name:<23} {rd:>12}  {rd / len(last_rows):>14.2f}  {one:>17}  {len(last_rows) - one:>14}")

over_threshold = sum(1 for c, rec in RECORD.items() if rec["prediction"] == 1)
commercial_decisive = sum(1 for c, rec in RECORD.items() if rec["prediction"] == 1
                          and rec["score"] - rec["weights"][2] * rec["commercial"] <= rec["decision_threshold"])
print(f"\nreason read from the decision record: predictions over the threshold {over_threshold} ,"
      f" of which {commercial_decisive} have the subscriber type decisive on its own")
record_count = len(MONITORED) * len(last_rows)
source_fields = len(SUBSCRIBERS) * len(PERIODS) * 6
print(f"had the six monitored periods been recorded: {record_count} records x {len(FIELDS)} fields ="
      f" {record_count * len(FIELDS)} stored fields")
print(f"    source readings {len(SUBSCRIBERS) * len(PERIODS)} rows x 6 fields = {source_fields} fields")
print(f"    decision record to source ratio {record_count * len(FIELDS) / source_fields:.4f}")

search                     rows read  per prediction   model identified    unidentified
with decision record             200            1.00                200               0
without decision record        84740          423.70                 11             189

reason read from the decision record: predictions over the threshold 109 , of which 57 have the subscriber type decisive on its own
had the six monitored periods been recorded: 1200 records x 10 fields = 12000 stored fields
    source readings 2400 rows x 6 fields = 14400 fields
    decision record to source ratio 0.8333

The rows read column gives the expected difference: 1.00 rows per prediction with the decision record, 423.70 rows without it. The factor of four hundred between them comes from the scope of the feature path; region_mean cannot be computed for a single subscriber, it asks for the history of all subscribers in the region. This much is a difference in cost, and it closes if enough data is read.

The last two columns show what does not close. With the decision record 200 of 200 predictions are tied to a single model. In the search without a record this number falls to 11, and the model of 189 predictions cannot be determined. The reason is written in the table: in the canary period the two models say the same thing for 189 of the 200 predictions, so the observed output agrees with both. Recomputation brings the feature back but does not bring back the information of which model was running; that information is the product not of a measurable trace but of a written record.

The inverse here is worth noting. The only group the search without a record can resolve is the 11 predictions on which the two models disagree — that is, the place where the audit works by itself is the place where the candidate really departs from the old model. The small shares of a canary are chosen precisely to reduce this disagreement. The same choice that shrinks the risk also shrinks retrospective auditability; the trade-off experienced with the resolution of the measurement in the first lesson meets us again here with the audit.

The explanations for these two predictions are not the same. Two models that give the same output carry different weights, different decision thresholds and different scores; when it cannot be chosen which one was valid, there is not one explanation but two incompatible candidate explanations. The answer to be given when a contest arrives cannot be either one of these two.

The record itself does not come free. Had all six monitored periods been recorded, it would come to 1200 records and 12,000 stored fields; 0.8333 times the source readings, that is, a load nearly as large as the raw data the pipeline carries. In return the record gives not only the model but also the reason in a single pass: of the 109 predictions that exceed the threshold, in 57 the subscriber type is decisive on its own, that is, when the commercial contribution is removed from the score the threshold is not exceeded. This number is read from the score and commercial fields in the record; without the record, 84,740 rows must be read again for the same number, and which model the result belongs to for 189 predictions is still unknown.

Summary

• The decision record ties a prediction in ten fields: model, weights, decision threshold, three feature values, score and prediction. Rows read per prediction: 1.00.
• The search without a record reads 84,740 rows for the same two hundred predictions, which comes to 423.70 per prediction, and even so the model of 189 predictions cannot be determined; the 11 predictions that can be determined are the ones on which the two models disagree.
• Recomputation brings the feature back, it does not bring back which model was running. Interpretability works when the model is at hand; the audit trail is the written record that says which model it was, and it cannot be produced if it was not recorded.
• Because the small share of a canary reduces disagreement, the choice that shrinks the risk also shrinks retrospective auditability; two models that give the same output carry different scores and different thresholds, so a prediction that cannot be determined has not one but two incompatible explanations.
• The cost of the record is 12,000 stored fields for six periods and is 0.8333 times the source readings. In return the reason is read in a single pass: in 57 of the 109 predictions that exceed the threshold the subscriber type is decisive on its own. The base class ratio is 0.5342 and the step of accuracy is 0.0050.

Next Step

Over four lessons the model was released, compared, retrained, and every prediction it produced was recorded. All of these measures face inward: they were kept so that the team could make its own decision. The thing not asked is what the same setup owes to the outside — a decision’s justification being requested, a prediction being contested, a record being expected to be kept for a certain period. The next lesson counts these expectations by their classes and writes the engineering counterpart of each class: how many additional fields, how many stored records, and which of them are already met by the decision record built in this lesson.