--- title: 'Compliance Requirements' source: 'https://academia.sh/en/courses/model-monitoring/compliance-requirements' course: 'Model Monitoring and Governance' language: en updated: '2026-10-21T05:03:02+00:00' license: 'CC BY-SA 4.0' objectives: - 'Translates each of the four obligation classes into fields in the decision record and a number of stored records.' - 'Separates which class is met by the existing record and which one really asks for new fields.' - 'Measures, through late-arriving rows, in how many decisions recomputation differs without a decision record.' - 'Computes the number of records where the retention policy collides with the other three classes.' --- # Compliance Requirements Counting how many additional fields and how many stored records each of the four obligation classes owed to the outside asks for: the baseline record keeps three fields per decision, and 36 of the 1200 decisions differ when recomputed from today's source, because the 80 late-arriving rows changed the features of 239 decisions. The record-keeping obligation adds four fields per decision and makes recomputation 1200 of 1200; the explanation obligation adds zero fields, because all five explanation values are derived from those four fields; the right to contest adds three fields only on the 108 negative decisions within one point of the threshold, and those 108 decisions cover 26 of the model's 26 wrong predictions; the retention policy adds no fields at all, it deletes records. With a retention of three periods 600 decision records are dropped, and the number of decisions deleted while the contest window is still open is 600. In four lessons the model was released, measured in the shadow and the canary, compared, and its decisions were recorded. The four share one trait: the addressee of the measure is **internal**. The team that sets the canary share, the minimum detectable effect and the room the decision record takes also decides whether they are enough. This lesson moves the addressee outside. A decision a model produces creates a debt toward its subject, and what that debt is is not left to the team's preference. The lesson takes it as **four obligation classes** — record-keeping, explanation, contest and retention period — and counts one engineering counterpart for each: **how many additional fields per decision and how many stored records.** - **RG41.** The source, the pipeline, the feature path, the model and the decision log are **fictional**. No name of a regulation, standard, institution, country, certificate or audit body is written; no product, service or library is called. Seed 20260218. - **RG42.** Obligations are referred to by their **classes**, and what is measured is not their sources but **what they ask of the system**: record-keeping asks that a decision be recomputable, explanation that the justification of a single decision be shown, the right to contest that the result of the review be written, and the retention period that the record be **deleted after a certain time**. - **RG43.** The baseline is **the setup without obligations**: the decision log keeps only the three fields needed to present, and it is assumed that nothing is owed to the outside. - **RG44.** The model and the label come from the shared definition: weights `(0.6, 0.0, 4)`, decision threshold 18, balanced accuracy 0.9864. Monitoring is six periods, 1200 decisions, **base class ratio 0.5342**, and the smallest step of period accuracy is 0.0050. - **RG45.** The record is tested with a **fictional late-arriving reading**: in the period `2026-04` the meters of 40 subscribers were not read on time, and the rows were added to the source **after** the decisions were made. The source is 2120 rows at decision time and 2200 rows today. - **RG46.** The contest window is **three periods**: a decision can be contested at the latest at the end of the third period. The retention period is also written in periods. Audit trail, ownership and retention policy were measured in the Introduction to Data Engineering course and the Governance, Risk and Compliance course; their procedures are **not repeated**. The previous lesson built the decision record. This lesson takes that record as **data** and asks one question: what does each of the four classes put on top of it. ## Translating an Obligation into a Record The first block sets up the source, the consumption stage of the pipeline, the feature path and the model; then it produces the same decisions twice — once from the source at decision time, once from today's source. ```python # FICTIONAL. The meter readings of the measurement network, the first three stages of the # pipeline, the feature path, the threshold model and the decision log are generated inside # the lesson; no regulation , audit body , product or service is called. Seed 20260218. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value REGION = ["east", "west", "north", "south", "central"] TYPE = ["residential", "commercial"] PERIODS = [f"2026-{a:02d}" for a in range(1, 13)] SUBSCRIBERS = {} for i in range(200): r = rng(SEED + 37 * i) SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)], "start": 10000 + int(r() * 60000)} def generate_source(): records = [] for i, (sub, a) in enumerate(SUBSCRIBERS.items()): e = a["start"] for p in PERIODS: r = rng(SEED + 101 * i + 7 * PERIODS.index(p)) e += 4 + int(r() * 46) records.append({"subscriber": sub, "period": p, "region": a["region"], "type": a["type"], "index": e, "unit": "m3"}) return records def consumption(records): """Consumption from the period difference. If there is no previous period the row cannot be computed.""" table = {(k["subscriber"], k["period"]): k for k in records} out = [] for k in records: i = PERIODS.index(k["period"]) previous = table.get((k["subscriber"], PERIODS[i - 1])) if i else None if previous is not None: out.append(dict(k, m3=int(k["index"]) - int(previous["index"]))) return out def features(records): """Independent of the weights , computed once.""" ordered = sorted(records, key=lambda k: PERIODS.index(k["period"])) sub, reg = {}, {} for k in ordered: sub.setdefault(k["subscriber"], []).append((PERIODS.index(k["period"]), k["m3"])) reg.setdefault(k["region"], []).append((PERIODS.index(k["period"]), k["m3"])) def average(series, i): chosen = [v for d, v in series if d < i] return round(sum(chosen) / len(chosen), 4) if chosen else 0.0 return {(k["subscriber"], k["period"]): { "subscriber_mean": average(sub[k["subscriber"]], PERIODS.index(k["period"])), "region_mean": average(reg[k["region"]], PERIODS.index(k["period"])), "commercial": 1 if k["type"] == "commercial" else 0} for k in records} GRID = [(a, b, c) for a in (0.6, 0.8, 1.0) for b in (0.0, 0.2, 0.4) for c in (0, 4, 8)] DECISION_GRID = (18, 22, 26, 30, 34) MONITORED = PERIODS[6:] RULE_THRESHOLD, RULE_COMMERCIAL = 30, 6 def score(o, w): return w[0] * o["subscriber_mean"] + w[1] * o["region_mean"] + w[2] * o["commercial"] def predict(o, w, decision): return 1 if score(o, w) > decision else 0 def label(o, threshold=RULE_THRESHOLD): return 1 if o["subscriber_mean"] + RULE_COMMERCIAL * o["commercial"] > threshold else 0 def train(records, feats): """The training criterion is balanced accuracy; the decision threshold is A SEPARATE setting and is searched.""" subset = [k for k in records if k["period"] in PERIODS[1:9]] best = (-1.0, None, None) for w in GRID: for decision in DECISION_GRID: tp = tn = fp = fn = 0 for k in subset: o = feats[(k["subscriber"], k["period"])] p, g = predict(o, w, decision), label(o) tp, fp, fn, tn = (tp + (p and g), fp + (p and not g), fn + (not p and g), tn + (not p and not g)) total = (tp / (tp + fn) if tp + fn else 0.0) + (tn / (tn + fp) if tn + fp else 0.0) if total / 2 > best[0]: best = (total / 2, w, decision) return {"weights": best[1], "decision_threshold": best[2], "balanced": round(best[0], 4)} # FICTIONAL late-arriving reading: in 2026-04 the meters of 40 subscribers were not read on time # and the rows were added to the source AFTER the decisions were made. RAW = generate_source() AT_DECISION = consumption([k for k in RAW if not (k["period"] == "2026-04" and k["subscriber"] <= "A040")]) TODAY = consumption(RAW) FEATS_A, FEATS_B = features(AT_DECISION), features(TODAY) M = train(TODAY, FEATS_B) MONITORED_ROWS = [k for k in TODAY if k["period"] in MONITORED] share1 = sum(label(FEATS_B[(k["subscriber"], k["period"])]) for k in MONITORED_ROWS) / len(MONITORED_ROWS) print(f"model: weights {M['weights']} , decision threshold {M['decision_threshold']} ," f" balanced accuracy {M['balanced']}") print(f"monitoring periods {len(MONITORED)} , decisions {len(MONITORED_ROWS)} ," f" base class ratio {round(max(share1, 1 - share1), 4)}") # Baseline record: the least that is needed to present. BASE_FIELDS = ("subscriber", "period", "prediction") # The counterpart of the record-keeping obligation: EVERYTHING the decision READ is stored. S1_FIELDS = ("model_version", "subscriber_mean", "region_mean", "commercial") S1_MODEL_FIELDS = ("weights", "decision_threshold") VERSION = {"s1": {"weights": M["weights"], "decision_threshold": M["decision_threshold"]}} ledger = [] for k in [k for k in AT_DECISION if k["period"] in MONITORED]: o = FEATS_A[(k["subscriber"], k["period"])] ledger.append({"subscriber": k["subscriber"], "period": k["period"], "model_version": "s1", "prediction": predict(o, M["weights"], M["decision_threshold"]), "subscriber_mean": o["subscriber_mean"], "region_mean": o["region_mean"], "commercial": o["commercial"]}) ledger_same = sum(1 for r in ledger if predict(r, VERSION[r["model_version"]]["weights"], VERSION[r["model_version"]]["decision_threshold"]) == r["prediction"]) feature_diff = sum(1 for r in ledger if FEATS_B[(r["subscriber"], r["period"])]["subscriber_mean"] != r["subscriber_mean"]) decision_diff = sum(1 for r in ledger if predict(FEATS_B[(r["subscriber"], r["period"])], M["weights"], M["decision_threshold"]) != r["prediction"]) print(f"\nsource rows: at decision time {len(AT_DECISION)} , today {len(TODAY)}") print(f"recomputation from the ledger : exact {ledger_same}/{len(ledger)} , differing" f" {len(ledger) - ledger_same}") print(f"recomputation from today's source: exact {len(ledger) - decision_diff}/{len(ledger)} ," f" differing {decision_diff} , feature values changed {feature_diff}") ``` ``` model: weights (0.6, 0.0, 4) , decision threshold 18 , balanced accuracy 0.9864 monitoring periods 6 , decisions 1200 , base class ratio 0.5342 source rows: at decision time 2120 , today 2200 recomputation from the ledger : exact 1200/1200 , differing 0 recomputation from today's source: exact 1164/1200 , differing 36 , feature values changed 239 ``` The last two lines show why the record-keeping obligation is a question of **fields**. When the decision log carries the value of the features at the time of the decision, 1200 of 1200 decisions are recomputed exactly. When the same decisions are to be reproduced from today's source, the features of 239 decisions have changed and **36 decisions come out different**. Nobody touched the source; only the 80 late rows have settled into place. This is a measurement topic before it is a compliance topic. When the justification of a decision is asked, "we look at the source and recompute" is no answer unless the **state of the source at decision time** has been stored. The late-arriving reading is one of the six degradation classes of the pipeline and was measured as detection lag in the Introduction to Data Engineering course; what is new here is the **retrospective** face of the same degradation — the pipeline has recovered, the record has been corrupted. ## The Field Cost of the Four Classes The second block counts the four classes in a single table and sweeps the retention period. ```python # This block continues the PERIODS , MONITORED , FEATS_B , M , VERSION , ledger , score , # predict and label definitions of the first block. MARGIN, CONTEST_WINDOW, RETENTION = 1.0, 3, 3 def explanation(r): """DERIVED from the fields of class 1; asks for no new field.""" w = VERSION[r["model_version"]]["weights"] e = VERSION[r["model_version"]]["decision_threshold"] p = score(r, w) return {"subscriber_contribution": round(w[0] * r["subscriber_mean"], 4), "region_contribution": round(w[1] * r["region_mean"], 4), "type_contribution": round(w[2] * r["commercial"], 4), "score": round(p, 4), "distance_to_threshold": round(p - e, 4)} contested = [r for r in ledger if r["prediction"] == 1 and abs(explanation(r)["distance_to_threshold"]) <= MARGIN] example = contested[0] print(f"example of a negative decision closest to the threshold: {example['subscriber']} {example['period']}" f" prediction {example['prediction']} actual {label(example)}") print(f"explanation {explanation(example)}") wrong_near = sum(1 for r in contested if label(r) != r["prediction"]) wrong_all = sum(1 for r in ledger if label(r) != r["prediction"]) n, nb, ns = len(ledger), len(BASE_FIELDS), len(S1_FIELDS) s1 = n * (nb + ns) + len(S1_MODEL_FIELDS) print(f"\nclass fields per decision stored values recomputation") print(f"baseline (obligation-free) {nb:>19} {n * nb:>14} {n - decision_diff:>9}/{n}") print(f"record-keeping {nb + ns:>19} {s1:>14} {n:>9}/{n}") print(f"explanation {nb + ns:>19} {s1:>14} {n:>9}/{n}") print(f"contest {str(nb + ns) + ' (+3)':>19} {s1 + 3 * len(contested):>14}" f" {n:>9}/{n}") kept = len([d for d in MONITORED if PERIODS.index(d) > len(PERIODS) - 1 - RETENTION]) * 200 print(f"retention ({RETENTION} periods) {nb + ns:>19} {kept * (nb + ns) + 2:>14} {kept:>9}/{n}") print(f"\nnegative decisions within {MARGIN} point of the threshold: {len(contested)}" f" (share {round(len(contested) / n, 4)}) ; of these wrong {wrong_near} ," f" wrong in the whole log {wrong_all}") print(f"monitoring accuracy {round((n - wrong_all) / n, 4)} , step {round(1 / 200, 4)}") print(f"\nretention (periods) records kept records deleted deleted while contest open") for R in (1, 2, 3, 6, 12): kept_periods = [d for d in MONITORED if PERIODS.index(d) > len(PERIODS) - 1 - R] broken = 0 for t in range(len(PERIODS)): for d in MONITORED: i = PERIODS.index(d) if i <= t - R and i + CONTEST_WINDOW >= t: broken += 200 print(f"{R:>19} {len(kept_periods) * 200:>13} {n - len(kept_periods) * 200:>15} {broken:>26}") ``` ``` example of a negative decision closest to the threshold: A002 2026-08 prediction 1 actual 1 explanation {'subscriber_contribution': 18.6, 'region_contribution': 0.0, 'type_contribution': 0, 'score': 18.6, 'distance_to_threshold': 0.6} class fields per decision stored values recomputation baseline (obligation-free) 3 3600 1164/1200 record-keeping 7 8402 1200/1200 explanation 7 8402 1200/1200 contest 7 (+3) 8726 1200/1200 retention (3 periods) 7 4202 600/1200 negative decisions within 1.0 point of the threshold: 108 (share 0.09) ; of these wrong 26 , wrong in the whole log 26 monitoring accuracy 0.9783 , step 0.005 retention (periods) records kept records deleted deleted while contest open 1 200 1000 2400 2 400 800 1400 3 600 600 600 6 1200 0 0 12 1200 0 0 ``` The table shows that the four classes are **not equal**. Only **record-keeping** really asks for new fields per decision: four fields, from 3600 stored values to 8402. **The explanation obligation adds zero fields.** The example row shows this on one decision: for subscriber `A002`'s `2026-08` decision the three contributions, the score and the distance to the threshold are written, and all five are derived from the fields of the record-keeping class. As long as the weights and the threshold are stored, the explanation is a **computation**, not a record. The **right to contest** is met in part: record-keeping has already paid for recomputing the decision, what is new is only the result of the review, and that is written not for all decisions but for **108** of them. The number 108 is the sharpest reading of the lesson. The negative decisions within one point of the threshold make up 0.0900 of the 1200 decisions, but **26 of the model's 26 wrong predictions** are inside this band. With a monitoring accuracy of 0.9783 and a period step of 0.0050, 26 wrong predictions are a measurable size. Because recall is 1.0000, all of the wrong predictions are **false positives**: subscribers who fell immediately above the decision threshold and are in reality low. **The engineering counterpart of the right to contest is to open nine percent of the decisions to review and in return be able to reach all of the wrong ones** — nearness to the boundary is not a coincidence, it is a structural property of every model with a threshold. ## Retention Cuts Across the Other Three The counterpart of the fourth class is unlike the other three, because it **adds no field**: the period is already recorded and the duration is one policy number. What the retention policy does is **delete records**, and the records it deletes are the ones the first three classes rely on. With a retention of three periods the log falls from 1200 decisions to 600, 8402 stored values drop to 4202 and **the justification of 600 decisions can no longer be asked**. The last lesson of the Introduction to Data Engineering course measured this on the data side — the retention policy leaves periods that cannot be audited retrospectively; on the decision side the same thing happens by the same mechanism. The last column of the sweep turns the collision into a number. With a contest window and a retention period of three periods each, one period's set of decisions is deleted **while the right to contest is still open** in every period: 600 decisions in total. At two periods it becomes 1400, at one 2400; at six it drops to zero. This number is not an implementation error, it is **two obligation classes cutting each other**: one asks for the record, the other deletes it, and both are in force in the same system at the same time. The measure of the trade-off is exactly this column — when the retention period is not **longer than the contest window**, the number of unmet obligations is greater than zero; equality is not enough. The rule that follows is independent of the source of the classes: **an obligation class counts as unmeasured unless its counterpart is written into a field in the decision record or into a record count.** ## Summary - When the decision log holds the features at decision time, 1200 of 1200 decisions are recomputed exactly; when computed from today's source the features of 239 decisions change and **36 decisions come out different**. Nobody touched the source; the 80 late-arriving rows have settled into place. - **Record-keeping** adds four fields per decision and raises the stored values from 3600 to 8402. It is the only one among the four classes that really asks for new fields. - **Explanation** adds zero fields: the three contributions, the score and the distance to the threshold are derived from the fields of record-keeping. As long as the weights and the threshold are stored, the explanation is a computation. - **Contest** adds three fields only on the 108 negative decisions within one point of the threshold; those 108 decisions are 0.0900 of 1200 and cover **all 26 of the model's 26 wrong predictions**. - **Retention** adds no fields, it deletes records: with three periods the log falls to 600 and the number of decisions deleted while the contest window is still open is 600. Two obligation classes cut each other in the same system. ## Next Step The counterparts of the obligations were written as fields and records; all of them either added something to what is stored or took something away from it. But no line of this lesson asked what keeping the model running itself costs. A model does two separate jobs: it is **retrained** period by period and it produces **inference** in every period. The two are written to the same budget and read as a single total. The next lesson splits that total in two and counts which item grows as the retraining frequency increases and which decision reading it as a single number hides.