Data Governance

In the previous lesson the search went down to the source and stopped there. Lineage said which rows were missing, wrote that the loss stage was source and fell silent. Why forty rows never arrived and who should be told the record could not say. The pipeline has five stages, six counting the source; how many carry an owner.

This lesson measures a gap: the distance between where a corruption is caught and where it can be repaired. The gap is read with three numbers: who the finding goes to, whether that role can make the repair, whether it can look at the evidence. The second measure reads the same gap on the time axis: how many periods can no longer be audited retroactively when the retention window narrows. All the numbers are counts; no ratio is printed.

• QP47. The source, the five-stage pipeline, the corruption classes and the ownership map are fictional and are defined within the lesson. No store, catalog product, orchestrator or policy text is called. Seed 20260218.
• QP48. An owner is the role class with the authority to change the output of a position; it is not a person or an institution. Access is which positions’ rows a role can read.
• QP49. A finding is born at the position where it is caught. If that position has no owner, the finding walks downstream to the first owned position; if none remains, it goes to no one.
• QP50. The repair point is the earliest position at which a corruption can be fixed; it need not be the position where the finding was received. The distance is the number of stages between the two.
• QP51. The detecting check and the detection lag were measured in the first lesson of this course; here they are taken as input and not measured again. Access control, the retention policy and the personal data procedure were measured in the Identity, Access and Cryptography and the Governance, Risk and Compliance courses of the Cybersecurity curriculum; they are not repeated.
• QP52. The retention window keeps the last N periods and deletes what came before. A period is retroactively auditable if its consumption row can be recomputed from the remaining source.

Ownership Map and the Path a Finding Walks

Ownership is not a document but a mapping: who can change the output of which position. The map below recognizes two roles; the remaining positions are empty. The gap is not contrived but a result of the pipeline; intermediate layers are no one’s product.

# FICTIONAL. The ownership, access and repair map is defined within the lesson; no
# regulation text, institution, catalog product or orchestrator is called.
POSITION = ["source", "screen", "dedupe", "consumption", "summary", "serve"]
# An owner is not a person but a role class; the value of an ownerless position is None.
OWNER = {"source": None, "screen": "pipeline team", "dedupe": "pipeline team",
         "consumption": None, "summary": None, "serve": "reporting team"}
ACCESS = {"pipeline team": {"screen", "dedupe", "consumption", "summary"},
          "reporting team": {"summary", "serve"}}
# The detecting check and the detection lag (stage) were measured in the first lesson of this course.
DETECTION = {"late_arrival": ("distribution", 4), "duplicated": ("uniqueness", 0),
             "schema_changed": ("schema", 0), "unit_changed": ("distribution", 4),
             "missing_partition": ("partition_period", 0), "type_drift": ("type", 0)}
# The earliest position at which the corruption can really be repaired.
REPAIR = {"late_arrival": "source", "duplicated": "dedupe", "schema_changed": "screen",
          "unit_changed": "source", "missing_partition": "source", "type_drift": "screen"}
# The classes the flawed check set never catches, again from the first lesson.
MISSED = {"late_arrival", "unit_changed", "missing_partition"}


def route(i):
    """A finding is born at the position where it is caught; if the position is ownerless it walks to the first owned one."""
    for j in range(i, len(POSITION)):
        if OWNER[POSITION[j]]:
            return j, OWNER[POSITION[j]]
    return None, None


def yn(x):
    return "yes" if x else "no"


print(f"{sum(1 for a in POSITION[1:] if OWNER[a])} of the five stages are owned ;"
      " the source position is outside the pipeline and ownerless")
print("\ncorruption                 detector      lag  position owner  finding goes to   repair"
      "  distance  authority  access   in flawed set")
authorized = accessed = 0
for cls, (det, lag) in DETECTION.items():
    j, recipient = route(lag)
    o = REPAIR[cls]
    can_fix, can_see = OWNER[o] == recipient, o in ACCESS[recipient]
    authorized, accessed = authorized + can_fix, accessed + can_see
    print(f"{cls:<17} {det:>17} {lag:>8}  {OWNER[POSITION[lag]] or 'none':>14}  {recipient:<17}"
          f" {o:<7} {abs(POSITION.index(o) - j):>8}  {yn(can_fix):>9}  {yn(can_see):>6}"
          f"  {'escapes' if cls in MISSED else 'caught':>14}")
ownerless = {c for c in REPAIR if OWNER[REPAIR[c]] is None}
print(f"\nthe owner receiving the finding can repair in {authorized} corruptions ,"
      f" {accessed} corruptions have access to the evidence")
print(f"{len(ownerless)} classes with an ownerless repair point , {len(MISSED)} classes the flawed set never catches"
      f" , same set: {yn(ownerless == MISSED)}")

3 of the five stages are owned ; the source position is outside the pipeline and ownerless

corruption                 detector      lag  position owner  finding goes to   repair  distance  authority  access   in flawed set
late_arrival           distribution        4            none  reporting team    source         5         no      no         escapes
duplicated               uniqueness        0            none  pipeline team     dedupe         1        yes     yes          caught
schema_changed               schema        0            none  pipeline team     screen         0        yes     yes          caught
unit_changed           distribution        4            none  reporting team    source         5         no      no         escapes
missing_partition  partition_period        0            none  pipeline team     source         1         no      no         escapes
type_drift                     type        0            none  pipeline team     screen         0        yes     yes          caught

the owner receiving the finding can repair in 3 corruptions , 3 corruptions have access to the evidence
3 classes with an ownerless repair point , 3 classes the flawed set never catches , same set: yes

The harshest column is the fourth, and it holds a single value: none. Although three of the five stages are owned, not one of the six corruptions is caught at an owned position. The reason is where the checks sit: of the seven checks, six run at the source and one at the summary layer, and both positions are ownerless. Ownership was written where the pipeline produces, not where the check looks. Owning a position and running a check there are two separate decisions; that they do not overlap is a measurable flaw.

Because the finding walks, it still finds a recipient, but the recipient is most often the wrong role. In three corruptions the pipeline team is both authorized and has access: duplicated is repaired in deduplication, schema_changed and type_drift in screening. In the other three the repair point is source, which has no owner. The missing_partition finding lands one stage away; late_arrival and unit_changed go five stages away, to the reporting team, which holds only a deviated served value.

The last line is the narrowest finding of this lesson. The three classes whose repair point is ownerless are exactly the same as the three the flawed check set never catches. The two lists came out independently, one from the coverage of the checks, the other from the role map. That they match shows the least audited position is the least owned position.

Retention Window and Retroactive Audit

The ownership gap has a second face in time. Deleted rows free space and also take away the basis of the retroactive audit. The block below runs the same fictional source with five retention windows.

# FICTIONAL. The meter readings of the measurement network are generated within the lesson. In an
# intact source the first two stages drop no rows, so consumption is computed directly from the source.
SEED, M32 = 20260218, 0xFFFFFFFF


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


REGION = ["east", "west", "north", "south", "central"]
TYPE = ["residential", "commercial"]
PERIODS = [f"2026-{a:02d}" for a in range(1, 13)]
SUBSCRIBERS = {}
for i in range(200):
    r = rng(SEED + 37 * i)
    SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)],
                                    "start": 10000 + int(r() * 60000)}


def generate_source():
    records = []
    for i, (sub, a) in enumerate(SUBSCRIBERS.items()):
        e = a["start"]
        for p in PERIODS:
            r = rng(SEED + 101 * i + 7 * PERIODS.index(p))
            e += 4 + int(r() * 46)
            records.append({"subscriber": sub, "period": p, "region": a["region"], "type": a["type"],
                            "index": e, "unit": "m3"})
    return records


def key_of(k):
    return (k["subscriber"], k["period"])


def s3_consumption(v):
    """Consumption from the period difference; a row cannot be computed if there is no previous period."""
    table, out = {key_of(k): k for k in v}, []
    for k in v:
        i = PERIODS.index(k["period"])
        prev = table.get((k["subscriber"], PERIODS[i - 1])) if i else None
        if prev is not None:
            out.append(dict(k, m3=k["index"] - prev["index"]))
    return out


def c_region_period(v):
    """Correct partition check: has every region been written in every period."""
    return {(k["region"], k["period"]) for k in v} != {(b, d) for b in REGION for d in PERIODS}


def retain(records, window):
    """Retention policy: only the last `window` periods are kept, the rest is deleted."""
    kept = set(PERIODS[len(PERIODS) - window:])
    return [k for k in records if k["period"] in kept], kept


source = generate_source()
# Every consumption row has two ancestors: its own period and the previous period.
LINKS = [x for k in s3_consumption(source)
         for x in (key_of(k), (k["subscriber"], PERIODS[PERIODS.index(k["period"]) - 1]))]
print(f"source {len(source)} rows , consumption rows {len(s3_consumption(source))} ,"
      f" consumption links {len(LINKS)}")
print("\n window  deleted   kept          recomputable   unauditable           2026-07  alarm  dangling")
for p in (12, 9, 6, 3, 1):
    remaining, kept = retain(source, p)
    recomputable = sum(1 for i, d in enumerate(PERIODS)
                       if i and d in kept and PERIODS[i - 1] in kept)
    keys = {key_of(k) for k in remaining}
    status = ("deleted" if "2026-07" not in kept else
              "ancestor deleted" if "2026-06" not in kept else "auditable")
    print(f"{p:>7}  {12 - p:>7}  {len(remaining):>5}  {recomputable:>20}"
          f"  {12 - recomputable:>12}  {status:>16}"
          f"  {yn(c_region_period(remaining)):>5}  {sum(1 for b in LINKS if b not in keys):>8}")

source 2400 rows , consumption rows 2200 , consumption links 4400

 window  deleted   kept          recomputable   unauditable           2026-07  alarm  dangling
     12        0   2400                    11             1         auditable     no         0
      9        3   1800                     8             4         auditable    yes      1000
      6        6   1200                     5             7  ancestor deleted    yes      2200
      3        9    600                     2            10           deleted    yes      3400
      1       11    200                     0            12           deleted    yes      4200

The retention policy makes one more period unauditable than it deletes: with six periods deleted the unauditable periods are seven, with nine deleted ten. The extra period is always the oldest of the window; its rows remain, but consumption is a difference and the other end of that difference is deleted. In the six-period window the 2026-07 rows remain, yet they cannot be recomputed and the summary layer falls to 25 cells. Reading a retention period as “the last six periods are auditable” is a misreading; five periods are auditable.

The last column answers the question the previous lesson left open. Lineage had written 4,400 consumption links; in the six-period window 2,200 of them now point to a row that no longer exists, and 200 of those are precisely the ancestors of the boundary period. Keeping the record does not keep traceability; if a link’s row is deleted, the record only reports where the loss is.

The alarm column is the second cost. The period-aware partition check was built in this course as the correct check. When retention is applied, the set of periods it expects stays larger than the set that exists, and the check fires on intact data: no corruption, yet a finding. A correct check becomes a false alarm generator when the field it must look at is narrowed, because no one updated the expected period set with the policy. Whose job that is is the first section’s question: the position where the policy is applied is source, and it has no owner.

Summary

• Ownership is the authority to change the output of a position; although three of the five stages are owned, all six corruptions are caught at an ownerless position, because the checks run at the source and at the summary layer while ownership was written to the produced layers.
• A finding finds a recipient by walking downstream, but in three corruptions the recipient cannot make the repair; late_arrival and unit_changed go to a role five stages from the repair point, with no access to the evidence.
• The three classes whose repair point is ownerless are exactly the three classes the flawed check set never catches: the least audited position is the least owned position.
• The retention policy makes one more period unauditable than it deletes; with six periods deleted the unauditable periods become 7, the summary falls to 25 cells and the window’s oldest period cannot be recomputed though its rows remain.
• The record is not independent of retention: 2,200 of the 4,400 consumption links point to a deleted row, and the correct partition check turns into a false alarm.

Course Wrap-Up

The table is one rule written ten times. A data pipeline’s number is not the rows it carries but how many steps later a wrong row is caught; a pipeline whose check for catching a corruption is not written counts as unmeasured. No number in the uninstrumented column comes from bad code; they are the numbers of an arrangement that counts the carried rows but never asks whether they are correct.

The second claim is also read from the same table: an added check also corrupts. The rows that say so by name are these. In quality-and-principles/02 the strict contract blocks 1200 good rows on a harmless field addition and drops the summary to 25 cells, though the corruption caught is zero; in the same lesson load-level enforcement blocks 2200 good rows and yields no output. In quality-and-principles/01 the partition check stays green while 46 rows are unwritten: false assurance costs more than no check. In domain-and-lifecycle/04 the row count check with a 0.05 share catches no class. In domain-and-lifecycle/05 stream mode permanently pollutes 10 summary cells on a late-arriving row, while batch mode leaves 55 correct. In quality-and-principles/03 deduplication absorbs the corrected load. In quality-and-principles/05 the retention policy turns the correct partition check into a false alarm.

The third claim is why looking at the served number is not enough, and it is counted with three corruptions by name. duplicated leaves all five values exactly the same, because deduplication absorbs the surplus; domain-and-lifecycle/02, domain-and-lifecycle/04 and quality-and-principles/01 each measured this separately. type_drift does the same in a type conversion. unit_changed is the quietest: it moves ten summary cells between 1812946491 and −1812944121, the two periods cancel each other, and the value of north stays at 1212.55. None of the three can be seen by observing the served output.

Across ten lessons the rows the pipeline carries were counted and how many stages later a corruption was caught was measured. One thing was never asked: where those rows sit. The source was cited as 2400 rows, the summary as 55 cells, the links as 11455 records; but the layout they are written in, its partitioning, and how much it multiplies a query were never counted. The next course turns this layout into an object of measurement.