Report Reproducibility

The previous lesson sat the report on a chain of premises and counted the links of the chain. That report was produced once. Whether another person could reproduce the same report three months later so that it gives the same numbers was never asked. A report is not a file but the product of a run, and there is a decision at every step of the run.

Part of this question was closed earlier. The Data Collection and Preparation course wrote the preparation pipeline as seventeen steps and thirty-nine decisions; when the run ledger was kept, thirty-nine of the thirty-nine decisions became readable after the run. The Data Analysis with Python course went one step further and wrote the working environment as a list of nine items: with the lock file, eleven of the seventeen steps gave the same result on two machines, and the remaining six depended on five items that were not written in the lock file at all. Neither computation is repeated here; they are taken as input. What this lesson adds is the report itself: even when the pipeline is the same and the environment is the same, the report may not be the same.

• RP25. The dataset is inherited from the Data Collection and Preparation course and is fictional: a reading table of 3,199 rows, 1,329 subscribers, three periods, five regions. The seed is written into the code.
• RP26. The arrival lags of the readings are also fictional and are generated with the seed of the previous lesson.
• RP27. The report’s items are fictional: eleven items, each with a fixed value and an equally defensible alternative.
• RP28. Thresholds: 0.05 points for the rate (a single subscriber in the denominator is worth 0.075 points), 0.0005 m³ for the mean — the second is the threshold used by the Data Analysis with Python course. Floating-point comparison is not made with ==.
• RP29. The 39 decisions of the Data Collection and Preparation course and the 9 environment items of the Data Analysis with Python course are taken as input.
• RP30. The report is seven lines: two headline numbers and five region lines.

The Report’s Items

# reproduce.py — MODEL: the fictional dataset from K03 is generated with the same seed; on top
# of it the fictional arrival lags and the item list that produces the report are built.
import math

SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF
REGION = [("north", 0.28, 21), ("south", 0.22, 17), ("east", 0.18, 26),
          ("west", 0.14, 14), ("central", 0.18, 23)]


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


SUBSCRIBERS, DISCARDED = [], []
for i in range(RAW):
    r = rng(SEED + i)
    b = REGION[pick(r(), [x[1] for x in REGION])]
    pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05])
    pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17])
    (DISCARDED if r() < 0.046 else SUBSCRIBERS).append(
        {"subscriber_id": 10001 + i, "region": b[0], "base": b[2]})
VALID = {k["subscriber_id"] for k in SUBSCRIBERS}
READINGS = {}
for k in SUBSCRIBERS + DISCARDED:
    r = rng(SEED + 7000 + k["subscriber_id"])
    for d in range(pick(r(), [0.05, 0.12, 0.21, 0.62])):
        t = 0.0 if r() < 0.038 else math.floor(
            k["base"] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - d * 0.05) * 100
            + 0.5) / 100
        if k["subscriber_id"] in VALID:
            READINGS.setdefault(k["subscriber_id"], {})[d + 1] = t
        r()

# Each reading enters the source some time after the period closes. The lag is fictional and is
# drawn with a separate seed; it does not change the value of the reading, only its visibility.
ARRIVAL_LAG = {}
for k in SUBSCRIBERS:
    r = rng(SEED + 120000 + k["subscriber_id"])
    for d in sorted(READINGS.get(k["subscriber_id"], {})):
        ARRIVAL_LAG[(k["subscriber_id"], d)] = pick(r(), [0.58, 0.19, 0.13, 0.07, 0.03]) * 6


def snapshot(day):
    """The source snapshot in which the readings whose lag does not exceed the day value are visible."""
    g = {}
    for (a, d), lag in ARRIVAL_LAG.items():
        if lag <= day:
            g.setdefault(a, {})[d] = READINGS[a][d]
    return g


# The report's own items. Each item has a fixed value and an equally defensible alternative;
# if the alternative is chosen, the report does not give the same report.
ITEMS = [("data cut-off", "12 days", "40 days"),
         ("refresh scheme", "live query", "pre-aggregated model"),
         ("denominator definition", "registered subscriber", "subscriber with readings"),
         ("zero reading", "counted as a reading", "not counted as a reading"),
         ("observation window", "period 3", "period 2 or 3"),
         ("subscriber filter", "all subscribers", "all three periods complete"),
         ("region filter", "five regions", "excluding central"),
         ("aggregation weighting", "subscriber-weighted", "region-unweighted"),
         ("presentation decimals", "three decimals", "whole number"),
         ("presentation unit", "rate", "percent of target"),
         ("target value", "65", "60")]
FIXED = {a: b for a, b, _ in ITEMS}


def report(k):
    """The report that produces the two headline numbers from the item list. Every decision is read from an item."""
    cut = 12 if k["data cut-off"] == "12 days" else 40
    cut = 0 if k["refresh scheme"] != "live query" else cut
    data = snapshot(cut)
    chosen = [a for a in SUBSCRIBERS if k["region filter"] == "five regions" or a["region"] != "central"]
    if k["subscriber filter"] != "all subscribers":
        chosen = [a for a in chosen if len(data.get(a["subscriber_id"], {})) == 3]
    windows = (2, 3) if k["observation window"] != "period 3" else (3,)
    numerator, values, by_region = [], [], {}
    for a in chosen:
        o = data.get(a["subscriber_id"], {})
        v = [o[d] for d in windows if d in o and (k["zero reading"] == "counted as a reading" or o[d] > 0)]
        by_region.setdefault(a["region"], [0, 0])
        if v:
            numerator.append(a)
            values.extend(v)
            by_region[a["region"]][0] += 1
        by_region[a["region"]][1] += 1
    denominator = len(chosen) if k["denominator definition"] == "registered subscriber" else \
        len([a for a in chosen if data.get(a["subscriber_id"])])
    rate = (sum(x[0] / x[1] for x in by_region.values()) / len(by_region) * 100
            if k["aggregation weighting"] != "subscriber-weighted" else len(numerator) / denominator * 100)
    if k["presentation unit"] != "rate":
        rate = rate / float(k["target value"]) * 100
    decimals = 3 if k["presentation decimals"] == "three decimals" else 0
    return round(rate, decimals), round(sum(values) / len(values), 3)


THRESHOLD_RATE, THRESHOLD_M3 = 0.05, 0.0005
base_rate, base_mean = report(FIXED)
print(f"fictional dataset (seed {SEED}): {len(SUBSCRIBERS)} subscribers, {len(ARRIVAL_LAG)} readings")
print(f"report items: {len(ITEMS)}")
print(f"report with fixed items: reading rate {base_rate:.3f}, "
      f"period 3 mean {base_mean:.3f} m3")
print(f"thresholds: rate {THRESHOLD_RATE} points, mean {THRESHOLD_M3} m3")

fictional dataset (seed 20260218): 1329 subscribers, 3199 readings
report items: 11
report with fixed items: reading rate 56.659, period 3 mean 18.128 m3
thresholds: rate 0.05 points, mean 0.0005 m3

None of the eleven items is hidden in a line of code; all of them stand in a list that the report function reads. This is the first condition of fixing: if an item is buried in the code, it cannot even be asked whether it is released or fixed.

When One Item Is Released

# Each item is released one at a time: the alternative value is chosen, the other ten items
# stay fixed and the report is produced again.
def moves(o, m):
    return abs(o - base_rate) > THRESHOLD_RATE or abs(m - base_mean) > THRESHOLD_M3


print(f"{'released item':<24}{'alternative':<30}{'reading rate':>12}"
      f"{'mean':>10}{'moved?':>12}")
moved = 0
for name, fixed, alt in ITEMS:
    k = dict(FIXED)
    k[name] = alt
    o, m = report(k)
    moved += 1 if moves(o, m) else 0
    print(f"{name:<24}{alt:<30}{o:>12.3f}{m:>10.3f}"
          f"{('yes' if moves(o, m) else 'no'):>12}")
print(f"\n{moved} of {len(ITEMS)} items move the report when released alone")
pair = dict(FIXED)
pair["presentation unit"], pair["target value"] = "percent of target", "60"
o, m = report(pair)
print(f"'target value', which does not move alone, when released together with "
      f"'presentation unit': {o:.3f}")

released item           alternative                   reading rate      mean      moved?
data cut-off            40 days                             63.055    18.321         yes
refresh scheme          pre-aggregated model                36.042    18.166         yes
denominator definition  subscriber with readings            60.531    18.128         yes
zero reading            not counted as a reading            54.628    18.803         yes
observation window      period 2 or 3                       80.512    19.020         yes
subscriber filter       all three periods complete         100.000    18.160         yes
region filter           excluding central                   55.842    17.476         yes
aggregation weighting   region-unweighted                   56.746    18.128         yes
presentation decimals   whole number                        57.000    18.128         yes
presentation unit       percent of target                   87.168    18.128         yes
target value            60                                  56.659    18.128          no

10 of 11 items move the report when released alone
'target value', which does not move alone, when released together with 'presentation unit': 94.432

When a single item is released, the reading rate goes anywhere between 36.042 and 100.000. The other ten items are fixed; the only thing that moves is the one that was released.

The sixth row also deserves to be read. The “subscribers who have all three periods” filter is a reasonable sentence, and it makes the metric exactly 100.000, because the filter filters the denominator as well: a subscriber without a period 3 reading does not stay in the dataset anyway. The metric no longer measures anything, and the report says so nowhere. When a filter touches the denominator of a metric, it can stop the metric from being a metric.

The last row determines how the decision to fix is to be made. When the target value is released alone, the report does not budge, because as long as the presentation unit is “rate” the target enters no computation. Released together with the presentation unit, the same item carries 87.168 to 94.432. An item that does not move the report in this run is not an item that does not need to be fixed. Items move the report not one by one but together, and an item looking ineffective may depend on another item being fixed.

The Second Production

# The report is seven lines: two headline numbers and five region lines. The second production
# runs with the same item list; then each item is released one at a time and the number of
# lines that change is counted.
def lines(k):
    o, m = report(k)
    s = [f"reading rate {o:.3f}", f"period 3 mean {m:.3f} m3"]
    cut = 12 if k["data cut-off"] == "12 days" else 40
    cut = 0 if k["refresh scheme"] != "live query" else cut
    data = snapshot(cut)
    for b, _, _ in REGION:
        v = [data[a["subscriber_id"]][3] for a in SUBSCRIBERS
             if a["region"] == b and 3 in data.get(a["subscriber_id"], {})]
        s.append(f"{b} mean {sum(v) / len(v):.3f} m3 ({len(v)} readings)")
    return s


first = lines(FIXED)
print(f"second production with the same item list: "
      f"{sum(1 for a, b in zip(first, lines(dict(FIXED))) if a == b)}/{len(first)} lines identical")
print(f"\n{'released item':<24}{'changed lines':>14}   which ones")
total = 0
for name, fixed, alt in ITEMS:
    k = dict(FIXED)
    k[name] = alt
    changed = [i for i, (a, b) in enumerate(zip(first, lines(k))) if a != b]
    total += len(changed)
    print(f"{name:<24}{len(changed):>14}   {', '.join(str(i + 1) for i in changed) or '-'}")
print(f"\nwhen each of the {len(ITEMS)} items is released alone, {total} lines change in total "
      f"(in a {len(first)}-line report)")
LEDGER = [("Data Collection and Preparation", 39, "decisions in the 17 steps of the pipeline"),
          ("Data Analysis with Python", 9, "items of the run environment"),
          ("this lesson — report", len(ITEMS), "the report's own items")]
print(f"\n{'source':<32}{'items':>7}   what it fixes")
for a, n, s in LEDGER:
    print(f"{a:<32}{n:>7}   {s}")
print(f"{'total':<32}{sum(n for _, n, _ in LEDGER):>7}")

second production with the same item list: 7/7 lines identical

released item            changed lines   which ones
data cut-off                         7   1, 2, 3, 4, 5, 6, 7
refresh scheme                       7   1, 2, 3, 4, 5, 6, 7
denominator definition               1   1
zero reading                         2   1, 2
observation window                   2   1, 2
subscriber filter                    2   1, 2
region filter                        2   1, 2
aggregation weighting                1   1
presentation decimals                1   1
presentation unit                    1   1
target value                         0   -

when each of the 11 items is released alone, 26 lines change in total (in a 7-line report)

source                            items   what it fixes
Data Collection and Preparation      39   decisions in the 17 steps of the pipeline
Data Analysis with Python             9   items of the run environment
this lesson — report                 11   the report's own items
total                                59

The second production gives the same result in all seven of the seven lines. This shows that carrying the item list is sufficient; what is fixed is not the code itself but the list the code reads.

The items are not of equal weight. The data cut-off and the refresh scheme change all seven lines at once, because both determine which snapshot of the source will be read, and that snapshot lies under every line of the report. The presentation items change only one line. The difference gives a priority list: if a single item is to be carried alongside the report, that item is the data cut-off.

The total of fifty-nine items comes from three different places and the three do not substitute for one another. The thirty-nine decisions of the preparation ledger fix the steps of the pipeline but do not know the cut-off of the report. The nine items of the working environment fix the run but do not know the filter of the report. The eleven items of the report guarantee nothing unless both of the others are fixed: two reports running with the same item list still diverge if the pipeline or the environment underneath has changed.

Summary

• Reproducing a report requires fixing 59 items from three sources: the 39 decisions of the preparation ledger, the 9 items of the working environment and the report’s own 11 items.
• Of the eleven items, 10 move the report when released alone; the reading rate goes anywhere between 36.042 and 100.000.
• An item that does not move the report alone is not an item that does not need to be fixed: the target value, released together with the presentation unit, carries 87.168 to 94.432.
• When a filter touches the denominator of a metric, it can stop the metric from being a metric: the “subscribers who have all three periods” filter makes the reading rate 100.000 by definition.
• Items are not of equal weight: the data cut-off and the refresh scheme change all seven lines, the presentation items change only one; the eleven items move 26 lines in total.
• When the item list is carried, the second production is identical in all seven of the seven lines; what is fixed is not the code but the list the code reads.

Next Step

In five lessons the topic built a report from start to finish and wrote down the decision at every step. When the same twelve metrics were encoded in three layouts, 1, 2 and 9 metrics were read back at a glance and the number of supported decisions came out 0, 0 and 3. A 60.948% metric was tried in six ways without a single reading changing in the measurement network, and five of them exceeded the threshold. The same metric gave 56.659, 48.081 and 36.042 under three refresh schemes. Nine of the eleven wrong defaults of the chain that runs from finding to recommendation strengthened the recommendation. And producing the report a second time required 59 items.

All of this work was done with a silent assumption: the dataset fits on a single machine. The 3,199 rows were read from start to finish in every block, every aggregation was done in a single loop, and trying a filter took seconds. When the measurement network grows, this assumption falls, and the moment it falls it is not the report itself but the computation itself that has to be moved. The next topic asks this: when the dataset does not fit on a single machine, which part of the report can be split, how many records does the split part carry across which shuffles, and what does the serial fraction cost.