Working Environment

The Data Collection and Preparation course closed with a seventeen-step preparation pipeline. The measure of that pipeline was readability: once each step’s decision and row accounting were written to the output, 39 of 39 decisions were readable after the run. Readability concerns how a pipeline is read and does not fully close the course’s question. This lesson asks the next question: when the same notebook is run on another machine, does it give the same number.

The question comes before the choice of tool. As soon as an array or dataframe interface is used, part of the calculation no longer sits in the code that is written but inside the components that are installed; the version of those components, the convention by which the machine reads a number, and the number of pieces the calculation is divided into all move the result. For this reason the environment is treated here not as a list of install commands but as a list of items that must be pinned. The measure is likewise not the number of commands but how many steps give the same number on two machines.

• NC1. A notebook gives the same number on another machine if every item that determines the number it produces is written down. The lock file writes only some of these items; the rest stays nowhere unless the notebook itself declares it.
• NC2. An item that comes from the environment moves the number in two ways: at the order-of-magnitude level, that is, with an error visible to the eye; or around the threshold, that is, with a difference visible only when compared. The second escapes a check that looks at row counts.

Writing the Environment as Data

# environment.py — MODEL: a run environment and the documents that pin it are written as data.
# "level" is the document that pins an item. "step" was read from the seventeen-step
# preparation pipeline of Data Collection and Preparation; beside each step are the items that determine its result.
LEVELS = ["installed", "list", "lock", "declaration"]
ITEMS = [
    ("dependency name",             "list"),
    ("direct dependency version",   "lock"),
    ("indirect dependency version", "lock"),
    ("interpreter version",         "lock"),
    ("locale",                      "declaration"),
    ("chunk count",                 "declaration"),
    ("random seed",                 "declaration"),
    ("notebook cell order",         "declaration"),
    ("data file digest",            "declaration"),
]
STEPS = [
    ("01 file formats",            ["dependency name", "locale"]),
    ("02 pulling from databases",  ["dependency name", "data file digest"]),
    ("03 collecting from interfaces", ["dependency name"]),
    ("04 structural parsing",      ["dependency name", "indirect dependency version"]),
    ("05 sampling",                ["direct dependency version", "random seed"]),
    ("06 missing values",          ["dependency name"]),
    ("07 duplicate records",       ["direct dependency version"]),
    ("08 type conversions",        ["interpreter version", "locale"]),
    ("09 outliers",                ["direct dependency version"]),
    ("10 categorical fixing",      ["indirect dependency version"]),
    ("11 text normalization",      ["interpreter version"]),
    ("12 merging",                 ["direct dependency version", "notebook cell order"]),
    ("13 reshaping",               ["dependency name"]),
    ("14 grouping",                ["direct dependency version", "chunk count"]),
    ("15 derived variables",       ["interpreter version"]),
    ("16 categorical encoding",    []),
    ("17 preparation pipeline",    []),
]
PINNED_BY = dict(ITEMS)
pinned = lambda d: {a for a, k in ITEMS if LEVELS.index(k) <= LEVELS.index(d)}
reproduced = lambda d: [name for name, deps in STEPS if set(deps) <= pinned(d)]

row_fmt = lambda w: lambda h: "".join(str(v).ljust(w[0]) if j == 0 else str(v).rjust(w[j])
                                      for j, v in enumerate(h))
y = row_fmt([12, 16, 22, 16])
print(y(["level", "pinned item", "reproduced step", "moving step"]))
for d in LEVELS:
    u = reproduced(d)
    print(y([d, f"{len(pinned(d))}/{len(ITEMS)}", f"{len(u)}/{len(STEPS)}",
             len(STEPS) - len(u)]))

locked = set(reproduced("lock"))
print("\nsteps that still move after the lock file, and why:")
for name, deps in STEPS:
    if name not in locked:
        print("  " + name + " <- " + ", ".join(b for b in deps if PINNED_BY[b] == "declaration"))

level            pinned item       reproduced step     moving step
installed                0/9                  2/17              15
list                     1/9                  5/17              12
lock                     4/9                 11/17               6
declaration              9/9                 17/17               0

steps that still move after the lock file, and why:
  01 file formats <- locale
  02 pulling from databases <- data file digest
  05 sampling <- random seed
  08 type conversions <- locale
  12 merging <- notebook cell order
  14 grouping <- chunk count

Each of the four levels corresponds to a different document. At the installed level there is no document; the components are installed system-wide on the machine, and you can learn which version is there only by looking at the machine. Only 2 of the seventeen steps give the same result on two machines — these are the steps that touch no outside item.

An isolated environment comes on top of this level and does a single thing: it separates the components the notebook uses from the rest of the machine. Isolation does not pin a number by itself, it makes pinning possible; writing the version of a component that does not sit in a separate set makes no sense, because another job can change it.

A dependency list records the names of the components the notebook uses directly and usually gives a range. This raises the number of pinned items to 1 and the reproduced steps to 5: only the steps whose behavior does not change from version to version are gained. The list says which component is used, not which version of it.

The lock file closes the gap in between. It writes the exact versions of the direct components, the indirect components they pull in, and the interpreter version in a single resolution; the second machine installs the same resolution. The pinned items rise to 4 and the reproduced steps to 11. The gain ends here and the hidden default stands at exactly this point: because the lock file writes a full version resolution, the whole environment is assumed to be pinned. The five items that are not pinned keep moving 6 steps, and none of these six steps has anything to do with versions.

The last level is not a file but an environment declaration that sits at the head of the notebook: the locale it is read with, the number the calculation is divided into, the seed used, the order in which the cells run, and the digest of the input file. When these five items are written, the ratio becomes 17 of 17.

The functions of the four documents do not replace one another, and this distinction cannot be read from install commands. The isolated environment draws a boundary, the dependency list writes an intent, the lock file records a single resolution of that intent, and the environment declaration writes the run conditions that remain outside the resolution. The pair most often confused is the list and the lock: the list is the document a person writes and updates by hand, while the lock is the document derived from the list and not written by hand. Narrowing the list does not take the place of the lock, because the indirect components never appear in the list, and steps 04 and 10 in the table depend on exactly them.

The Same Notebook, Two Machines

# run.py — MODEL: the same notebook is run on two machines. The fictional readings dataset
# is the one Data Collection and Preparation prepared: 1,260 subscribers, 3 periods, 3,199 of 3,780 cells filled.
def rng(seed):
    s = seed & 0xFFFFFFFF
    def next_value():
        nonlocal s
        s = (1664525 * s + 1013904223) & 0xFFFFFFFF
        return s / 4294967296
    return next_value

SUBSCRIBERS, PERIODS = 1260, 3
r = rng(20260218)
RAW = [round(4 + r() * 56, 2) for _ in range(SUBSCRIBERS * PERIODS)]
r2 = rng(9110723)
p = [r2() for _ in range(SUBSCRIBERS * PERIODS)]
EMPTY = set(sorted(range(len(p)), key=lambda j: p[j])[:581])
READINGS = [None if j in EMPTY else RAW[j] for j in range(SUBSCRIBERS * PERIODS)]
mean = lambda v: sum(v) / len(v)

# Item 5 — locale. The meter index arrives as text; which of the two separators is
# the decimal and which the thousands separator is read from the machine's locale.
METER_INDEX = [round(200 + (i * 7919) % 1400
                     + sum(x for x in READINGS[i*PERIODS:(i+1)*PERIODS] if x is not None), 2)
               for i in range(SUBSCRIBERS)]
def format_value(v):
    whole, _, frac = f"{v:.2f}".partition(".")
    g = ""
    while len(whole) > 3:
        whole, g = whole[:-3], "." + whole[-3:] + g
    return whole + g + "," + frac
parse = lambda m, thousands, decimal: float(m.replace(thousands, "").replace(decimal, "."))
TEXT = [format_value(v) for v in METER_INDEX]
A = [parse(m, ".", ",") for m in TEXT]   # thousands dot, decimal comma
B = [parse(m, ",", ".") for m in TEXT]   # thousands comma, decimal dot

# Item 6 — chunk count. When the aggregation is divided into chunks, the mean of the
# chunk means is taken; the chunks are unequal, because the missing cells are not evenly distributed.
def chunked_mean(k):
    bounds = [i * SUBSCRIBERS // k for i in range(k + 1)]
    chunks = [[x for j in range(bounds[i], bounds[i+1]) for x in READINGS[j*PERIODS:(j+1)*PERIODS]
               if x is not None] for i in range(k)]
    return mean([mean(x) for x in chunks])
THRESHOLD = 0.0005  # m3; the reading resolution is 0.01 m3, a difference below one twentieth of it cannot be read

# Item 8 — notebook cell order. There are two cells: filling an empty cell with the
# subscriber's own mean and counting the cells that exceed the upper tariff tier (40 m3).
def fill(v):
    o = []
    for i in range(SUBSCRIBERS):
        h = [v[i*PERIODS+d] for d in range(PERIODS) if v[i*PERIODS+d] is not None]
        o.append(mean(h) if h else 0.0)
    return [o[j // PERIODS] if x is None else x for j, x in enumerate(v)]
count = lambda v: sum(1 for x in v if x is not None and x > 40.0)

row_fmt2 = row_fmt([22, 20, 20, 14])
print(row_fmt2(["environment item", "machine A", "machine B", "same"]))
print(row_fmt2(["locale (total)", round(sum(A), 2), round(sum(B), 2), "no"]))
print(row_fmt2(["chunk count (mean)", round(chunked_mean(1), 6), round(chunked_mean(8), 6),
                "yes" if abs(chunked_mean(1) - chunked_mean(8)) < THRESHOLD else "no"]))
print(row_fmt2(["cell order (over 40)", count(READINGS), count(fill(READINGS)), "no"]))

shrunk = sum(1 for a, b in zip(A, B) if b < a)
diff = abs(chunked_mean(1) - chunked_mean(8))
print(f"\nlocale: {len(TEXT)} of {len(TEXT)} meter indexes changed; {shrunk} shrank because the "
      f"thousands separator was read as a decimal, {len(TEXT)-shrunk} grew")
print(f"chunk count: difference {diff:.6f} m3, {diff/THRESHOLD:.0f} times the {THRESHOLD} m3 threshold")
print(f"cell order: the same two cells, in reverse order {count(fill(READINGS))-count(READINGS)} more cells "
      f"were counted ({100*(count(fill(READINGS))-count(READINGS))/count(READINGS):.0f} percent)")

environment item                 machine A           machine B          same
locale (total)                  1236664.22         41433984.33            no
chunk count (mean)               31.895661           31.889608            no
cell order (over 40)                  1116                1264            no

locale: 1260 of 1260 meter indexes changed; 613 shrank because the thousands separator was read as a decimal, 647 grew
chunk count: difference 0.006053 m3, 12 times the 0.0005 m3 threshold
cell order: the same two cells, in reverse order 148 more cells were counted (13 percent)

All three rows assume the same code, the same data, and the same lock file. What produces the difference is only the environment item.

The first row is a difference at the order-of-magnitude level. The meter index arrives as text, and a choice is made between two conventions: which separator is the decimal and which is the thousands separator. All 1,260 meter indexes change, and the difference does not run in one direction either — the 613 indexes that carry a thousands separator shrink a thousandfold, and the 647 indexes that carry no separator grow a hundredfold. The total comes out as 41,433,984.33 m³ instead of 1,236,664.22 m³. An error of this kind is visible to the eye, but for it to be seen someone has to look at both machines at once.

The second row is the opposite. When the aggregation is divided into eight chunks, the mean of the chunk means is taken; the chunks are not of equal size, because the missing cells are not evenly distributed among the subscribers. The result is 31.889608 m³ instead of 31.895661. Two floating-point numbers are not compared with ==; a threshold is written instead. The threshold here is 0.0005 m³, one twentieth of the meter resolution of 0.01 m³: a difference below this value cannot be read from the measuring device anyway, so it counts as the same. The difference is 0.006053 m³, 12 times the threshold, and since it stays above the threshold the two runs do not give the same number. The row count is 3,199 in both runs; a check that looks at row counts does not see this difference.

The third row concerns the form of the notebook. There are two cells: filling an empty cell with the subscriber’s own mean and counting the cells that exceed the upper tariff tier. If the fill runs first, 581 cells gain a value and some of them pass the 40 m³ threshold; the count comes out 1,264 instead of 1,116, 148 cells more, 13%. No cell was written wrongly; only their order is different.

What the Notebook Records and What It Does Not

This last row shows why the notebook form is an environment item. A notebook file stores its cells in the order they were written, not the order they ran; beside each cell sits only a run counter. If the counter reads 3, 1, 2, a notebook that looks correct when read from top to bottom actually ran in another order. For this reason a one-condition rule is written for a notebook to count as reproducible: when the notebook is run from start to finish, in a single pass, from a clean state, it must give the same output. The saved output is not evidence for this rule, because a saved output does not say in which order it was produced.

It also becomes visible why the measure is set up in the first lesson of this course. In the lessons that follow, the body of the calculation leaves the hand-written loops and moves inside an abstraction; the decisions that abstraction makes about element type, shape and summation order add new rows to exactly the table above. That the eight-chunk aggregation moves the mean by 0.006053 m³ is not an error but a default; whoever decided how many chunks to divide into also decided that number. The environment declaration is for this reason not an extension of the version list but a part of the calculation.

The same reasoning applies to the saved output. Because a notebook keeps both the code and the output in a single file, it is easy to read, but it does not show where the difference between two runs begins; what shows that is the run ledger, in which each step writes its own decisions and row accounting. The lock file fixes the environment, the run ledger fixes the decision, and running the notebook from the start fixes the order; unless all three are written, “the same number” remains a claim.

Summary

• A working environment is not a list of commands but a list of nine items that must be pinned; a different document pins each item.
• With no document, 2 of 17 steps give the same result on two machines, with the dependency list 5, with the lock file 11.
• None of the 6 steps that still move after the lock file has anything to do with versions; the locale, chunk count, seed, cell order and data digest are not written in the lock file.
• The locale difference changes all 1,260 meter indexes and moves the total from 1,236,664.22 to 41,433,984.33; 613 indexes shrink and 647 indexes grow.
• The chunk count moves the mean by 0.006053 m³ — 12 times the 0.0005 m³ threshold — but never changes the row count; the cell order raises the count of cells above 40 m³ from 1,116 to 1,264.

Next Step

Once the environment is pinned, the notebook gives the same number on two machines, but the calculation itself is still done with hand-written loops, as in the previous course. In the code blocks of this section 3,780 cells were walked one by one, each cell was held as a separate object, and for every summation job the data was read again from start to finish. The next lesson does the same job with an array interface and sets up the unit of measure of the course: how many lines of code, how many passes and how many bytes the same summation job takes.