Packaging

The previous lesson ran the same team on a three-axis matrix and showed that the remaining difference was the real environment difference. That measurement had a silent assumption: what ran in every environment was the same source. Tests were read straight from the working directory.

Once a project is handed off for someone else’s use, this assumption is not automatically satisfied. The source directory does not travel as is; it is packaged into a distribution format, that package is carried, and it is unpacked on the other side. This lesson’s question is not how packaging is done: how many distinct contents does a distribution produced from the same source give, and what fixing brings that number down to one?

Distribution Format and Build System

Two distribution formats are separated, and the difference between them is this lesson’s final measurement’s subject.

A source distribution carries source files and the project manifest. A build step runs on the installing side and produces the tree to be installed. A built distribution carries that step’s result instead; only the unpacking work is left on the installing side.

The tool class that produces both these formats is the build system. Its job is to produce the answer to three questions: which files go into the distribution, in what order the distribution is written, and when the build step runs. This course does not write tool names; the build system is modeled here with three functions — an inclusion rule, an archive writer, and a build step. The model produces no real archive, writes no file to disk, and downloads no package; the only thing it produces is content digests.

Three Directory States

What the inclusion rule does depends on what stands next to the source. A working directory is not just the source: build artifacts, cache files, and local settings files accumulate there too. The measurement defines three states. A clean directory carries only the source. A cached directory has one generated file. A cluttered directory also has a log file and a local settings file.

Three inclusion rules are tried. Collect everything takes every file in the directory. Exclude by pattern leaves specific extensions out. Explicit list takes only the files named in the manifest.

The measurement’s assumptions:

• QP15 — The source tree is fixed and the same across all three states; states are separated only by the files standing next to the source.
• QP16 — The distribution’s content is the digest of file names and bodies, sorted by name; directory listing order does not enter this digest.
• QP17 — The archive digest, by contrast, is sensitive to order and to the stamp; two runs’ listing order and run stamp are modeled with fictional values, no real clock is read.
• QP18 — The build step produces a single file from the source and writes the name of the environment it runs in into it. A real build step does far more than this; the only property the model keeps is that its output depends on the environment it runs in.
• QP19 — Digests are truncated to eight hexadecimal digits; two digests being equal is read as their content being equal.
• QP20 — No file is written to disk, no archive is produced, no package is downloaded; the entire build is modeled in memory.
• QP21 — No duration is measured. What is counted is distinct content, distinct digest, and distinct install count.

Measurement

"""Packaging: how many distinct contents does a distribution from the same source give."""

import hashlib

SOURCE = {
    "metrics/__init__.py": "from .discount import discount\n",
    "metrics/discount.py": "def discount(amount, member, coupon):\n    return amount\n",
    "metrics/rate.json": '{"ceiling": 20}\n',
    "tests/test_discount.py": "def test_ceiling():\n    pass\n",
    "README.md": "# metrics\n",
    "project.toml": '[project]\nname = "metrics"\n',
}
CLUTTER = {
    "clean": {},
    "cached": {"metrics/__cache__/discount.chc": "compiled\n"},
    "cluttered": {"metrics/__cache__/discount.chc": "compiled\n",
                  "metrics/temp.log": "run log\n",
                  "local_settings.txt": "path = /home/user\n"},
}
MANIFEST = ("metrics/__init__.py", "metrics/discount.py", "metrics/rate.json",
            "project.toml")
RULES = {
    "collect everything": lambda name: True,
    "exclude by pattern": lambda name: not name.endswith((".chc", ".log")),
    "explicit list": lambda name: name in MANIFEST,
}


def tree(state):
    return {**SOURCE, **CLUTTER[state]}


def digest(text):
    return hashlib.sha256(text.encode()).hexdigest()[:8]


def content_digest(files):
    """A distribution's content: name and body, sorted by name."""
    return digest("".join(f"{a}\n{files[a]}" for a in sorted(files)))


def select(state, rule):
    d = tree(state)
    return {a: g for a, g in d.items() if RULES[rule](a)}


print(f"{'inclusion':<18s} " + " ".join(f"{d:>14s}" for d in CLUTTER)
      + "  distinct content")
total = []
for rule in RULES:
    row, digests = [], []
    for state in CLUTTER:
        chosen = select(state, rule)
        d = content_digest(chosen)
        digests.append(d)
        total.append(d)
        row.append(f"{len(chosen)}·{d}")
    print(f"{rule:<18s} " + " ".join(f"{s:>14s}" for s in row)
          + f"{len(set(digests)):11d}")
print(f"nine builds, {len(set(total))} distinct content")

print()
print(f"{'order rule':<18s} {'stamp rule':<22s} {'run A':>9s} "
      f"{'run B':>9s}  distinct")


def order(files, run):
    return sorted(files, reverse=(run == "run B"))


def archive(files, sorted_order, from_source, run, stamp):
    names = sorted(files) if sorted_order else order(files, run)
    d = content_digest(files) if from_source else stamp
    return digest("".join(f"{a}\n{files[a]}" for a in names) + f"stamp={d}\n")


CHOSEN = select("cluttered", "explicit list")
STAMP = {"run A": "s-101", "run B": "s-207"}
for s_name, sorted_order in (("listing order", False), ("sorted by name", True)):
    for d_name, from_source in (("run stamp", False),
                                 ("derived from source", True)):
        a = [archive(CHOSEN, sorted_order, from_source, k, STAMP[k]) for k in STAMP]
        print(f"{s_name:<18s} {d_name:<22s} {a[0]:>9s} {a[1]:>9s}"
              f"{len(set(a)):6d}")

print()


def build(files, environment):
    """Build step: produces one file from the source."""
    return {**files, "metrics/_build.py": f'BUILT = "{environment}"\n'}


sdist = CHOSEN
built_dist = build(CHOSEN, "production environment")
print(f"{'distribution format':<20s} " + " ".join(f"{d:>10s}" for d in CLUTTER)
      + "  distinct install")
for name, install in (("source distribution",
                        lambda e: content_digest(build(sdist, e))),
                       ("built distribution",
                        lambda e: content_digest(built_dist))):
    o = [install(d) for d in CLUTTER]
    print(f"{name:<20s} " + " ".join(f"{x:>10s}" for x in o)
          + f"{len(set(o)):13d}")

inclusion                   clean         cached      cluttered  distinct content
collect everything     6·7a83a9db     7·0bf44f2f     9·12b06330          3
exclude by pattern     6·7a83a9db     6·7a83a9db     7·bd9cba48          2
explicit list          4·c9929633     4·c9929633     4·c9929633          1
nine builds, 5 distinct content

order rule         stamp rule                 run A     run B  distinct
listing order      run stamp               6d80f316  547a38b3     2
listing order      derived from source     5455d4ab  ecb04028     2
sorted by name     run stamp               6d80f316  1fd59d06     2
sorted by name     derived from source     5455d4ab  5455d4ab     1

distribution format       clean     cached  cluttered  distinct install
source distribution    6897a192   85226fbb   fee3c544            3
built distribution     1440a878   1440a878   1440a878            1

Nine Builds, Five Contents

The top table gives the lesson’s first number: 9 builds made from the same source give 5 distinct contents.

Reading the rows one by one shows where the number comes from. The collect everything rule takes 6, 7, and 9 files across the three directory states and gives 3 distinct contents. This rule’s result depends not on the source but on what stands next to the source; two working directories pulled from the same repository produce separate packages.

The exclude by pattern rule gives the same result in two states — clean and cached carry the same digest, 7a83a9db — because the extension it excludes catches the cache file. In the third state, though, the local_settings.txt file does not match the pattern and enters the content: 7 files, a separate digest. A rule can only exclude as much as it knows to.

The explicit list gives 4 files and a single digest, c9929633, in all three states: 1 distinct content. The difference is methodological. The first two rules say “what stays out,” and every new unlisted file silently comes in. The third rule says “what comes in,” and every unlisted file silently stays out. The direction of silence changes, and unification comes from that change of direction.

Two digests appear in two of the nine builds at once: 7a83a9db is in both the collect everything/clean and exclude by pattern/clean cells. Nine cells, five distinct contents; where rules coincide, the count drops.

File counts make one more distinction visible. The explicit list takes 4 files, while the clean directory has 6 source files. The two left out are tests/test_discount.py and README.md. This is not a flaw, it is a decision: the distribution carries the installed tree, not everything in the repository. The test file stays in the repository and runs there; if it entered the distribution, it would take up disk space and import namespace on the installing side. The same decision could be made for the documentation file too, and the reverse is equally defensible — what is not defensible is the decision being never made. In the first two rules this decision is never made at all; whether a file enters the distribution is decided by its name’s extension.

The Manifest’s Two Jobs

All three inclusion rules are written somewhere, and that place is the project manifest. The manifest does two separate jobs, and because the two often sit in the same file, they get confused.

First job: the package’s identity. Name, version, dependency declaration, and the files that will enter the distribution. These define data; which tool reads them does not change the result.

Second job: choosing the build system. Which build system will produce the distribution, and what interface that system will be called with. This is a directive; it directly decides the result.

The distinction’s practical counterpart is this: the first job is declarative, and two separate tools reading the same manifest choose the same file set. The second job is a choice, and the layout of the produced archive, its stamp, and the build step’s behavior change depending on the system chosen. The measurement’s top table tests the first job, its middle table tests the second.

This is why a manifest being “complete” has two separate meanings. If the file list is complete, the distribution’s content unifies. If the build system and its calling interface are also written, every run that produces the distribution calls the same system the same way. When the second is not written, production falls back to the tool’s default and the shared setup’s resolution measurement’s situation repeats: no decision was made, so the environment gives one.

Two Decisions That Fix the Archive

Content unifying does not mean the archive unifies. The middle table archives the same file set — the four files the explicit list rule chose — in two runs and lays out four rule combinations side by side.

In three combinations, the two runs give 2 distinct digests. In one combination, 1.

When listing order is used, run A and run B write the files in separate order; the body is the same, the byte string is separate. This difference remains even when the stamp is derived from the source: the second row gives 5455d4ab against ecb04028. When run stamp is used, the difference remains even when order is fixed: the third row gives 6d80f316 against 1fd59d06.

Only the fourth row gives 1: order fixed by name and the stamp derived from the source. Both are needed, one is not enough. This is the definition of reproducibility: the same output from the same input, run to run.

This also explains why the archive digest is measured separately. The installing side cannot compute the content itself; what it gets is an archive, and it can only verify it through its bytes. Even if the content is single, if the archive’s bytes change run to run, it cannot be confirmed that two archives came from the same source. The reproducibility claim is therefore built not at the content level but at the byte level.

Because the 0ef2d766-equivalent digest in the third row matches the one in the first row, a detail becomes visible: run A’s listing order is already sorted by name, so in that run the two order rules give the same result. The difference only shows up in run B. A configuration flaw does not show up in every run; in the run where it does not show up, it is assumed not to exist.

When the Build Step Runs

The bottom table gives the last measurement and separates the two distribution formats.

The source distribution gives 3 distinct installs across the three directory states. The reason does not show up in the table but is explicit in the model: the build step runs on the installing side and writes the name of the environment it runs in into the file it produces. The installed tree therefore changes from environment to environment. The distribution itself was single — four files, one digest — but the install outcome is not.

The built distribution gives 1 distinct install across all three states. The build step ran once, on the production side; the installing side only unpacks.

The difference between the two is where the build step runs, and this is a distribution format decision. A source distribution carries the build to the installing side; in return, the installing side can produce output suited to its own environment. A built distribution does the build once; in return, it cannot reflect the environment difference into the distribution.

This does not mean the built distribution is free. If the build step reads something from the environment, the built distribution embeds the value it read and now only fits that environment. In that case which environment the distribution fits has to be written on the package itself; if it is not, the installing side unpacks a package that does not fit, and the error shows up much later than install, during the run. The source distribution’s 3 distinct installs are not a flaw, they are this adaptation itself; the flaw would be the three distinct outcomes being unexpected.

The principle of a build output produced once and distributed everywhere was established on the operations side in the DevOps Culture and Fundamentals course. The measurement here gives that principle’s counterpart on the language side, as a number: when the principle is applied, install outcome is 1; when it is not, it is as many as the environment count.

To gather the three measurements into one sentence: for a distribution from the same source to give a single content, three things have to be fixed together — the explicit list of files to include, the archive’s write order and stamp, and which side the build step runs on. If any one of the three is left open, the number is more than one, and its size depends on how much the open spot is affected by the environment.

Summary

• Distribution format splits in two: a source distribution carries the build step to the installing side, a built distribution carries that step’s result.
• 9 builds from the same source give 5 distinct contents; collect everything gives 3, exclude by pattern 2, explicit list 1 distinct content.
• Excluding rules silently let an unlisted file in; an explicit list silently leaves it out. Unification comes from this change of direction.
• For the archive digest to unify, order and stamp both have to be fixed together; only one of four combinations gives 1 distinct digest, three give 2.
• The source distribution gives 3 distinct installs across three environments, the built distribution gives 1; even when the distribution itself is single, install outcome may not be.

Next Step

This lesson produced the distribution and named its content with a digest. The digest is a number computed on the production side, and it stays there. Once a package is sent to a package registry, though, it is given a name and a version; the installing side no longer asks for the content, it asks for the name. The next lesson’s question grows from here: can a published version’s content change afterward, can the same name carry two distinct contents, and if it can, what does the version number the lock file writes actually guarantee?