Environment and Version Management

The previous lesson placed three indicators on top of the pipeline and counted what each of them sees. Every number leaned on a single silent assumption: the code was always the same and always ran with the same setting. Yet the band was 0.05, the tolerance was 1, the number of loaded periods was twelve — all three stood around the run, not in the code of the pipeline.

This lesson turns that surroundings into an object of measurement. A single pipeline code is deployed to three environments and two questions are asked: how different a result the same code gives in them, and whether a report in your hands shows in which environment, with which code it was produced.

• OO47. The source, the five-stage pipeline and the three environment settings are fictional and are generated inside the lesson. No orchestrator, deployment tool or version management product is called. The seed is 20260218.
• OO48. The procedure of continuous integration, delivery and environment promotion was measured in the Continuous Integration and Delivery and Infrastructure as Code courses of the DevOps and Platform Engineering curriculum; it is not repeated. The only measure here is the different result the same code produces across environments.
• OO49. An environment carries three settings and all three are outside the code: the number of loaded partitions, the indicator band and the version of the reporting rule. s1 reports commercial consumption with a coefficient of 1.00, s2 with 1.05; the backfill procedure was measured in the last lesson of the Transformation topic.
• OO50. A version stamp is the triple of period, code version and setting written to the output of a run. The baseline is the run without orchestration: a single environment is assumed, no stamp is written, and the number of reports that carry an identity is 0.

The Same Code, Three Environments

The block below defines a single report function and runs it with three settings. From the environment the function takes only the set of periods to load and the rule coefficient; because the band tunes the indicator and not the pipeline, it enters the table with the measure of the previous lesson, the largest loss it hides.

# FICTIONAL. The meter readings of the measurement network, the five-stage pipeline and three
# environment settings are generated inside the lesson; no orchestrator, deployment tool or
# version management product is called. Seed 20260218.
SEED, M32 = 20260218, 0xFFFFFFFF


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


REGION = ["east", "west", "north", "south", "central"]
TYPE = ["residential", "commercial"]
PERIODS = [f"2026-{a:02d}" for a in range(1, 13)]
SUBSCRIBERS = {}
for i in range(200):
    r = rng(SEED + 37 * i)
    SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)],
                                    "start": 10000 + int(r() * 60000)}


def generate_source():
    records = []
    for i, (sub, a) in enumerate(SUBSCRIBERS.items()):
        e = a["start"]
        for p in PERIODS:
            r = rng(SEED + 101 * i + 7 * PERIODS.index(p))
            e += 4 + int(r() * 46)
            records.append({"subscriber": sub, "period": p, "region": a["region"], "type": a["type"],
                            "index": e, "unit": "m3"})
    return records


def s3_consumption(records):
    table = {(k["subscriber"], k["period"]): k for k in records}
    out, dropped = [], 0
    for k in records:
        i = PERIODS.index(k["period"])
        if i == 0:
            continue
        prev = table.get((k["subscriber"], PERIODS[i - 1]))
        if prev is None:
            dropped += 1
            continue
        out.append(dict(k, m3=int(k["index"]) - int(prev["index"])))
    return out, dropped


def s4_summary(records):
    totals = {}
    for k in records:
        key = (k["region"], k["period"])
        totals[key] = totals.get(key, 0) + k["m3"]
    return totals, 0


def s5_serve(summary):
    region = {}
    for (b, d), v in summary.items():
        region.setdefault(b, []).append(v)
    return {b: round(sum(v) / len(v), 2) for b, v in region.items()}, 0


# ---- A SINGLE pipeline code. From the environment: the loaded periods and the rule coefficient.
# In a healthy source the screen and dedupe stages drop no rows, so consumption is
# computed directly from the loaded periods.
def report(records, periods, rule):
    v, _ = s3_consumption([k for k in records if k["period"] in periods])
    v = [dict(k, m3=k["m3"] * (rule if k["type"] == "commercial" else 1.0)) for k in v]
    summary, _ = s4_summary(v)
    serving, _ = s5_serve(summary)
    return serving, len(v)


def diff(a, b):
    return {x: round(a[x] - b[x], 2) for x in sorted(a)}


SOURCE = generate_source()
SETTINGS = {"development": (3, 0.30, 1.00), "staging": (6, 0.10, 1.00), "production": (12, 0.05, 1.05)}
LAST = {name: set(PERIODS[len(PERIODS) - n:]) for name, (n, _, _) in SETTINGS.items()}
print("environment  partitions  band  rule   consumption rows  hidden by the band  2026-07 loaded")
for name, (n, band, rule) in SETTINGS.items():
    s, rows = report(SOURCE, LAST[name], rule)
    print(f"{name:<12} {n:>10}  {band:<4}  {rule:<5}  {rows:>16}  {int(band * 200):>18}"
          f"  {'yes' if '2026-07' in LAST[name] else 'no':>14}")
print()
for name in SETTINGS:
    print(f"{name:<12} {report(SOURCE, LAST[name], SETTINGS[name][2])[0]}")

environment  partitions  band  rule   consumption rows  hidden by the band  2026-07 loaded
development           3  0.3   1.0                 400                  60              no
staging               6  0.1   1.0                1000                  20             yes
production           12  0.05  1.05               2200                  10             yes

development  {'central': 1384.0, 'north': 1265.0, 'east': 928.5, 'west': 853.5, 'south': 1102.5}
staging      {'central': 1387.6, 'north': 1229.0, 'east': 897.8, 'west': 846.2, 'south': 1077.8}
production   {'central': 1357.32, 'north': 1235.57, 'east': 902.13, 'west': 859.22, 'south': 1095.81}

Three rows are three outputs of the same code and none is wrong; each is right for its own setting. central is reported as 1384.00 in development, 1387.60 in staging and 1357.32 in production, yet the source is the same 2400 rows in all three runs.

The last column ties to the previous lesson. The development environment loads only the last three periods and 2026-07 falls outside that window, yet all six corruption classes of the Introduction to Data Engineering course corrupt exactly that period. The run that validates the pipeline never sees the period of the corruption; moreover, its band is 0.30 and already hides a loss of up to sixty rows. Validation ends green, production does not.

Where the Deviation Came From

The second block splits the deviation into two sources, then looks at the archive of eleven runs in which the rule version was deployed at the 2026-08 run.

# This block continues the report , diff , SOURCE , PERIODS , SETTINGS and LAST definitions of the first block.
ALL = set(PERIODS)
BASE, DEV, RULE = (report(SOURCE, ALL, 1.00)[0], report(SOURCE, LAST["development"], 1.00)[0],
                   report(SOURCE, ALL, 1.05)[0])
print("region   partition effect   rule effect  development minus production")
for b in sorted(BASE):
    partition_effect, rule_effect = round(DEV[b] - BASE[b], 2), round(RULE[b] - BASE[b], 2)
    print(f"{b:<8} {partition_effect:>16}  {rule_effect:>12}  {round(partition_effect - rule_effect, 2):>28}")

# ---- ARCHIVE: one run per period , all periods up to that period are loaded.
# The rule version was s1 (1.00) ; s2 (1.05) was deployed in the 2026-08 run.
CODE, DEPLOY = {"s1": 1.00, "s2": 1.05}, "2026-08"
archive = {d: ("s2" if d >= DEPLOY else "s1",
               report(SOURCE, set(PERIODS[:i + 1]), CODE["s2" if d >= DEPLOY else "s1"])[0])
           for i, d in enumerate(PERIODS) if i > 0}

# Usual period movement: how much the report would move if the code never changed.
# The first two runs are skipped , the average there carries fewer than two periods.
usual = [abs(v) for i in range(3, len(PERIODS))
         for v in diff(report(SOURCE, set(PERIODS[:i + 1]), 1.00)[0],
                       report(SOURCE, set(PERIODS[:i]), 1.00)[0]).values()]
y07, y08 = set(PERIODS[:7]), set(PERIODS[:8])
observed = diff(archive["2026-08"][1], archive["2026-07"][1])
from_data = diff(report(SOURCE, y08, 1.00)[0], report(SOURCE, y07, 1.00)[0])
from_code = diff(report(SOURCE, y08, 1.05)[0], report(SOURCE, y08, 1.00)[0])
print("\nregion      observed jump  from data  from code")
for b in sorted(observed):
    print(f"{b:<8} {observed[b]:>16}  {from_data[b]:>9}  {from_code[b]:>9}")
print(f"usual period movement (code fixed) {round(min(usual), 2)} to {round(max(usual), 2)} ;"
      f" trace left by the code {min(from_code.values())} to {max(from_code.values())}")

unstamped = {d: {"period": d, "serving": s} for d, (_, s) in archive.items()}
stamped = {d: dict(x, code=archive[d][0], setting=f"production/{len(PERIODS)}/{CODE[archive[d][0]]:.2f}")
           for d, x in unstamped.items()}
identity = [sum("code" in x and "setting" in x for x in a.values()) for a in (unstamped, stamped)]
print(f"\narchive {len(archive)} reports ; carrying identity: unstamped {identity[0]} ,"
      f" stamped {identity[1]} ; runs for identification: unstamped"
      f" {len(CODE) * len(archive)} , stamped 0")
print(f"2026-08 stamp: {stamped['2026-08']['code']} / {stamped['2026-08']['setting']}")

region   partition effect   rule effect  development minus production
central             62.55         35.87                         26.68
east                47.14         20.77                         26.37
north               52.45         23.02                         29.43
south               36.95         30.26                          6.69
west                20.41         26.13                         -5.72

region      observed jump  from data  from code
central             52.24      17.53      34.71
east                15.59      -4.81       20.4
north                9.88      -11.4      21.28
south                25.1      -3.76      28.86
west                13.74     -11.88      25.62
usual period movement (code fixed) 0.42 to 38.3 ; trace left by the code 20.4 to 34.71

archive 11 reports ; carrying identity: unstamped 0 , stamped 11 ; runs for identification: unstamped 22 , stamped 0
2026-08 stamp: s2 / production/12/1.05

The two effects in the first table are of the same order of magnitude: the number of partitions shifts serving between 20.41 and 62.55, the rule version between 20.77 and 35.87, both upward. Someone who sees the report rise by thirty units cannot tell by looking at serving whether it came from the rule version or from the number of loaded periods. The west row says something worse: the two effects are +20.41 and +26.13, and the total difference is −5.72. A number that says almost nothing has changed is two large effects cancelling each other. A small difference is not proof of a small change.

In the second table the reader sees only the observed jump, between 9.88 and 52.24. The decomposition says what it is: in four of the five regions the data went down and the report went up; in west, consumption fell by 11.88 while the report rose by 13.74. The last line closes the door: the trace left by the code is 20.40 to 34.71, the usual period movement 0.42 to 38.30. The trace of the code is inside the usual; there is no way to tell from the jump that a deployment took place.

What remains is the stamp. In the unstamped archive none of the eleven reports carries an identity; in the stamped one all eleven do. Finding out with which code a report in the unstamped archive was produced requires re-running two versions across eleven periods, that is 22 runs — if every old version can still be installed. The scope is measured too: the rule version was in the environment setting, not in the code, so a stamp holding only the code fingerprint would give s1 and s2 the same stamp. The boundary of the stamp is not the boundary of the code, it is the boundary of everything that determines the run.

Summary

• The same pipeline code produces three separate reports in three environments and none of them is wrong: 400, 1000 and 2200 consumption rows come out of the same 2400 rows, and central is reported as 1384.00, 1387.60 and 1357.32.
• The development environment never loads 2026-07 and its band hides a loss of up to sixty rows; the validating run ends green without seeing the period in which the corruption lies.
• The effect of the environment setting (20.41 … 62.55) and the effect of the rule version (20.77 … 35.87) are of the same order of magnitude; in west the two cancel each other and bring the difference down to −5.72.
• The trace left by the code (20.40 … 34.71) stays inside the usual period movement (0.42 … 38.30), and in four regions the report rises while the data goes down.
• The number of reports that carry an identity is 0 in the unstamped archive and 11 in the stamped one; identification requires 22 runs, and the stamp covers not the code but the whole setting, including the rule version.

Course Wrap-Up

The rows share no common unit; which measure each row is read by is written next to the lesson name. The difference in parentheses is the setup with orchestration minus the baseline without orchestration; the sign is read from the measure name.

The table is a single rule written sixteen times. A pipeline’s number is not that it ran correctly once, but that it gives the same result when re-run; a pipeline that cannot be re-run counts as unmeasured. The measure names are the rule itself: in Workflow Schedulers the re-run task falls from 48 to 5, in ETL and ELT the recomputable rule rises from 0 to 3, in Reverse ETL, as the pipeline reads its own output, the difference between runs leaves 0.00 and reaches 47.00.

The second claim is read from the same table: orchestration also corrupts. In Connectors and API Consumption the retry eats the quota, drops the rows ingested from 2075 to 2025 and wipes out one period. In Reverse ETL, writing back raises the difference between runs to 47.00 and corrupts 55 of 55 cells, in Retry and Alerting a three-try retry duplicates 240 rows, in Backfill a partial backfill skews the report between −16.98 and −29.57, in Dependency and Triggering the safety window raises the runs to 24. In Pipeline Observability, as the band widens, the number of classes volume sees falls to zero; in this lesson the environment setting skews the same code by up to 62.55.

The third claim is quieter: the served number does not change or looks plausible. In Stream Processing, in none of nine setups do the five served values change, and ten cells are dirty. In Delivery Guarantees nineteen duplicates leave the serving difference empty, and the loss is caught by no check. In Retry and Alerting the raw row count stays at 2320 in the half-written partition and two checks are green, in Testable Transformations the rule that shifts the period label leaves 2200 rows and five values the same, in Staged Layers three of the six stopped classes do not change the served value, in Pipeline Observability unit_changed is invisible to three indicators at once and in this lesson the trace left by the code falls inside the usual movement. In nine lessons the pipeline serves a wrong number and nothing says so.

In sixteen lessons the pipeline was handled as if it ran on a single machine: tasks lined up, failed, were retried and forty-eight nodes were laid out in a single list. When the data does not fit on a single machine, how the work will be split, which partition will delay the finish when the partitions are unbalanced and the cost of moving data over the network were never asked. The next course measures not the order of a task but into how many pieces a task is split and how much data those pieces send to one another.