--- title: 'Environment and Version Management' source: 'https://academia.sh/en/courses/data-pipelines/environment-and-version-management' course: 'Data Pipelines and Orchestration' language: en updated: '2026-10-21T05:02:58+00:00' license: 'CC BY-SA 4.0' objectives: - 'Parameterizes a pipeline code with settings that come from the environment and compares the serving the same code produces in three environments.' - 'Decomposes the deviation in the report into the share that comes from the environment setting and the share that comes from the rule version.' - 'Counts the identity a version stamp adds to a run archive and the run cost of identification without a stamp.' - 'Collects the figures of the sixteen lessons of the course, without and with orchestration, into a single table.' --- # Environment and Version Management Counting the different result the same pipeline code produces in three environments and the identity that is lost when no version stamp is written: the development, staging and production environments produce 400, 1000 and 2200 consumption rows respectively from the same 2400 rows and their reports differ from one another; the number of loaded partitions shifts serving between 20.41 and 62.55, the rule version between 20.77 and 35.87, and because the two effects are of the same order of magnitude they cannot be told apart by looking at serving, and in one region they cancel each other and bring the difference down to −5.72. In an archive of eleven runs, when the rule version changes after the seventh period, the share of the code in the observed jump is between 20.40 and 34.71 and stays within the range of 0.42 to 38.30 that the usual period movement would show if the code never changed; in four regions the data goes down while the report goes up. In the unstamped archive none of the 11 reports carries an identity and identification requires 22 runs. The lesson is the close of the course. The previous lesson placed three indicators on top of the pipeline and counted what each of them sees. Every number leaned on a single silent assumption: the code was always the same and always ran with the same setting. Yet the band was 0.05, the tolerance was 1, the number of loaded periods was twelve — all three stood **around** the run, not in the code of the pipeline. This lesson turns that surroundings into an object of measurement. A single pipeline code is deployed to three environments and two questions are asked: how different a result the same code gives in them, and whether a report in your hands shows **in which environment, with which code** it was produced. - **OO47.** The source, the five-stage pipeline and the three environment settings are **fictional** and are generated inside the lesson. No orchestrator, deployment tool or version management product is called. The seed is 20260218. - **OO48.** The procedure of continuous integration, delivery and environment promotion was measured in the Continuous Integration and Delivery and Infrastructure as Code courses of the DevOps and Platform Engineering curriculum; it is **not repeated**. The only measure here is the **different result** the same code produces across environments. - **OO49.** An environment carries three settings and all three are outside the code: the number of loaded **partitions**, the indicator **band** and the **version** of the reporting rule. `s1` reports commercial consumption with a coefficient of 1.00, `s2` with 1.05; the backfill procedure was measured in the last lesson of the Transformation topic. - **OO50.** A **version stamp** is the triple of period, code version and setting written to the output of a run. The baseline is the run without orchestration: a single environment is assumed, no stamp is written, and the number of reports that carry an identity is **0**. ## The Same Code, Three Environments The block below defines a single `report` function and runs it with three settings. From the environment the function takes only the set of periods to load and the rule coefficient; because the band tunes the indicator and not the pipeline, it enters the table with the measure of the previous lesson, the largest loss it hides. ```python # FICTIONAL. The meter readings of the measurement network, the five-stage pipeline and three # environment settings are generated inside the lesson; no orchestrator, deployment tool or # version management product is called. Seed 20260218. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value REGION = ["east", "west", "north", "south", "central"] TYPE = ["residential", "commercial"] PERIODS = [f"2026-{a:02d}" for a in range(1, 13)] SUBSCRIBERS = {} for i in range(200): r = rng(SEED + 37 * i) SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)], "start": 10000 + int(r() * 60000)} def generate_source(): records = [] for i, (sub, a) in enumerate(SUBSCRIBERS.items()): e = a["start"] for p in PERIODS: r = rng(SEED + 101 * i + 7 * PERIODS.index(p)) e += 4 + int(r() * 46) records.append({"subscriber": sub, "period": p, "region": a["region"], "type": a["type"], "index": e, "unit": "m3"}) return records def s3_consumption(records): table = {(k["subscriber"], k["period"]): k for k in records} out, dropped = [], 0 for k in records: i = PERIODS.index(k["period"]) if i == 0: continue prev = table.get((k["subscriber"], PERIODS[i - 1])) if prev is None: dropped += 1 continue out.append(dict(k, m3=int(k["index"]) - int(prev["index"]))) return out, dropped def s4_summary(records): totals = {} for k in records: key = (k["region"], k["period"]) totals[key] = totals.get(key, 0) + k["m3"] return totals, 0 def s5_serve(summary): region = {} for (b, d), v in summary.items(): region.setdefault(b, []).append(v) return {b: round(sum(v) / len(v), 2) for b, v in region.items()}, 0 # ---- A SINGLE pipeline code. From the environment: the loaded periods and the rule coefficient. # In a healthy source the screen and dedupe stages drop no rows, so consumption is # computed directly from the loaded periods. def report(records, periods, rule): v, _ = s3_consumption([k for k in records if k["period"] in periods]) v = [dict(k, m3=k["m3"] * (rule if k["type"] == "commercial" else 1.0)) for k in v] summary, _ = s4_summary(v) serving, _ = s5_serve(summary) return serving, len(v) def diff(a, b): return {x: round(a[x] - b[x], 2) for x in sorted(a)} SOURCE = generate_source() SETTINGS = {"development": (3, 0.30, 1.00), "staging": (6, 0.10, 1.00), "production": (12, 0.05, 1.05)} LAST = {name: set(PERIODS[len(PERIODS) - n:]) for name, (n, _, _) in SETTINGS.items()} print("environment partitions band rule consumption rows hidden by the band 2026-07 loaded") for name, (n, band, rule) in SETTINGS.items(): s, rows = report(SOURCE, LAST[name], rule) print(f"{name:<12} {n:>10} {band:<4} {rule:<5} {rows:>16} {int(band * 200):>18}" f" {'yes' if '2026-07' in LAST[name] else 'no':>14}") print() for name in SETTINGS: print(f"{name:<12} {report(SOURCE, LAST[name], SETTINGS[name][2])[0]}") ``` ``` environment partitions band rule consumption rows hidden by the band 2026-07 loaded development 3 0.3 1.0 400 60 no staging 6 0.1 1.0 1000 20 yes production 12 0.05 1.05 2200 10 yes development {'central': 1384.0, 'north': 1265.0, 'east': 928.5, 'west': 853.5, 'south': 1102.5} staging {'central': 1387.6, 'north': 1229.0, 'east': 897.8, 'west': 846.2, 'south': 1077.8} production {'central': 1357.32, 'north': 1235.57, 'east': 902.13, 'west': 859.22, 'south': 1095.81} ``` Three rows are three outputs of the same code and none is wrong; each is right for its own setting. `central` is reported as 1384.00 in development, 1387.60 in staging and 1357.32 in production, yet the source is the same 2400 rows in all three runs. The last column ties to the previous lesson. The development environment loads only the last three periods and `2026-07` falls outside that window, yet all six corruption classes of the Introduction to Data Engineering course corrupt exactly that period. The run that validates the pipeline **never sees** the period of the corruption; moreover, its band is 0.30 and already hides a loss of up to sixty rows. Validation ends green, production does not. ## Where the Deviation Came From The second block splits the deviation into two sources, then looks at the archive of eleven runs in which the rule version was deployed at the `2026-08` run. ```python # This block continues the report , diff , SOURCE , PERIODS , SETTINGS and LAST definitions of the first block. ALL = set(PERIODS) BASE, DEV, RULE = (report(SOURCE, ALL, 1.00)[0], report(SOURCE, LAST["development"], 1.00)[0], report(SOURCE, ALL, 1.05)[0]) print("region partition effect rule effect development minus production") for b in sorted(BASE): partition_effect, rule_effect = round(DEV[b] - BASE[b], 2), round(RULE[b] - BASE[b], 2) print(f"{b:<8} {partition_effect:>16} {rule_effect:>12} {round(partition_effect - rule_effect, 2):>28}") # ---- ARCHIVE: one run per period , all periods up to that period are loaded. # The rule version was s1 (1.00) ; s2 (1.05) was deployed in the 2026-08 run. CODE, DEPLOY = {"s1": 1.00, "s2": 1.05}, "2026-08" archive = {d: ("s2" if d >= DEPLOY else "s1", report(SOURCE, set(PERIODS[:i + 1]), CODE["s2" if d >= DEPLOY else "s1"])[0]) for i, d in enumerate(PERIODS) if i > 0} # Usual period movement: how much the report would move if the code never changed. # The first two runs are skipped , the average there carries fewer than two periods. usual = [abs(v) for i in range(3, len(PERIODS)) for v in diff(report(SOURCE, set(PERIODS[:i + 1]), 1.00)[0], report(SOURCE, set(PERIODS[:i]), 1.00)[0]).values()] y07, y08 = set(PERIODS[:7]), set(PERIODS[:8]) observed = diff(archive["2026-08"][1], archive["2026-07"][1]) from_data = diff(report(SOURCE, y08, 1.00)[0], report(SOURCE, y07, 1.00)[0]) from_code = diff(report(SOURCE, y08, 1.05)[0], report(SOURCE, y08, 1.00)[0]) print("\nregion observed jump from data from code") for b in sorted(observed): print(f"{b:<8} {observed[b]:>16} {from_data[b]:>9} {from_code[b]:>9}") print(f"usual period movement (code fixed) {round(min(usual), 2)} to {round(max(usual), 2)} ;" f" trace left by the code {min(from_code.values())} to {max(from_code.values())}") unstamped = {d: {"period": d, "serving": s} for d, (_, s) in archive.items()} stamped = {d: dict(x, code=archive[d][0], setting=f"production/{len(PERIODS)}/{CODE[archive[d][0]]:.2f}") for d, x in unstamped.items()} identity = [sum("code" in x and "setting" in x for x in a.values()) for a in (unstamped, stamped)] print(f"\narchive {len(archive)} reports ; carrying identity: unstamped {identity[0]} ," f" stamped {identity[1]} ; runs for identification: unstamped" f" {len(CODE) * len(archive)} , stamped 0") print(f"2026-08 stamp: {stamped['2026-08']['code']} / {stamped['2026-08']['setting']}") ``` ``` region partition effect rule effect development minus production central 62.55 35.87 26.68 east 47.14 20.77 26.37 north 52.45 23.02 29.43 south 36.95 30.26 6.69 west 20.41 26.13 -5.72 region observed jump from data from code central 52.24 17.53 34.71 east 15.59 -4.81 20.4 north 9.88 -11.4 21.28 south 25.1 -3.76 28.86 west 13.74 -11.88 25.62 usual period movement (code fixed) 0.42 to 38.3 ; trace left by the code 20.4 to 34.71 archive 11 reports ; carrying identity: unstamped 0 , stamped 11 ; runs for identification: unstamped 22 , stamped 0 2026-08 stamp: s2 / production/12/1.05 ``` The two effects in the first table are of the same order of magnitude: the number of partitions shifts serving between 20.41 and 62.55, the rule version between 20.77 and 35.87, both upward. Someone who sees the report rise by thirty units cannot tell **by looking at serving** whether it came from the rule version or from the number of loaded periods. The `west` row says something worse: the two effects are +20.41 and +26.13, and the total difference is **−5.72**. A number that says almost nothing has changed is two large effects cancelling each other. **A small difference is not proof of a small change.** In the second table the reader sees only the observed jump, between 9.88 and 52.24. The decomposition says what it is: in four of the five regions **the data went down and the report went up**; in `west`, consumption fell by 11.88 while the report rose by 13.74. The last line closes the door: the trace left by the code is 20.40 to 34.71, the usual period movement 0.42 to 38.30. The trace of the code is **inside the usual**; there is no way to tell from the jump that a deployment took place. What remains is the stamp. In the unstamped archive **none** of the eleven reports carries an identity; in the stamped one all eleven do. Finding out with which code a report in the unstamped archive was produced requires re-running two versions across eleven periods, that is **22 runs** — if every old version can still be installed. The scope is measured too: the rule version was in the environment setting, not in the code, so a stamp holding only the code fingerprint would give `s1` and `s2` **the same stamp**. **The boundary of the stamp is not the boundary of the code, it is the boundary of everything that determines the run.** ## Summary - The same pipeline code produces three separate reports in three environments and none of them is wrong: 400, 1000 and 2200 consumption rows come out of the same 2400 rows, and `central` is reported as 1384.00, 1387.60 and 1357.32. - The development environment never loads `2026-07` and its band hides a loss of up to sixty rows; the validating run ends green without seeing the period in which the corruption lies. - The effect of the environment setting (20.41 … 62.55) and the effect of the rule version (20.77 … 35.87) are of the same order of magnitude; in `west` the two cancel each other and bring the difference down to **−5.72**. - The trace left by the code (20.40 … 34.71) stays inside the usual period movement (0.42 … 38.30), and in four regions the report rises while the data goes down. - The number of reports that carry an identity is **0** in the unstamped archive and **11** in the stamped one; identification requires 22 runs, and the stamp covers not the code but the whole setting, including the rule version. ## Course Wrap-Up The rows share no common unit; **which measure** each row is read by is written next to the lesson name. The difference in parentheses is **the setup with orchestration minus the baseline without orchestration**; the sign is read from the measure name. | Lesson | Baseline without orchestration | Setup with orchestration | Cost | |---|---|---|---| | Ingestion Types — rows transported and changes missed | full load **15,560**, missed **0** | incremental load **2,360** (**−13,200**), missed **148** (**+148**) | only the distribution check reports the missed ones, at stage 4 | | Connectors and API Consumption — rows ingested, page 25 | without retry **2075** | with retry **2025** (**−50**) | 112 pages arrive late, `2026-12` is never ingested | | Messaging Systems — active consumers | single partition **1/4** | four partitions **4/4** (**+3**) | on the subscriber key the load skews between 708 and 492 | | Stream Processing — dropped rows and dirty cells | tolerance 0: **40** and **10/55** | tolerance 1: **0** and **0** (**−40**, **−10**) | the carried state grows from 400 to **600** entries | | Delivery Guarantees — consumption rows and deviation | at most once **2162**, −9.18 … −28.36 | at least once **2200** (**+38**), deviation **empty** | 19 duplicates arise, the loss is below both thresholds | | Reverse ETL — difference between runs | pipeline that does not write back **0.00** | reverse ETL **47.00 / 10.90 / 14.36** | **55** of 55 cells get corrupted, four checks are green | | ETL and ELT — rule that can be recomputed | ETL **0**, 55 rows to the target | ELT **3** (**+3**), 2400 rows to the target (**+2345**) | cells read in the target rise from 110 to **33110** | | Transformation Frameworks — violated dependency | manual order **1**, **7** models with missing inputs | resolver **0** and **0** (**−1**, **−7**) | the violating run raises no error, serving deviates by 17.46 … 28.37 | | Staged Layers — corruption class stopped | serving only **2/6**, 55 cells | three layers **6/6** (**+4**), 2400 + 2200 + 55 | **three** of the six caught do not change the served value | | Testable Transformations — corruption class caught | without expectations **0/6** | six expectations **4/6** (**+4**) | `duplicated` and `type_drift` are swallowed in the transformation | | Backfill — recomputed period | full apply **12** periods, 2200 rows | partial apply **2** periods, 400 rows (**−10**) | the report deviates by **−16.98 … −29.57**, six expectations are green | | Workflow Schedulers — tasks re-run per failed task | by hand **48** | with graph **5** (**−43**) | forty-seven edges are written by hand | | Dependency and Triggering — missing periods and consumption rows | fixed schedule **6** missing, **900** | data readiness trigger **0** and **2200** (**+1300**) | seven periods of total delay | | Retry and Alerting — deviation and the check that fires | half write **−49.09 … −121.82**, two checks green | atomic write **−3.25 … +12.80**, both fire | consumption falls to **1800** instead of 2040 | | Pipeline Observability — corruption class seen | no indicator **0/6**, lag undefined | three indicators **5/6** (**+5**), lag 0–1 periods | `unit_changed` stays invisible and leaves two periods of garbage | | Environment and Version Management — report carrying identity | unstamped archive **0/11** | stamped archive **11/11** (**+11**) | identification requires **22 runs** when unstamped | The table is a single rule written sixteen times. **A pipeline's number is not that it ran correctly once, but that it gives the same result when re-run; a pipeline that cannot be re-run counts as unmeasured.** The measure names are the rule itself: in Workflow Schedulers the re-run task falls from 48 to 5, in ETL and ELT the recomputable rule rises from 0 to 3, in Reverse ETL, as the pipeline reads its own output, the difference between runs leaves 0.00 and reaches 47.00. **The second claim is read from the same table: orchestration also corrupts.** In Connectors and API Consumption the retry eats the quota, drops the rows ingested from 2075 to 2025 and wipes out one period. In Reverse ETL, writing back raises the difference between runs to 47.00 and corrupts 55 of 55 cells, in Retry and Alerting a three-try retry duplicates 240 rows, in Backfill a partial backfill skews the report between −16.98 and −29.57, in Dependency and Triggering the safety window raises the runs to 24. In Pipeline Observability, as the band widens, the number of classes volume sees falls to zero; in this lesson the environment setting skews the same code by up to 62.55. **The third claim is quieter: the served number does not change or looks plausible.** In Stream Processing, in none of nine setups do the five served values change, and ten cells are dirty. In Delivery Guarantees nineteen duplicates leave the serving difference empty, and the loss is caught by no check. In Retry and Alerting the raw row count stays at 2320 in the half-written partition and two checks are green, in Testable Transformations the rule that shifts the period label leaves 2200 rows and five values the same, in Staged Layers three of the six stopped classes do not change the served value, in Pipeline Observability `unit_changed` is invisible to three indicators at once and in this lesson the trace left by the code falls inside the usual movement. In nine lessons the pipeline serves a wrong number and nothing says so. In sixteen lessons the pipeline was handled as if it ran on **a single machine**: tasks lined up, failed, were retried and forty-eight nodes were laid out in a single list. When the data does not fit on a single machine, **how the work will be split**, which partition will delay the finish when the partitions are unbalanced and the cost of **moving data over the network** were never asked. The next course measures not the order of a task but **into how many pieces a task is split** and how much data those pieces send to one another.