--- title: 'Environment Selection' source: 'https://academia.sh/en/courses/data-analysis-with-python/environment-selection' course: 'Data Analysis with Python' language: en updated: '2026-10-07T04:31:10+00:00' license: 'CC BY-SA 4.0' objectives: - 'Compares the three environments on the criteria of team, ecosystem and portability to production with countable units and shows that each criterion puts a different environment in front' - 'Finds on which criterion the choice changes with a weight sweep: for each criterion, the smallest weight that turns the choice is written' - 'Computes how many points and which choice three normalization rules produce from the same raw numbers' - 'Gathers the gain and the hidden default of twenty-two lessons in a single table and gives the hidden default count' --- # Environment Selection Three environments are compared by number on three criteria, and each criterion puts a different environment in front: the number of people who can write in a team of nine comes out 7 / 2 / 4, the share of the nine steps covered by ready-made parts 7 / 9 / 5, the share of the nine production items covered by the environment 3 / 1 / 9. At equal weights the raw total is 17 / 12 / 18 and the choice is decided by a score. A weight sweep shows that the choice changes on all three of the three criteria: when the team weight is 2, the ecosystem weight is 2 and the production weight is 0, the choice moves to another environment; the third environment is chosen only when the ecosystem weight reaches 4, with 39 points. When the same nine numbers are normalized by stretching them over the range, the scores become 175 / 100 / 140 and the choice turns without a single raw number changing. Three lessons measured three approaches: in how many statements the same analysis is expressed, with how many verbs the transformation is written, with how many layers the chart is built. In all three, the measured thing was the form of the expression, and in none of them was one environment said to be better than another. What remains is the choice itself. This lesson's number is the **number per criterion**. Three environments are counted on three criteria, and what is searched for is the criterion on which the choice changes. Next to it stands the course's second number: the operation that reduces three criteria to a single score is also an abstraction, and it carries its own defaults. - **AE13.** The dataset is inherited from Data Collection and Preparation and is **fiction**: 3,199 rows, 1,329 subscribers, 10/25/40 m³ tariff tiers. The seed is written in the code. - **AE14.** The three environments are named by their **approaches** and are **modeled** in Python: the array and dataframe interface of a general-purpose language, the whole-object environment of an analysis-oriented language, a declarative query environment where the computation is done in the data store. No library or product name appears. - **AE15.** The criteria and their units are as follows. **Team:** the number of people in a team of nine who can write the analysis in that environment. **Ecosystem:** the number of the analysis's nine steps that are covered by ready-made parts. **Portability to production:** the number of nine production items that the environment covers. - **AE16.** The inventory is **fiction** and is for this team's own nine steps; it is not a general ranking of the environments. **No claim of comparative superiority is written.** ## Three Environments, Three Criteria The analysis asked is the same in all three environments and is the anchor of the lesson: the mean consumption per region, the first three regions in descending order. What changes is not the answer but the inventory of the road to the answer. ```python # environment_selection.py — MODEL. First the 3,199-row fiction dataset of Data # Collection and Preparation is regenerated from the same seed, then the inventory # of the three environments is modeled with the standard library. import math SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21), ("south", 0.22, 17), ("east", 0.18, 26), ("west", 0.14, 14), ("central", 0.18, 23)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 # Household and satisfaction are drawn but not used in this lesson; the generator order is kept. _subscribers, _dropped = [], [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) (_dropped if r() < 0.046 else _subscribers).append( {"subscriber_id": 10001 + i, "region": b[0], "base": b[2]}) _valid = {k["subscriber_id"] for k in _subscribers} READINGS = [] for k in _subscribers + _dropped: r = rng(SEED + 7000 + k["subscriber_id"]) for d in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): t = 0.0 if r() < 0.038 else math.floor( k["base"] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - d * 0.05) * 100 + 0.5) / 100 if k["subscriber_id"] not in _valid: r() continue READINGS.append({"region": k["region"], "consumption_m3": t}) r() ENVIRONMENTS = ["general-purpose", "analysis-oriented", "declarative"] G, A, B = ENVIRONMENTS STEPS = ["read", "type", "filter", "join", "group", "sample", "text", "chart", "publish"] ITEMS = ["version", "lock", "scheduling", "monitoring", "error", "access", "log", "rollback", "scale"] # The inventory is FICTION and is for this team's own nine steps; it is not a general # ranking of the environments. The second column is the environments that can write, the third those that can read. TEAM = [("analyst-1", {G, A, B}, {G, A, B}), ("analyst-2", {G, A}, {G, A, B}), ("analyst-3", {G, B}, {G, A, B}), ("analyst-4", {G, B}, {G, B}), ("analyst-5", {G}, {G, B}), ("analyst-6", {G}, {G, B}), ("analyst-7", {G}, {G}), ("analyst-8", {B}, {G, B}), ("analyst-9", set(), {G})] READY = {G: set(STEPS) - {"chart", "publish"}, A: set(STEPS), B: set(STEPS[:5])} COVERED = {G: {"version", "lock", "log"}, A: {"version"}, B: set(ITEMS)} CRITERIA = {"team": {o: sum(1 for _, y, _ in TEAM if o in y) for o in ENVIRONMENTS}, "ecosystem": {o: len(READY[o]) for o in ENVIRONMENTS}, "production": {o: len(COVERED[o]) for o in ENVIRONMENTS}} def align(w): return lambda h: "".join(str(v).ljust(w[0]) if j == 0 else str(v).rjust(w[j]) for j, v in enumerate(h)) def top_three(rows): t, n = {}, {} for s in rows: t[s["region"]] = t.get(s["region"], 0.0) + s["consumption_m3"] n[s["region"]] = n.get(s["region"], 0) + 1 return sorted(((b, t[b] / n[b]) for b in t), key=lambda p: -p[1])[:3] print(f"fiction dataset (seed {SEED}): {len(READINGS)} rows, {len(_valid)} subscribers") print("the analysis asked is the same in all three environments: " + ", ".join(f"{b} {v:.3f}" for b, v in top_three(READINGS))) y = align([27, 18, 20, 14]) print("\n" + y(["criterion (out of 9)"] + ENVIRONMENTS)) for k, d in CRITERIA.items(): print(y([k] + [d[o] for o in ENVIRONMENTS])) print(y(["raw total (out of 27)"] + [sum(CRITERIA[k][o] for k in CRITERIA) for o in ENVIRONMENTS])) ``` ``` fiction dataset (seed 20260218): 3199 rows, 1329 subscribers the analysis asked is the same in all three environments: east 24.674, central 22.396, north 20.210 criterion (out of 9) general-purpose analysis-oriented declarative team 7 2 4 ecosystem 7 9 5 production 3 1 9 raw total (out of 27) 17 12 18 ``` The table starts with a single observation: **all three criteria put a different environment in front**. In the team, the general-purpose environment is ahead with 7 people; in the ecosystem, the analysis-oriented environment with 9 steps; in portability to production, the declarative environment with 9 items. In the declarative environment there are four steps to be built by hand: resampling, text parsing, layered charts and publishing. In such a table there is no answer to the question "which is better", because the criteria count different things. The only question that has an answer is this: how will these three numbers be combined. The raw total gives 17, 12 and 18, and the choice is decided by **a single score**. ## On Which Criterion the Choice Changes The raw total gives equal weight to the three criteria; this equality is not measured, it is assumed. Changing the weight criterion by criterion and looking at where the choice turns is the countable way of testing the assumption. ```python def score(weight): return {o: sum(weight[k] * CRITERIA[k][o] for k in CRITERIA) for o in ENVIRONMENTS} def choice(p): return max(ENVIRONMENTS, key=lambda o: p[o]) def fmt(p): return " / ".join(str(p[o]) for o in ENVIRONMENTS) ONE = {k: 1 for k in CRITERIA} BASE = score(ONE) y2 = align([18, 20, 20, 20, 20, 20]) print("\n" + y2(["criterion weight"] + list(range(5)))) for k in CRITERIA: print(y2([k] + [choice(score({**ONE, k: w})) for w in range(5)])) print(f"\nchoice at equal weights: {choice(BASE)} ({fmt(BASE)})") for k in CRITERIA: candidates = [(abs(w - 1), w) for w in range(13) if w != 1 and choice(score({**ONE, k: w})) != choice(BASE)] w = min(candidates)[1] p = score({**ONE, k: w}) print(f" {k} weight 1 -> {w}: choice {choice(p)} ({fmt(p)})") which = [w for w in range(13) if choice(score({**ONE, "ecosystem": w})) == A] p = score({**ONE, "ecosystem": which[0]}) print(f" the analysis-oriented environment is chosen only when the ecosystem weight is " f"{which[0]} ({fmt(p)})") ``` ``` criterion weight 0 1 2 3 4 team declarative declarative general-purpose general-purpose general-purpose ecosystem declarative declarative general-purpose general-purpose analysis-oriented production general-purpose declarative declarative declarative declarative choice at equal weights: declarative (17 / 12 / 18) team weight 1 -> 2: choice general-purpose (24 / 14 / 22) ecosystem weight 1 -> 2: choice general-purpose (24 / 21 / 23) production weight 1 -> 0: choice general-purpose (14 / 11 / 9) the analysis-oriented environment is chosen only when the ecosystem weight is 4 (38 / 39 / 33) ``` The answer is in the table: **the choice changes on all three of the three criteria**, but the thresholds are different. When the team weight rises from 1 to 2, the choice moves to the general-purpose environment and the scores become 24 and 22; when the ecosystem weight is 2, it moves to the same place; if portability to production is not counted at all, again to the same place. The reason all three can turn the choice is a starting gap of one point. The behavior of the third environment has to be written separately. The analysis-oriented environment gets the lowest total, 12 points, at equal weights; when the ecosystem weight is 4 it becomes the chosen environment with 39 points. No number has changed; the only thing that has changed is how many times the ready-made coverage of the nine steps is counted. A weight is not a measurement but a decision. ## The Criterion's Own Default Weight is not the only default. Before the numbers are added, they are made comparable, and that too is a rule. Three rules are put side by side: adding the raw numbers, counting the highest in each criterion as 100 and taking the others in proportion, counting the highest as 100 and the lowest as 0 and stretching what lies between. ```python def normalize(rule, criteria): total = {o: 0.0 for o in ENVIRONMENTS} for d in criteria.values(): v = [d[o] for o in ENVIRONMENTS] for o in ENVIRONMENTS: if rule == "raw count": total[o] += d[o] elif rule == "ratio to best": total[o] += 100.0 * d[o] / max(v) else: total[o] += 100.0 * (d[o] - min(v)) / (max(v) - min(v)) return total y3 = align([22, 18, 20, 14, 18]) print("\n" + y3(["normalization"] + ENVIRONMENTS + ["choice"])) for rule in ("raw count", "ratio to best", "range stretching"): t = normalize(rule, CRITERIA) print(y3([rule] + [f"{t[o]:.1f}" for o in ENVIRONMENTS] + [choice(t)])) READER = dict(CRITERIA, team={o: sum(1 for _, _, k in TEAM if o in k) for o in ENVIRONMENTS}) print("\n" + y3(["team criterion"] + ENVIRONMENTS + ["clear margin"])) for name, criteria in (("can write", CRITERIA), ("can read", READER)): t = normalize("raw count", criteria) s = sorted((t[o] for o in ENVIRONMENTS), reverse=True) print(y3([name] + [f"{t[o]:.0f}" for o in ENVIRONMENTS] + [f"{s[0] - s[1]:.0f}"])) ``` ``` normalization general-purpose analysis-oriented declarative choice raw count 17.0 12.0 18.0 declarative ratio to best 211.1 139.7 212.7 declarative range stretching 175.0 100.0 140.0 general-purpose team criterion general-purpose analysis-oriented declarative clear margin can write 17 12 18 1 can read 19 13 21 2 ``` The three rules produce three different scores from the same nine numbers, and the third one turns the choice. A one-point difference in the raw total corresponds to 1.6 points between 211.1 and 212.7 in the ratio rule, and the choice stays the same. The range stretching rule zeroes the lowest value in every criterion and compresses the middle value according to the width of the range: in the production criterion the range is 8 units, so the middle value of 3 points falls to 25 points and the totals become 175 and 140. The ranking has changed without a single raw number changing. The second table shows one level further in. The team criterion was defined as the number of people who can write; the number of people who can read is also a defensible definition. When the definition changes, the choice does not change in this set, but the totals become 19 / 13 / 21 and the clear margin rises from 1 point to 2 points; the same choice appears to have been given with twice the confidence. The function that reduces three criteria to a single score is an abstraction; the weight, the normalization rule and the criterion definition are its defaults. A score whose defaults are not written is not a measurement but a choice. ## Summary - When three environments are counted on three criteria, each criterion puts a different environment in front: team 7 / 2 / 4, ecosystem 7 / 9 / 5, portability to production 3 / 1 / 9. - At equal weights the raw total is 17 / 12 / 18 and the choice is decided by a one-point difference. - The choice changes on all three of the three criteria, and the thresholds are different: team 2, ecosystem 2, production 0. The third environment is chosen only when the ecosystem weight reaches 4, with 39 points. - The three normalization rules give 17 / 12 / 18, 211.1 / 139.7 / 212.7 and 175 / 100 / 140 points; the third turns the choice without a single raw number changing. - When the team criterion is defined as the number who can read, the choice stays the same and the clear margin rises from 1 point to 2 points. ## Course Wrap-Up The account of twenty-two lessons stands in a single table, and each number was read from its own lesson. | Lesson | By hand | With abstraction | Hidden default | |---|---|---|---| | Numerical 01 Working Environment | 2/17 steps identical | 11/17 with lock file | 5 decisions, 1,116 → 1,264 | | Numerical 02 Arrays | 22 lines, 188,864 bytes | 2 lines, 30,320 bytes | 2 decisions, 31.896 ↔ 26.839 | | Numerical 03 Vectorized Operations | 10 lines, 1 pass | 2 lines, 3 passes | 3 decisions, 102,034 → 1,240 | | Numerical 04 Broadcasting | 30,320 bytes extra | 0 bytes extra | 2 decisions, 1,260 → 1,587,600 | | Numerical 05 Basic Linear Algebra Operations | 823,059,744 operations | 704 operations | 2 decisions, 50.09% deviation | | Dataframes 01 Series and Dataframe | 781,710 comparisons | 2,361 passes | 2 decisions, 3,316.42 m³ difference | | Dataframes 02 Reading and Writing Data | 7 conversion rules | 1 read call | 3 decisions, 1,123 cells to missing | | Dataframes 03 Indexing and Slicing | 2,658 comparisons | 21 passes | 3 decisions, 95 / 94 / 617 rows | | Dataframes 04 Filtering and Querying | 5,034 passes, 16,184 bytes | 6,398 passes, 3,399 bytes | 2 decisions, 1,835 / 2,026 / 1,915 | | Dataframes 05 Grouping and Aggregation | 15,995 key lookups | 3,199 passes | 2 decisions, 160 rows drop | | Dataframes 06 Join Operations | 2,616,782 comparisons | 4,017 passes | 2 decisions, 12,250.65 m³ excess | | Dataframes 07 Time Series Operations | 41,587 passes | 3,212 passes | 3 decisions, 4,850.82 ↔ 7,882.58 | | Dataframes 08 Text Operations | 9 lines, 0 lines of output | 3 calls, 3,199 rows | 3 decisions, 256 rows silently | | Performance 01 Memory Usage | 173,837 bytes | 3,628 bytes, 47.9 times | 3 decisions, 108 rows change tier | | Performance 02 Chunked Processing | 1,152,229 bytes peak | 36,132 bytes, 32 times | 3 decisions, 1,035 → 0 subscribers | | Performance 03 Columnar Engines | 19,194 values | 2,835 values, 6.8 times | 3 decisions, 2,835 → 4,883 values | | Performance 04 Parallel and Distributed Dataframes | 3,199 units | 8,921 units, threshold 10,833 | 3 decisions, 10,833 → 20,119 threshold | | Performance 05 Taking the Query to the Data | 3,199 rows moved | 5 rows, 1,919 times | 2 decisions, 1,919 → 11 times | | Alternative 01 Statistical Programming Languages | 6 statements, 6 intermediate variables | 1 statement, 0 intermediate variables | 5 decisions, 1.473 m³ gap | | Alternative 02 Grammar of Dataframes | 7 statements, 6 intermediate variables | 1 statement, 0 intermediate variables | 2 decisions, 7.301 → 0.593 m³ | | Alternative 03 Grammar of Graphics | 9 arguments | 5 layers, 5 reused | 2 decisions, 921 ↔ 24.674 | | Alternative 04 Environment Selection | 3 criteria, 3 leading | 1 score, 1 choice | 3 decisions, 1-point difference | The direction of the second and third columns is not fixed: sometimes the abstraction shortens the work (2,616,782 comparisons fall to 4,017 passes), sometimes it does not (11,340 multiplications are the same in both writings), sometimes it raises the price (30,320 bytes rise to 90,960 bytes). The direction of the gain comes not from the abstraction but from the measured work. The fourth column is the rule of the course: **an abstraction shortens the work as much as it hides decisions; a function whose defaults you do not know is a function that decides on your behalf. A call whose defaults are not written down is counted as unmeasured.** The **hidden default count** written by the twenty-two lessons by name **is 60**. This number is not a defect but the output of the course: a default that is not counted looks like zero, and a default that looks like zero is applied silently at every call. In twenty-two lessons the toolset was built and every default of it was written down. But the tool still says only **what is**: the mean of the east region is 24.674 m³, that of the north 20.210 m³. Whether these numbers came from a sample or from the population was not asked; whether the 4.464 m³ difference between them is real or noise was not asked either. The next course, Statistical Analysis and Exploratory Data Analysis, takes up these two questions.