--- title: 'Data Analysis with Python' source: 'https://academia.sh/en/courses/data-analysis-with-python' language: en updated: '2026-10-07T04:31:10+00:00' license: 'CC BY-SA 4.0' lessons: 22 outcomes: - 'Uses the array and dataframe abstractions and writes down the default of every call' - 'Manages memory and scale constraints with a counted unit and justifies the chunking decision' - 'Runs the analysis in a reproducible environment and ties the choice of environment to criteria' --- # Data Analysis with Python Measuring together how much the array and dataframe abstractions shorten the same job and which decision they make invisible: the reproducibility condition of an isolated environment, the line, pass and byte difference between a list and the array interface, the type promotion that vectorization hides, the result broadcasting produces silently, the alignment the label takes on, the column type inference misreads, how a missing value behaves in a filter and in a group key, the row count each join type produces, the counting of an empty period in resampling, the bytes a type choice sets aside, how chunk size relates to peak memory, the number of values a columnar read reads, the cost of a distributed pass, the rows carried by taking the filter to the data, and the comparison of environment choices criterion by criterion. ## Numerical Computing Separating, by number, what the array abstraction gains from what it hides: the items that must be pinned in an isolated environment for the same notebook to give the same number on another machine, the lines, passes and bytes a list and an array interface take for the same summation job, the type promotion and overflow that vectorization hides, broadcasting on incompatible shapes returning an unexpected array instead of an error, and the difference in pass count between hand-written loops and the abstraction in matrix multiplication and solving. ### Working Environment Source: https://academia.sh/en/courses/data-analysis-with-python/working-environment Counts the items that must be pinned for a notebook to give the same number on another machine: when the nine environment items and the seventeen-step preparation pipeline of the Data Collection and Preparation course are written side by side, nothing pinned reproduces 2 of 17 steps, the dependency list 5, the lock file 11; the remaining 6 steps depend on five items the lock file never writes. Running the same notebook on two machines, the locale changes all 1,260 meter indexes and moves the total from 1,236,664.22 to 41,433,984.33, the chunk count moves the mean by 0.006053 m³ (12 times the threshold), and the notebook cell order raises the count of cells above 40 m³ from 1,116 to 1,264. The Data Collection and Preparation course closed with a seventeen-step preparation pipeline. The measure of that pipeline was **readability**: once each step's decision and row accounting were written to the output, 39 of 39 decisions were readable after the run. Readability concerns how a pipeline is **read** and does not fully close the course's question. This lesson asks the next question: when the same notebook is run on another machine, does it give the same number. The question comes before the choice of tool. As soon as an array or dataframe interface is used, part of the calculation no longer sits in the code that is written but inside the components that are installed; the version of those components, the convention by which the machine reads a number, and the number of pieces the calculation is divided into all move the result. For this reason the environment is treated here not as a list of install commands but as a **list of items that must be pinned**. The measure is likewise not the number of commands but how many steps give the same number on two machines. - **NC1.** A notebook gives the same number on another machine if every item that determines the number it produces is written down. The lock file writes only some of these items; the rest stays nowhere unless the notebook itself declares it. - **NC2.** An item that comes from the environment moves the number in two ways: at the order-of-magnitude level, that is, with an error visible to the eye; or around the threshold, that is, with a difference visible only when compared. The second escapes a check that looks at row counts. #### Writing the Environment as Data ```python # environment.py — MODEL: a run environment and the documents that pin it are written as data. # "level" is the document that pins an item. "step" was read from the seventeen-step # preparation pipeline of Data Collection and Preparation; beside each step are the items that determine its result. LEVELS = ["installed", "list", "lock", "declaration"] ITEMS = [ ("dependency name", "list"), ("direct dependency version", "lock"), ("indirect dependency version", "lock"), ("interpreter version", "lock"), ("locale", "declaration"), ("chunk count", "declaration"), ("random seed", "declaration"), ("notebook cell order", "declaration"), ("data file digest", "declaration"), ] STEPS = [ ("01 file formats", ["dependency name", "locale"]), ("02 pulling from databases", ["dependency name", "data file digest"]), ("03 collecting from interfaces", ["dependency name"]), ("04 structural parsing", ["dependency name", "indirect dependency version"]), ("05 sampling", ["direct dependency version", "random seed"]), ("06 missing values", ["dependency name"]), ("07 duplicate records", ["direct dependency version"]), ("08 type conversions", ["interpreter version", "locale"]), ("09 outliers", ["direct dependency version"]), ("10 categorical fixing", ["indirect dependency version"]), ("11 text normalization", ["interpreter version"]), ("12 merging", ["direct dependency version", "notebook cell order"]), ("13 reshaping", ["dependency name"]), ("14 grouping", ["direct dependency version", "chunk count"]), ("15 derived variables", ["interpreter version"]), ("16 categorical encoding", []), ("17 preparation pipeline", []), ] PINNED_BY = dict(ITEMS) pinned = lambda d: {a for a, k in ITEMS if LEVELS.index(k) <= LEVELS.index(d)} reproduced = lambda d: [name for name, deps in STEPS if set(deps) <= pinned(d)] row_fmt = lambda w: lambda h: "".join(str(v).ljust(w[0]) if j == 0 else str(v).rjust(w[j]) for j, v in enumerate(h)) y = row_fmt([12, 16, 22, 16]) print(y(["level", "pinned item", "reproduced step", "moving step"])) for d in LEVELS: u = reproduced(d) print(y([d, f"{len(pinned(d))}/{len(ITEMS)}", f"{len(u)}/{len(STEPS)}", len(STEPS) - len(u)])) locked = set(reproduced("lock")) print("\nsteps that still move after the lock file, and why:") for name, deps in STEPS: if name not in locked: print(" " + name + " <- " + ", ".join(b for b in deps if PINNED_BY[b] == "declaration")) ``` ``` level pinned item reproduced step moving step installed 0/9 2/17 15 list 1/9 5/17 12 lock 4/9 11/17 6 declaration 9/9 17/17 0 steps that still move after the lock file, and why: 01 file formats <- locale 02 pulling from databases <- data file digest 05 sampling <- random seed 08 type conversions <- locale 12 merging <- notebook cell order 14 grouping <- chunk count ``` Each of the four levels corresponds to a different document. At the **installed** level there is no document; the components are installed system-wide on the machine, and you can learn which version is there only by looking at the machine. Only 2 of the seventeen steps give the same result on two machines — these are the steps that touch no outside item. An **isolated environment** comes on top of this level and does a single thing: it separates the components the notebook uses from the rest of the machine. Isolation does not pin a number by itself, it makes pinning **possible**; writing the version of a component that does not sit in a separate set makes no sense, because another job can change it. A **dependency list** records the names of the components the notebook uses directly and usually gives a range. This raises the number of pinned items to 1 and the reproduced steps to 5: only the steps whose behavior does not change from version to version are gained. The list says **which component** is used, not **which version** of it. The **lock file** closes the gap in between. It writes the exact versions of the direct components, the indirect components they pull in, and the interpreter version in a single resolution; the second machine installs the same resolution. The pinned items rise to 4 and the reproduced steps to **11**. The gain ends here and the hidden default stands at exactly this point: because the lock file writes a full version resolution, the whole environment is assumed to be pinned. The five items that are not pinned keep moving **6 steps**, and none of these six steps has anything to do with versions. The last level is not a file but an **environment declaration** that sits at the head of the notebook: the locale it is read with, the number the calculation is divided into, the seed used, the order in which the cells run, and the digest of the input file. When these five items are written, the ratio becomes 17 of 17. The functions of the four documents do not replace one another, and this distinction cannot be read from install commands. The isolated environment draws a **boundary**, the dependency list writes an **intent**, the lock file records a single **resolution** of that intent, and the environment declaration writes the run conditions that remain outside the resolution. The pair most often confused is the list and the lock: the list is the document a person writes and updates by hand, while the lock is the document **derived** from the list and not written by hand. Narrowing the list does not take the place of the lock, because the indirect components never appear in the list, and steps 04 and 10 in the table depend on exactly them. #### The Same Notebook, Two Machines ```python # run.py — MODEL: the same notebook is run on two machines. The fictional readings dataset # is the one Data Collection and Preparation prepared: 1,260 subscribers, 3 periods, 3,199 of 3,780 cells filled. def rng(seed): s = seed & 0xFFFFFFFF def next_value(): nonlocal s s = (1664525 * s + 1013904223) & 0xFFFFFFFF return s / 4294967296 return next_value SUBSCRIBERS, PERIODS = 1260, 3 r = rng(20260218) RAW = [round(4 + r() * 56, 2) for _ in range(SUBSCRIBERS * PERIODS)] r2 = rng(9110723) p = [r2() for _ in range(SUBSCRIBERS * PERIODS)] EMPTY = set(sorted(range(len(p)), key=lambda j: p[j])[:581]) READINGS = [None if j in EMPTY else RAW[j] for j in range(SUBSCRIBERS * PERIODS)] mean = lambda v: sum(v) / len(v) # Item 5 — locale. The meter index arrives as text; which of the two separators is # the decimal and which the thousands separator is read from the machine's locale. METER_INDEX = [round(200 + (i * 7919) % 1400 + sum(x for x in READINGS[i*PERIODS:(i+1)*PERIODS] if x is not None), 2) for i in range(SUBSCRIBERS)] def format_value(v): whole, _, frac = f"{v:.2f}".partition(".") g = "" while len(whole) > 3: whole, g = whole[:-3], "." + whole[-3:] + g return whole + g + "," + frac parse = lambda m, thousands, decimal: float(m.replace(thousands, "").replace(decimal, ".")) TEXT = [format_value(v) for v in METER_INDEX] A = [parse(m, ".", ",") for m in TEXT] # thousands dot, decimal comma B = [parse(m, ",", ".") for m in TEXT] # thousands comma, decimal dot # Item 6 — chunk count. When the aggregation is divided into chunks, the mean of the # chunk means is taken; the chunks are unequal, because the missing cells are not evenly distributed. def chunked_mean(k): bounds = [i * SUBSCRIBERS // k for i in range(k + 1)] chunks = [[x for j in range(bounds[i], bounds[i+1]) for x in READINGS[j*PERIODS:(j+1)*PERIODS] if x is not None] for i in range(k)] return mean([mean(x) for x in chunks]) THRESHOLD = 0.0005 # m3; the reading resolution is 0.01 m3, a difference below one twentieth of it cannot be read # Item 8 — notebook cell order. There are two cells: filling an empty cell with the # subscriber's own mean and counting the cells that exceed the upper tariff tier (40 m3). def fill(v): o = [] for i in range(SUBSCRIBERS): h = [v[i*PERIODS+d] for d in range(PERIODS) if v[i*PERIODS+d] is not None] o.append(mean(h) if h else 0.0) return [o[j // PERIODS] if x is None else x for j, x in enumerate(v)] count = lambda v: sum(1 for x in v if x is not None and x > 40.0) row_fmt2 = row_fmt([22, 20, 20, 14]) print(row_fmt2(["environment item", "machine A", "machine B", "same"])) print(row_fmt2(["locale (total)", round(sum(A), 2), round(sum(B), 2), "no"])) print(row_fmt2(["chunk count (mean)", round(chunked_mean(1), 6), round(chunked_mean(8), 6), "yes" if abs(chunked_mean(1) - chunked_mean(8)) < THRESHOLD else "no"])) print(row_fmt2(["cell order (over 40)", count(READINGS), count(fill(READINGS)), "no"])) shrunk = sum(1 for a, b in zip(A, B) if b < a) diff = abs(chunked_mean(1) - chunked_mean(8)) print(f"\nlocale: {len(TEXT)} of {len(TEXT)} meter indexes changed; {shrunk} shrank because the " f"thousands separator was read as a decimal, {len(TEXT)-shrunk} grew") print(f"chunk count: difference {diff:.6f} m3, {diff/THRESHOLD:.0f} times the {THRESHOLD} m3 threshold") print(f"cell order: the same two cells, in reverse order {count(fill(READINGS))-count(READINGS)} more cells " f"were counted ({100*(count(fill(READINGS))-count(READINGS))/count(READINGS):.0f} percent)") ``` ``` environment item machine A machine B same locale (total) 1236664.22 41433984.33 no chunk count (mean) 31.895661 31.889608 no cell order (over 40) 1116 1264 no locale: 1260 of 1260 meter indexes changed; 613 shrank because the thousands separator was read as a decimal, 647 grew chunk count: difference 0.006053 m3, 12 times the 0.0005 m3 threshold cell order: the same two cells, in reverse order 148 more cells were counted (13 percent) ``` All three rows assume the same code, the same data, and the same lock file. What produces the difference is only the environment item. The first row is a difference at the **order-of-magnitude level**. The meter index arrives as text, and a choice is made between two conventions: which separator is the decimal and which is the thousands separator. All 1,260 meter indexes change, and the difference does not run in one direction either — the 613 indexes that carry a thousands separator shrink a thousandfold, and the 647 indexes that carry no separator grow a hundredfold. The total comes out as 41,433,984.33 m³ instead of 1,236,664.22 m³. An error of this kind is visible to the eye, but for it to be seen someone has to look at **both machines at once**. The second row is the opposite. When the aggregation is divided into eight chunks, the mean of the chunk means is taken; the chunks are not of equal size, because the missing cells are not evenly distributed among the subscribers. The result is 31.889608 m³ instead of 31.895661. Two floating-point numbers are not compared with `==`; a threshold is written instead. The threshold here is **0.0005 m³**, one twentieth of the meter resolution of 0.01 m³: a difference below this value cannot be read from the measuring device anyway, so it counts as the same. The difference is 0.006053 m³, 12 times the threshold, and since it stays above the threshold the two runs do not give the same number. The row count is 3,199 in both runs; a check that looks at row counts does not see this difference. The third row concerns the **form** of the notebook. There are two cells: filling an empty cell with the subscriber's own mean and counting the cells that exceed the upper tariff tier. If the fill runs first, 581 cells gain a value and some of them pass the 40 m³ threshold; the count comes out 1,264 instead of 1,116, 148 cells more, 13%. No cell was written wrongly; only their order is different. #### What the Notebook Records and What It Does Not This last row shows why the notebook form is an environment item. A notebook file stores its cells **in the order they were written**, not the order they ran; beside each cell sits only a run counter. If the counter reads 3, 1, 2, a notebook that looks correct when read from top to bottom actually ran in another order. For this reason a one-condition rule is written for a notebook to count as reproducible: **when the notebook is run from start to finish, in a single pass, from a clean state, it must give the same output.** The saved output is not evidence for this rule, because a saved output does not say in which order it was produced. It also becomes visible why the measure is set up in the first lesson of this course. In the lessons that follow, the body of the calculation leaves the hand-written loops and moves inside an abstraction; the decisions that abstraction makes about element type, shape and summation order add new rows to exactly the table above. That the eight-chunk aggregation moves the mean by 0.006053 m³ is not an error but a **default**; whoever decided how many chunks to divide into also decided that number. The environment declaration is for this reason not an extension of the version list but a part of the calculation. The same reasoning applies to the saved output. Because a notebook keeps both the code and the output in a single file, it is easy to read, but it does not show **where** the difference between two runs begins; what shows that is the run ledger, in which each step writes its own decisions and row accounting. The lock file fixes the environment, the run ledger fixes the decision, and running the notebook from the start fixes the order; unless all three are written, "the same number" remains a claim. #### Summary - A working environment is not a list of commands but a list of nine items that must be pinned; a different document pins each item. - With no document, 2 of 17 steps give the same result on two machines, with the dependency list 5, with the lock file 11. - None of the 6 steps that still move after the lock file has anything to do with versions; the locale, chunk count, seed, cell order and data digest are not written in the lock file. - The locale difference changes all 1,260 meter indexes and moves the total from 1,236,664.22 to 41,433,984.33; 613 indexes shrink and 647 indexes grow. - The chunk count moves the mean by 0.006053 m³ — 12 times the 0.0005 m³ threshold — but never changes the row count; the cell order raises the count of cells above 40 m³ from 1,116 to 1,264. #### Next Step Once the environment is pinned, the notebook gives the same number on two machines, but the calculation itself is still done with hand-written loops, as in the previous course. In the code blocks of this section 3,780 cells were walked one by one, each cell was held as a separate object, and for every summation job the data was read again from start to finish. The next lesson does the same job with an **array interface** and sets up the unit of measure of the course: how many lines of code, how many passes and how many bytes the same summation job takes. ### Arrays Source: https://academia.sh/en/courses/data-analysis-with-python/arrays The unit of measure of the course is set up: the same job (total per subscriber, overall mean, count of cells above 40 m³) is done with nested lists and with an array interface, and the line count falls from 22 to 2 and the allocated bytes from 188,864 to 30,320 (from 49.96 to 8.02 bytes per element), while the pass count falls from 3 to 3, that is, does not fall at all. The hidden default is that an array has no empty value: when the 581 empty cells are filled with zero the mean comes out 26.993180, with a sentinel value 26.839476, with filled cells only 31.895661 m³; the count above 40 m³ is 1,116 under all three decisions. The dtype choice keeps 3,780 elements between 7,560 and 30,240 bytes. In the previous lesson the environment was pinned, and the notebook now gives the same number on two machines. The calculation itself did not change, though: 3,780 cells were walked one by one with nested loops, each cell was held as a separate object, and for every question the data was read again from start to finish. This lesson does the same job with an **array interface**. The lesson does not merely show a convenience; it sets up the **unit of measure** of the course. From here on, two numbers will stand side by side in every lesson: what the abstraction gains and what the abstraction hides. The gain is measured here with three units — how many lines of code, how many full passes over the data, how many bytes. The hidden default is the decision the array makes for you without asking. No library is called; the array behavior is modeled with the standard library, because the aim is not to imitate a tool but to make its behavior measurable. - **NC3.** An array interface fixes three things together: **contiguous memory**, a **single dtype** and a **shape**. The gain is a result of this triple and shows up in bytes and in the number of lines at the call site; it does not show up in the number of passes over the data. - **NC4.** An array has no empty value. A missing cell has to turn into a number, and which number it turns into is a decision. This decision moves some summaries and does not move others at all. #### The Same Job, Two Structures ```python # arrays.py — MODEL: the three properties of an array interface are written with the standard library: # contiguous memory, a single dtype, shape. No library is called; the aim is not to imitate # a tool but to make its behavior measurable. The dataset is the fictional one prepared by the Data Collection and Preparation course. import inspect import sys from array import array def rng(seed): s = seed & 0xFFFFFFFF def next_value(): nonlocal s s = (1664525 * s + 1013904223) & 0xFFFFFFFF return s / 4294967296 return next_value SUBSCRIBERS, PERIODS = 1260, 3 r = rng(20260218) RAW = [round(4 + r() * 56, 2) for _ in range(SUBSCRIBERS * PERIODS)] r2 = rng(9110723) p = [r2() for _ in range(SUBSCRIBERS * PERIODS)] EMPTY = set(sorted(range(len(p)), key=lambda j: p[j])[:581]) READINGS = [None if j in EMPTY else RAW[j] for j in range(SUBSCRIBERS * PERIODS)] TABLE = [READINGS[i*PERIODS:(i+1)*PERIODS] for i in range(SUBSCRIBERS)] PASSES = 0 # one full scan over the data from start to finish def by_hand(t): global PASSES subscriber_totals = [] for s in t: a = 0.0 for x in s: if x is not None: a += x subscriber_totals.append(a) overall, n = 0.0, 0 for s in t: for x in s: if x is not None: overall += x n += 1 above = 0 for s in t: for x in s: if x is not None and x > 40.0: above += 1 PASSES += 3 return subscriber_totals, overall / n, above class Array: def __init__(self, data, shape, dtype="d"): self.d, self.shape, self.dtype = array(dtype, data), shape, dtype @property def size(self): return self.shape[0] * self.shape[1] def axis_sum(self, axis): global PASSES PASSES += 1 a, b = self.shape if axis == 1: return [sum(self.d[i*b:(i+1)*b]) for i in range(a)] return [sum(self.d[i*b+j] for i in range(a)) for j in range(b)] def total(self): global PASSES PASSES += 1 return sum(self.d) def above(self, threshold): global PASSES PASSES += 1 return sum(1 for x in self.d if x > threshold) def with_array(a): return a.axis_sum(1), a.total() / a.size, a.above(40.0) def list_bytes(t): b = sys.getsizeof(t) for s in t: b += sys.getsizeof(s) + sum(sys.getsizeof(x) for x in s if x is not None) return b FLAT = Array([0.0 if x is None else x for x in READINGS], (SUBSCRIBERS, PERIODS)) PASSES = 0 E = by_hand(TABLE) pass_hand = PASSES PASSES = 0 D = with_array(FLAT) pass_arr = PASSES lines = lambda f: len(inspect.getsource(f).strip().splitlines()) bytes_list, bytes_arr = list_bytes(TABLE), sys.getsizeof(FLAT.d) row_fmt = lambda w: lambda h: "".join(str(v).ljust(w[0]) if j == 0 else str(v).rjust(w[j]) for j, v in enumerate(h)) y = row_fmt([20, 20, 16, 10]) print(f"shape {FLAT.shape}, element count {FLAT.size}, dtype 64-bit float, " f"empty cells {len(EMPTY)}") print("job: total per subscriber, overall mean, count of cells above 40 m3\n") print(y(["measure", "hand-written loop", "array interface", "ratio"])) print(y(["line count", lines(by_hand), lines(with_array), f"{lines(by_hand)/lines(with_array):.1f}x"])) print(y(["pass count", pass_hand, pass_arr, f"{pass_hand/pass_arr:.1f}x"])) print(y(["bytes allocated", bytes_list, bytes_arr, f"{bytes_list/bytes_arr:.1f}x"])) print(y(["bytes per element", f"{bytes_list/FLAT.size:.2f}", f"{bytes_arr/FLAT.size:.2f}", f"{bytes_list/bytes_arr:.1f}x"])) print(f"\nsame job, two results: mean by hand {E[1]:.6f} m3, with the array interface {D[1]:.6f} m3; " f"cells above 40 m3 on both paths {E[2]}") ``` ``` shape (1260, 3), element count 3780, dtype 64-bit float, empty cells 581 job: total per subscriber, overall mean, count of cells above 40 m3 measure hand-written loop array interface ratio line count 22 2 11.0x pass count 3 3 1.0x bytes allocated 188864 30320 6.2x bytes per element 49.96 8.02 6.2x same job, two results: mean by hand 31.895661 m3, with the array interface 26.993180 m3; cells above 40 m3 on both paths 1116 ``` The definition of the three measures stays fixed throughout the course. The **line count** is the number of lines of code written at the call site, and it is read from the code, not counted by hand. The **pass count** is the number of full scans made over the data from start to finish; in the model each scan raises a counter by one, so it is a number that is measured, not claimed. The **allocated bytes** is the space taken up by the structure that holds the data and by the objects inside it. None of the three changes from run to run; time does change, which is why time is not measured. The gain of an abstraction is defined by which of these three numbers it lowers, and its cost by which it does not. The first row of the table is the expected gain. Written by hand, the same three questions take 22 lines; with the array interface they take 2. The difference is not just brevity of writing: in the hand-written version the missing-cell check is repeated three times, the loop bounds are written three times, and at each repetition there is a chance of leaving one of them out. The abstraction moves these three repetitions to a single place. The third row is the second gain, and its cause is that the decision about **shape** and **dtype** was made once. In nested lists each of the 3,780 numbers is an object of its own; the object itself takes about 24 bytes, the pointer that points to it 8 bytes, and each of the 1,260 small lists that hold the rows takes space as well. The total is 188,864 bytes, 49.96 per element. In the array the 3,780 numbers sit in one contiguous block of memory, exactly 8 bytes per element, and the shape is nothing more than two integers: 30,320 bytes, 8.02 per element. The ratio is 6.2 times. The second row, however, is not a gain. The pass count is **3** on both paths, because there are three questions and each question reads the data from start to finish once. The array interface lifts the loop out of the call site, not out of the data. This is the most often misread number of the course: because two lines of code are written, a single pass is assumed. Every chained call adds its own pass, and that is exactly what the next lesson measures. In the hand-written version, reducing these three passes to one means merging three loops into one, and writing that takes two lines; in the array interface, making the same merge requires leaving the call chain and going back to the loop. Brevity and pass count stand against each other here. The last row points to the hidden default. The same job, the same data, two different means: 31.895661 and 26.993180 m³. The 4.90 m³ between them is not a calculation error but a decision that comes from the structure of the array. #### Shape and Axis The `Array` class in the model carries two fields, and keeping the two apart is the most defining property of an array interface. The `d` field holds the 3,780 numbers in a single flat block; the `shape` field is nothing more than two integers and says how that flat block is to be read. The data is not two-dimensional — its **reading** is. The direct consequence of this separation is that changing the shape does not copy the data at all. Writing (3, 1260) in place of (1260, 3) moves none of the 30,240 bytes; it only changes the slice bounds in the `axis_sum` function. This cheapness is the most useful side of the abstraction and at the same time its quietest side: a wrong shape does not produce an error, it only produces another number. **Axis** is a direction within that shape, and the numbering in the model runs from left to right: axis 0 is the subscriber direction, axis 1 is the period direction. `axis_sum(1)` adds the three periods of each subscriber and returns 1,260 values; `axis_sum(0)` adds all the subscribers of each period and returns 3 values. Both carry the name "sum", both distribute the same 102,034.22 m³, but one describes subscriber behavior and the other period behavior. In the hand-written loop the direction of the sum is read from the loop lines; in the array interface it is read from a single integer, and that integer is often left as the default. In this lesson the axis is written out explicitly; the rule to be kept until the next lesson is this: **an aggregation with no written axis is an aggregation whose direction was not chosen.** #### An Array Has No Empty Value ```python # missing.py — MODEL: an array has no "empty" value. The 581 empty cells have to turn into a number # and which number they turn into is a decision. Three decisions, three means. DECISIONS = [ ("fill with zero", lambda v: [0.0 if x is None else x for x in v]), ("sentinel -1", lambda v: [-1.0 if x is None else x for x in v]), ("filled cells only", lambda v: [x for x in v if x is not None]), ] z = row_fmt([24, 12, 16, 12]) print(z(["581 empty cells", "elements", "mean (m3)", "over 40 m3"])) for name, f in DECISIONS: a = Array(f(READINGS), (len(f(READINGS)), 1)) print(z([name, a.size, f"{a.total()/a.size:.6f}", a.above(40.0)])) DTYPES = [("16-bit integer", "h"), ("32-bit integer", "i"), ("32-bit float", "f"), ("64-bit float", "d")] LITERS = [int(round((0.0 if x is None else x) * 1000)) for x in READINGS] z2 = row_fmt([18, 15, 15, 30]) print("\n" + z2(["dtype", "bytes/element", "3780 elements", "representation range"])) for name, t in DTYPES: b = array(t).itemsize span = (f"-{2**(8*b-1)} .. {2**(8*b-1)-1}" if t in ("h", "i") else ("7 significant digits" if t == "f" else "15 significant digits")) print(z2([name, b, b * FLAT.size, span])) TOTAL = FLAT.total() small = array("f", [TOTAL])[0] print(f"\nlargest reading {max(LITERS)} liters; a 16-bit integer cannot hold this value") print(f"dataset total {TOTAL:.2f} m3; stored as {small} in the 32-bit float, " f"difference {abs(small-TOTAL):.5f} m3 and above the 0.0005 m3 threshold of the Working Environment lesson") ``` ``` 581 empty cells elements mean (m3) over 40 m3 fill with zero 3780 26.993180 1116 sentinel -1 3780 26.839476 1116 filled cells only 3199 31.895661 1116 dtype bytes/element 3780 elements representation range 16-bit integer 2 7560 -32768 .. 32767 32-bit integer 4 15120 -2147483648 .. 2147483647 32-bit float 4 15120 7 significant digits 64-bit float 8 30240 15 significant digits largest reading 60000 liters; a 16-bit integer cannot hold this value dataset total 102034.22 m3; stored as 102034.21875 in the 32-bit float, difference 0.00125 m3 and above the 0.0005 m3 threshold of the Working Environment lesson ``` The three rows are three defensible decisions, and they move the mean between 26.839476 and 31.895661 m³. **Filling with zero** counts a missing reading as "no water was used", and because it raises the denominator without raising the numerator it lowers the mean by 4.90 m³. The **sentinel value** is chosen to mark the missing cell, but if excluding the marker is forgotten, 581 m³ is subtracted from the total and the mean falls even further. The **filled cells only** decision uses 3,199 as the denominator and gives the number the previous course measured. All three have a reason; the only thing without a reason is leaving the decision unwritten. The fourth column shows where this decision does not show. **The count of cells above 40 m³ is 1,116 under all three decisions**, because all three fill values lie below the threshold. An audit that looks only at this number cannot tell the three decisions apart. The number the decision moves and the number that makes the decision invisible sit in the same table, and which of the two is reported matters as much as the decision itself. Notice that the hand-written version never had to make this decision. In nested lists the empty cell stays as `None`; this value is not a number, it does not enter the sum, and the `is not None` check leaves it out explicitly inside the loop. The decision is there but it is **visible**, because it was written one by one in every loop. In the array the decision is never written; a fill value is chosen on the line that builds the array, and that value, never mentioned again, mixes into all the summaries. What the abstraction hides is not the decision itself but **where the decision is made**. The lower table measures the dtype choice. The same 3,780 elements take between 7,560 and 30,240 bytes depending on the chosen dtype; the fourfold difference is paid not in memory but in representation range. The largest reading is 60,000 liters and exceeds the 32,767 limit of the 16-bit integer — choosing this dtype is not a fourfold memory gain but a silent overflow. The 32-bit float does not overflow but holds roughly seven significant digits; because the dataset total of 102,034.22 m³ needs eight significant digits, it is stored as 102,034.21875. The difference is 0.00125 m³ and above the 0.0005 m³ threshold of the previous lesson, which means this dtype choice alone is enough to make two runs differ. #### Summary - An array interface fixes contiguous memory, a single dtype and shape together; here the shape is (1260, 3) and the element count is 3,780. - The same three questions take 22 lines by hand and 2 lines with the array interface; the allocated bytes fall from 188,864 to 30,320, from 49.96 to 8.02 bytes per element. - The pass count is 3 on both paths: the abstraction lifts the loop out of the call site, not out of the data. - Because an array has no empty value, 581 empty cells turn into a number; the mean comes out 26.993180 with zero fill, 26.839476 with a sentinel value, and 31.895661 m³ with filled cells only. - The same three decisions do not change the count of cells above 40 m³ at all (1,116); the dtype choice keeps the 3,780 elements between 7,560 and 30,240 bytes, and a 16-bit integer cannot hold the 60,000-liter reading. #### Next Step The unit of measure is set up, and one of the three numbers came out contrary to expectation: the pass count did not fall. The next lesson takes up exactly this number. The same derivation job can be done in a single loop with one pass, or it can be written as a chain of vectorized calls; the second way adds one pass and one intermediate array with every call. The same lesson also counts the two decisions vectorization gives silently: the promotion of the dtype and overflow in a narrow dtype. In the last table of this lesson it was seen that a 16-bit integer cannot hold the 60,000-liter reading; the next lesson measures what happens when this limit is exceeded. ### Vectorized Operations Source: https://academia.sh/en/courses/data-analysis-with-python/vectorized-operations The same derivation job is done with a single loop and with a chain of vectorized calls: the line count falls from 10 to 2 while the pass count rises from 1 to 3 and the allocated bytes from 30,320 to 90,960; every link in the chain adds one pass and one intermediate array. Three decisions the call makes silently are counted: a single float constant carries the chain to 64 bits and raises the allocated bytes from 30,400 to 60,720, a 16-bit dtype shifts 1,538 of the 3,780 elements so that the total of 102,034.22 m³ shows as 1,239.85 m³, and of the 581 empty cells marked with a sentinel value 0 stay distinguishable after the first operation, while the count of cells whose excess looks zero rises to 1,783. The previous lesson showed that one of the three measures came out contrary to expectation: the line count fell eleven to one and the allocated bytes fell six to one, but the pass count did not fall at all. This lesson takes up that number, and it starts with worse news — in a vectorized chain the pass count does not fall, it **rises**. The job itself comes from the derived variable step of the previous course: the share of each cell that remains above the second tariff tier is to be computed. 25 m³ is subtracted from the reading, negative results are clipped at zero, and the remainder is converted to liters. Three operations, one result. It is written in two ways: fused in a single loop, and as a chain of three vectorized calls. - **NC5.** A vectorized call saves you from **writing** the loop, not from the **pass**. Every link in the chain scans the data once and holds its result in a new array; the gain is in the line, the cost is in the pass and in the allocated bytes. - **NC6.** A vectorized operation chooses the dtype itself. This choice moves the number in three ways: by promoting the type it enlarges the bytes, by overflowing in a narrow type it shrinks the result, and by turning a sentinel value into an ordinary number it makes the missing cell indistinguishable. #### One Job, Two Paths ```python # vectorized.py — MODEL: a vectorized operation is written as a call that does not walk the # elements one by one. Each call scans the data once (one pass) and holds its result in a new # array (one intermediate array). The dataset is the fictional one prepared by the Data Collection and Preparation course. import inspect import sys from array import array def rng(seed): s = seed & 0xFFFFFFFF def next_value(): nonlocal s s = (1664525 * s + 1013904223) & 0xFFFFFFFF return s / 4294967296 return next_value SUBSCRIBERS, PERIODS = 1260, 3 r = rng(20260218) RAW = [round(4 + r() * 56, 2) for _ in range(SUBSCRIBERS * PERIODS)] r2 = rng(9110723) p = [r2() for _ in range(SUBSCRIBERS * PERIODS)] EMPTY = set(sorted(range(len(p)), key=lambda j: p[j])[:581]) READINGS = [0.0 if j in EMPTY else RAW[j] for j in range(SUBSCRIBERS * PERIODS)] PASSES = 0 # one full scan over the data from start to finish ALLOC = 0 # bytes allocated by the result and the intermediate arrays class Array: def __init__(self, data, dtype="d"): self.d, self.dtype = array(dtype, data), dtype def apply_op(a, f, dtype=None): global PASSES, ALLOC PASSES += 1 c = Array([f(x) for x in a.d], dtype or a.dtype) ALLOC += sys.getsizeof(c.d) return c UPPER = 25.0 # upper bound of the second tariff tier, m3 LITER = 1000 # m3 -> liter def by_hand(data): global PASSES, ALLOC output = [] for x in data: f = x - UPPER output.append((f if f > 0.0 else 0.0) * LITER) PASSES += 1 c = Array(output) ALLOC += sys.getsizeof(c.d) return c subtract = lambda a, s: apply_op(a, lambda x: x - s) clip = lambda a, lower: apply_op(a, lambda x: x if x > lower else lower) multiply = lambda a, s: apply_op(a, lambda x: x * s) def chain(a): return multiply(clip(subtract(a, UPPER), 0.0), LITER) A = Array(READINGS) PASSES, ALLOC = 0, 0 E = by_hand(A.d); pass_hand, alloc_hand = PASSES, ALLOC PASSES, ALLOC = 0, 0 V = chain(A); pass_chain, alloc_chain = PASSES, ALLOC lines = lambda f: len(inspect.getsource(f).strip().splitlines()) row_fmt = lambda w: lambda h: "".join(str(v).ljust(w[0]) if j == 0 else str(v).rjust(w[j]) for j, v in enumerate(h)) y = row_fmt([22, 18, 20, 10]) print(f"job: (reading - {UPPER} m3) is clipped at zero and converted to liters; {len(READINGS)} elements\n") print(y(["measure", "single hand loop", "vectorized", "ratio"])) print(y(["line count", lines(by_hand), lines(chain), f"{lines(by_hand)/lines(chain):.1f}x"])) print(y(["pass count", pass_hand, pass_chain, f"{pass_chain/pass_hand:.1f}x"])) print(y(["bytes allocated", alloc_hand, alloc_chain, f"{alloc_chain/alloc_hand:.1f}x"])) same = sum(1 for x, z in zip(E.d, V.d) if abs(x - z) < 1e-9) print(f"\nsame result: {same}/{len(READINGS)} elements are the same below the 1e-9 liter threshold " f"(floating-point comparison is not done with ==; the threshold is below the liter resolution)") print(f"excess total {sum(V.d)/LITER:.2f} m3") ``` ``` job: (reading - 25.0 m3) is clipped at zero and converted to liters; 3780 elements measure single hand loop vectorized ratio line count 10 2 5.0x pass count 1 3 3.0x bytes allocated 30320 90960 3.0x same result: 3780/3780 elements are the same below the 1e-9 liter threshold (floating-point comparison is not done with ==; the threshold is below the liter resolution) excess total 34707.55 m3 ``` Both paths produce the same 3,780 numbers. The comparison was made with a threshold far below the liter resolution, 1e-9 liters; two floating-point values are not compared with `==`, because two calculations that reach the same mathematical result in a different order can part ways in the last digit. The threshold here does not hide a meaningful difference: 1e-9 liters is a billionth of the smallest value a meter can read. The gain is in the line, and it is real. The hand-written version takes 10 lines, the chain 2. The difference is not just length: in the hand-written version the loop bound, the accumulation list and the clipping condition are nested in the same place, and changing one of the three operations requires touching the body of the loop. In the chain the three links are independent of each other, and `subtract`, `clip` and `multiply` are written once and used everywhere. The cost is in the next two rows, and it is mostly left out of the account. Because the chain consists of three links, the data is scanned **three times**; the hand-written loop scans once. Because each link holds its result in a new array, three arrays of 3,780 elements are allocated: 90,960 bytes. Of this, 60,640 bytes are **intermediate arrays**, that is, two temporary results thrown away after use. The hand-written version allocates only the output array: 30,320 bytes. On small datasets this difference does not show; when the scale grows, the length of the chain is written directly onto the memory peak, because each of the intermediate arrays takes as much space as the input. From here comes a reading that ties into the standing rule of the course: the number of links in the chain is a decision made by whoever writes the call, and usually without noticing. A chain of three links means three passes, a chain of five links means five passes, and the line of code stays the same in both. **The length of the chain is visible in the code; its cost is not.** #### Fusing the Chain ```python # fused.py — MODEL: the third path. The three links of the chain are fused into a single # vectorized call; the loop is still not written but the data is scanned once. def fused(a): return apply_op(a, lambda x: (x - UPPER if x > UPPER else 0.0) * LITER) PASSES, ALLOC = 0, 0 K = fused(A); pass_fused, alloc_fused = PASSES, ALLOC w = row_fmt([20, 10, 12, 12, 18]) print(w(["measure", "by hand", "chain", "fused", "lowest"])) print(w(["line count", lines(by_hand), lines(chain), lines(fused), "chain / fused"])) print(w(["pass count", pass_hand, pass_chain, pass_fused, "by hand / fused"])) print(w(["bytes allocated", alloc_hand, alloc_chain, alloc_fused, "by hand / fused"])) print(w(["reusable parts", 0, 3, 0, "chain"])) deviating = sum(1 for x, z in zip(V.d, K.d) if abs(x - z) > 1e-9) print(f"\nall three paths give the same result: {deviating} elements deviate above the 1e-9 liter threshold") print(f"the fused path is at the same line count as the chain and at the same pass count and bytes as the by-hand version; its cost is " f"that three decisions go into a single expression and the number of reusable parts drops from 3 to 0") ``` ``` measure by hand chain fused lowest line count 10 2 2 chain / fused pass count 1 3 1 by hand / fused bytes allocated 30320 90960 30320 by hand / fused reusable parts 0 3 0 chain all three paths give the same result: 0 elements deviate above the 1e-9 liter threshold the fused path is at the same line count as the chain and at the same pass count and bytes as the by-hand version; its cost is that three decisions go into a single expression and the number of reusable parts drops from 3 to 0 ``` The third path shows that the cost of the chain is not inevitable. When the work of the three links is fused into a single vectorized call, the line count stays the same as in the chain, and the pass count drops to 1 as in the hand-written loop and the allocated bytes to 30,320. This is the lowest of all three measures. Where the cost lies stands in the last row. In the chain, `subtract`, `clip` and `multiply` are three independent parts, and they are reused in another calculation in another order. In the fused expression the same three decisions enter a single body and cannot be taken out from the outside; the number of reusable parts falls from 3 to 0. If the tariff threshold moves by one step, in the chain only a parameter of a call changes; in the fused expression the body is rewritten. This is the form the measure takes in the course: the gain of an abstraction does not show in all three numbers at once, and "writing vectorized" is not a single thing. **How many links are written is a decision**, and that decision determines the pass count and the memory peak together. #### Three Decisions the Call Makes ```python # hidden.py — MODEL: three decisions a vectorized call makes in your place. LITER_VALUES = [int(round(x * LITER)) for x in READINGS] # 1. Type promotion. The first link of the chain is an integer operation, the second multiplies by a float constant. INTS = Array(LITER_VALUES, "i") PASSES, ALLOC = 0, 0 remaining = apply_op(INTS, lambda x: x - 25000, "i") int_bytes = ALLOC + sys.getsizeof(INTS.d) PASSES, ALLOC = 0, 0 charge = apply_op(apply_op(INTS, lambda x: x - 25000, "i"), lambda x: x * 0.0042, "d") float_bytes = ALLOC + sys.getsizeof(INTS.d) # 2. Overflow. If the dtype is chosen as a 16-bit integer to save memory, the liter values # do not fit the limit. The wrapping is written out explicitly; it is a model of the dtype's behavior. LIMIT = 2 ** 15 wrap = lambda v: ((v + LIMIT) % (2 * LIMIT)) - LIMIT overflowing = [v for v in LITER_VALUES if v > LIMIT - 1] wrapped = [wrap(v) for v in LITER_VALUES] # 3. Loss of the sentinel value. The 581 empty cells had been marked with -1. FLAGGED = Array([-1.0 if j in EMPTY else RAW[j] for j in range(SUBSCRIBERS * PERIODS)]) final = multiply(clip(subtract(FLAGGED, UPPER), 0.0), LITER) distinct = sum(1 for j in EMPTY if abs(final.d[j]) > 1e-9) zeros = sum(1 for x in final.d if abs(x) < 1e-9) z = row_fmt([26, 22, 30]) print(z(["hidden default", "written assumption", "measured result"])) print(z(["type promotion", "stays integer", f"{int_bytes} -> {float_bytes} bytes"])) print(z(["16-bit overflow", "no overflow", f"{len(overflowing)}/{len(LITER_VALUES)} elements shifted"])) print(z(["sentinel cell", "581 cells marked", f"{distinct}/581 distinguishable"])) print(f"\ntype promotion: a single float constant carries the rest of the chain to 64 bits, " f"allocated bytes grow by {float_bytes - int_bytes}; charge total {sum(charge.d):.2f}") print(f"overflow: {len(overflowing)} elements exceed the 32,767-liter limit; the total " f"comes out {sum(wrapped)/LITER:.2f} m3 instead of {sum(LITER_VALUES)/LITER:.2f} m3") print(f"sentinel: after the chain, the cells whose excess looks zero number {zeros}; of these 581 are " f"not real zeros but marked empty cells") ``` ``` hidden default written assumption measured result type promotion stays integer 30400 -> 60720 bytes 16-bit overflow no overflow 1538/3780 elements shifted sentinel cell 581 cells marked 0/581 distinguishable type promotion: a single float constant carries the rest of the chain to 64 bits, allocated bytes grow by 30320; charge total 31643.72 overflow: 1538 elements exceed the 32,767-liter limit; the total comes out 1239.85 m3 instead of 102034.22 m3 sentinel: after the chain, the cells whose excess looks zero number 1783; of these 581 are not real zeros but marked empty cells ``` **Type promotion** is the quietest item of the chain. The array was built as a 32-bit integer and the first link stayed integer; when the second link multiplies by a fractional unit price, the result cannot be represented in an integer and the whole array is carried to the 64-bit float. The allocated bytes rise from 30,400 to 60,720. In the written code there is no sign at all of the growth; what decides the growth is that the constant `0.0042` is fractional. In long chains this means that every link after the first fractional link takes twice the space. **Overflow** goes the opposite way. Narrowing the dtype saves memory: a 16-bit integer holds the 3,780 elements in 7,560 bytes. But 1,538 of the readings in liters exceed the 32,767 limit, and these values shift to the negative side. The total of the dataset comes out 1,239.85 m³ instead of 102,034.22 m³ — a loss of 98.8%. The result arrives not with an error message but as a small number that looks reasonable; a check that only asks "is the total positive" lets it through. The wrapping behavior was written out explicitly here, because what a dtype does when its limit is exceeded is the type's own decision, and the model makes it visible. The **sentinel value** is the third decision, and it continues from where the previous lesson left off. Marking empty cells with `-1` is the known way of holding a missing value in an array, but the marker lives only **until the first operation**: subtracting 25 from `-1` gives -26, the clip pulls it to zero, and the multiplication leaves zero. At the end of the chain **none** of the 581 cells can be told apart. Of the 1,783 cells whose excess looks zero, 581 are in fact cells that were never read, and no trace is left to show the difference between the two. The clip operation here reduces two different meanings to a single value: "stayed below the threshold" and "was not measured". What the three items have in common is this: none of them produces an error. Type promotion gives the correct result and only enlarges the memory, overflow gives a number that looks reasonable, and the clipping of the sentinel gives a valid excess value. These are exactly the decisions an abstraction hides — the decisions made in cases where the call runs and produces a result. For this reason the rule of the course looks not at whether a call gives the correct result but at **whether its default was written down**. #### Summary - The vectorized chain lowers the line count from 10 to 2, raises the pass count from 1 to 3 and the allocated bytes from 30,320 to 90,960; 60,640 of these bytes are intermediate arrays. - The number of links in the chain is written one to one onto the pass count; the line of code stays constant, and the cost is not visible in the code. - A fractional constant carries the rest of the chain to 64 bits and raises the allocated bytes from 30,400 to 60,720. - A 16-bit dtype shifts 1,538 of the 3,780 elements and shows the total of 102,034.22 m³ as 1,239.85 m³; the result is not an error but a number that looks reasonable. - Of the 581 empty cells marked with a sentinel value, 0 stay distinguishable after the first operation; the count of cells whose excess looks zero rises to 1,783, and "below the threshold" and "was not measured" fall to a single value. - When three links are fused into a single call, the pass count falls to 1, the bytes to 30,320 and the line count does not change; the cost is that the number of reusable parts falls from 3 to 0. #### Next Step In this lesson all the operations were done on a single shape: between an array of 3,780 elements and one number. In real calculations two arrays meet, and their shapes are most often not the same — a base per period, a household count per subscriber, a unit price per tariff tier. The next lesson measures how different shapes are counted as compatible and why two incompatible shapes, instead of giving an error, produce an unexpected array. ### Broadcasting Source: https://academia.sh/en/courses/data-analysis-with-python/broadcasting The broadcasting rule is modeled with the standard library and run on eight shape pairs: 4 pairs give the correct result, 2 pairs give an error, 2 pairs silently produce an unexpected array. The cost of the silent case is counted: multiplying (1260, 1) by (1, 1260) gives 1,587,600 elements instead of 1,260, the allocated bytes rise from 10,160 to 12,700,880, only 1,260 of the elements are correct, and the reported per-person mean becomes 36.497564 instead of 36.938180 m³. In the second silent case the period base produces 9 values instead of 3, and the adjusted base total comes out 251.846368 m³ instead of 83.901904 m³. In the previous lesson all the operations were done between an array and a number, and the shape never changed. In real calculations two arrays meet, and their shapes are most often not the same: next to the reading table of 3,780 cells stand a base per period (3 values), a household count per subscriber (1,260 values) and a unit price per tariff tier (3 values). Instead of matching these one by one with loops, the array interface applies the **broadcasting** rule: the smaller shape is counted as repeated to fit the larger shape. The rule is told in two lines, and it is dangerous precisely because it is only two lines. Shapes are **aligned from the right**; in every aligned dimension either the two numbers are equal or one of them is 1. If this condition holds, the operation is carried out. The condition holding does not mean the operation is **correct**, only that it is **possible**. What this lesson measures is the difference between the two. - **NC7.** The broadcasting rule checks the compatibility of shapes, not the meaning of the operation. When the rule holds, the calculation is done and no warning is produced. - **NC8.** An incompatible shape pair gives one of two results: an error or an array of unexpected size. The first stops the calculation, the second continues it and can enlarge the element count by thousands of times. #### Writing the Rule ```python # broadcasting.py — MODEL: the broadcasting rule is written with the standard library. The rule: # shapes are aligned from the right; in every dimension either the two numbers are equal or one is 1. # If the rule holds, the operation is carried out. No library is called. import math def broadcast_shape(shape1, shape2): n = max(len(shape1), len(shape2)) s1 = (1,) * (n - len(shape1)) + tuple(shape1) s2 = (1,) * (n - len(shape2)) + tuple(shape2) c = [] for x, y2 in zip(s1, s2): if x == y2 or x == 1 or y2 == 1: c.append(max(x, y2)) else: return None return tuple(c) count = lambda b: 0 if b is None else math.prod(b) PAIRS = [ ("reading x single coefficient", (1260, 3), (1,), 3780), ("reading x period base", (1260, 3), (3,), 3780), ("reading x subscriber household count", (1260, 3), (1260, 1), 3780), ("reading x subscriber (flat form)", (1260, 3), (1260,), 3780), ("reading x period (two dimensions)", (1260, 3), (1, 3), 3780), ("subscriber total x household row", (1260, 1), (1, 1260), 1260), ("period base x period adjustment", (3,), (3, 1), 3), ("reading x two tiers", (1260, 3), (2,), 3780), ] row_fmt = lambda w: lambda h: "".join(str(v).ljust(w[0]) if j == 0 else str(v).rjust(w[j]) for j, v in enumerate(h)) y = row_fmt([38, 12, 11, 13, 11, 11, 9]) print(y(["operation", "shape 1", "shape 2", "result shape", "elements", "expected", "outcome"])) errors = silent = correct = 0 for name, shape1, shape2, expected in PAIRS: c = broadcast_shape(shape1, shape2) if c is None: status, errors = "error", errors + 1 elif count(c) != expected: status, silent = "silent", silent + 1 else: status, correct = "correct", correct + 1 print(y([name, str(shape1), str(shape2), str(c) if c else "-", count(c), expected, status])) print(f"\n{len(PAIRS)} pairs: {correct} correct, {errors} errors, {silent} silently unexpected shape") ``` ``` operation shape 1 shape 2 result shape elements expected outcome reading x single coefficient (1260, 3) (1,) (1260, 3) 3780 3780 correct reading x period base (1260, 3) (3,) (1260, 3) 3780 3780 correct reading x subscriber household count (1260, 3) (1260, 1) (1260, 3) 3780 3780 correct reading x subscriber (flat form) (1260, 3) (1260,) - 0 3780 error reading x period (two dimensions) (1260, 3) (1, 3) (1260, 3) 3780 3780 correct subscriber total x household row (1260, 1) (1, 1260) (1260, 1260) 1587600 1260 silent period base x period adjustment (3,) (3, 1) (3, 3) 9 3 silent reading x two tiers (1260, 3) (2,) - 0 3780 error 8 pairs: 4 correct, 2 errors, 2 silently unexpected shape ``` The first three rows are where the rule works, and the gain is real. Subtracting a base per period takes two nested loops and an index calculation when written by hand; with broadcasting it is a single operation, and the base of 3 values behaves as if it were repeated 1,260 times — without the repeated copy being built in memory. The household count per subscriber is spread in the right direction in the same way, as long as the shape is written (1260, 1). The fourth row is where the rule **protects**. If the household count per subscriber is written as a flat array, that is, in the shape (1260,), the right alignment compares the last dimensions: 3 and 1,260. They are not equal to each other and neither is 1, so the rule does not hold and the operation is not carried out. This is not a flaw but the one real guarantee of the rule: **an attempt to align on the wrong axis is, more often than not, stopped.** The eighth row is of the same kind; a coefficient array of two tiers cannot be broadcast onto a table of three periods. The sixth and seventh rows, however, are not stopped. The rule holds, the operation is carried out, and the result has a shape other than the expected one. Two of the eight pairs are in this situation, and what the two have in common is this: the person writing has written **one** of the shapes transposed. The next section counts the cost of this. #### The Silently Produced Result ```python # silent.py — MODEL: the cost of the silently produced result. The total consumption per # subscriber is divided by the household count per subscriber. The household count is fictional. import sys from array import array def rng(seed): s = seed & 0xFFFFFFFF def next_value(): nonlocal s s = (1664525 * s + 1013904223) & 0xFFFFFFFF return s / 4294967296 return next_value SUBSCRIBERS, PERIODS = 1260, 3 r = rng(20260218) RAW = [round(4 + r() * 56, 2) for _ in range(SUBSCRIBERS * PERIODS)] r2 = rng(9110723) p = [r2() for _ in range(SUBSCRIBERS * PERIODS)] EMPTY = set(sorted(range(len(p)), key=lambda j: p[j])[:581]) READINGS = [0.0 if j in EMPTY else RAW[j] for j in range(SUBSCRIBERS * PERIODS)] r3 = rng(31081977) HOUSEHOLD = [1 + int(r3() * 5) for _ in range(SUBSCRIBERS)] TOTALS = [sum(READINGS[i*PERIODS:(i+1)*PERIODS]) for i in range(SUBSCRIBERS)] # Right form: (1260, 1) with (1260, 1) RIGHT = array("d", [TOTALS[i] / HOUSEHOLD[i] for i in range(SUBSCRIBERS)]) # Silent form: (1260, 1) with (1, 1260) -> (1260, 1260) WRONG = array("d", [TOTALS[i] / HOUSEHOLD[j] for i in range(SUBSCRIBERS) for j in range(SUBSCRIBERS)]) diagonal = [WRONG[i * SUBSCRIBERS + i] for i in range(SUBSCRIBERS)] correct_count = sum(1 for i in range(SUBSCRIBERS) if abs(diagonal[i] - RIGHT[i]) < 1e-9) z = row_fmt([26, 18, 20, 12]) print(z(["measure", "right form", "silent form", "ratio"])) print(z(["result shape", "(1260, 1)", "(1260, 1260)", "-"])) print(z(["element count", len(RIGHT), len(WRONG), f"{len(WRONG)//len(RIGHT)}x"])) print(z(["bytes allocated", sys.getsizeof(RIGHT), sys.getsizeof(WRONG), f"{sys.getsizeof(WRONG)//sys.getsizeof(RIGHT)}x"])) print(z(["per-person mean", f"{sum(RIGHT)/len(RIGHT):.6f}", f"{sum(WRONG)/len(WRONG):.6f}", "-"])) print(f"\nof the {len(WRONG)} elements of the silent array, {correct_count} are correct (diagonal), " f"{len(WRONG)-correct_count} elements divide one subscriber's consumption by another subscriber's household count") print(f"reported per-person mean {sum(WRONG)/len(WRONG):.6f} instead of " f"{sum(RIGHT)/len(RIGHT):.6f} m3, difference {abs(sum(WRONG)/len(WRONG)-sum(RIGHT)/len(RIGHT)):.6f} m3") # Second silent case: period base (3,) with seasonal adjustment (3, 1). PERIOD_MEAN = [sum(READINGS[i*PERIODS+d] for i in range(SUBSCRIBERS)) / SUBSCRIBERS for d in range(PERIODS)] ADJUSTMENT = [0.98, 1.03, 1.10] # fictional seasonal adjustment adjusted = [PERIOD_MEAN[d] * ADJUSTMENT[d] for d in range(PERIODS)] adjusted_wrong = [PERIOD_MEAN[d] * ADJUSTMENT[e] for e in range(PERIODS) for d in range(PERIODS)] print(f"\nsecond silent case: in the product of period base (3,) and adjustment (3, 1) the result is " f"{len(adjusted_wrong)} values instead of {len(adjusted)}") print(f"adjusted base total {sum(adjusted_wrong):.6f} m3 instead of {sum(adjusted):.6f} m3; " f"because the shape grew, the total rose to {sum(adjusted_wrong)/sum(adjusted):.4f} times") ``` ``` measure right form silent form ratio result shape (1260, 1) (1260, 1260) - element count 1260 1587600 1260x bytes allocated 10160 12700880 1250x per-person mean 36.938180 36.497564 - of the 1587600 elements of the silent array, 1260 are correct (diagonal), 1586340 elements divide one subscriber's consumption by another subscriber's household count reported per-person mean 36.497564 instead of 36.938180 m3, difference 0.440616 m3 second silent case: in the product of period base (3,) and adjustment (3, 1) the result is 9 values instead of 3 adjusted base total 251.846368 m3 instead of 83.901904 m3; because the shape grew, the total rose to 3.0017 times ``` The calculation is this: each subscriber's three-period total consumption is to be divided by that subscriber's household count. The correct result is 1,260 values. When the household count array is written as a row, that is, in the shape (1, 1260) instead of (1260, 1), the rule still holds — in the first dimension 1,260 meets 1, in the second dimension 1 meets 1,260, and in both of them one of the sides is 1. The result is an array of **1,587,600 elements**, and in it every subscriber's consumption stands divided by every subscriber's household count. The cost has three separate measures. The element count grows 1,260 times, the allocated bytes from 10,160 to 12,700,880, that is, 1,250 times; as long as it fits in memory this growth produces no warning, and precisely because it fits it goes unnoticed. The accuracy measure is sharper: of the 1,587,600 elements only **1,260** are correct, and these are the ones on the diagonal. The remaining **1,586,340 elements** divide one subscriber's consumption by another subscriber's household count, that is, they have no real counterpart. The reported number, however, reflects neither of these two measures. The per-person mean consumption comes out 36.938180 m³ in the right array and 36.497564 m³ in the silent array; the difference between them is 0.440616 m³, about 1.2%. This is the most dangerous side of broadcasting: **the wrong result looks plausible.** When a finding moves between 36.5 and 36.9, nobody thinks of checking the shape. The number itself does not give the error away; the only thing that gives the error away is **the element count**, and that number usually does not enter the report. The second silent case is smaller and for exactly that reason more insidious. The average base of three periods is to be multiplied by three seasonal adjustments; when the adjustment is written as a column, the result is 9 values instead of 3. The adjusted base total comes out 251.846368 m³ instead of 83.901904 m³, that is, 3.0017 times. The multiplier here is not exactly 3, and this detail matters: had an exact multiple been seen, one could have said "I must have multiplied by three somewhere" and gone looking; 3.0017 looks like a number that comes from the calculation itself. The working rule that comes out of these two cases is one sentence: **in every operation that uses broadcasting, the shape of the result is written out and compared with the expected element count.** It cannot be assumed that the result is correct because the rule holds; the rule holding says only that the calculation could be done. In two of the eight pairs above the calculation was not done, and that was the good news; in two it was done, and that was the bad news. #### What the Rule Gains ```python # gain.py — MODEL: the gain of broadcasting. A base per period is to be subtracted. # The by-hand way copies the base to 3,780 elements; broadcasting does not copy, it computes the index. BASE = array("d", [9.80, 10.40, 11.20]) # fictional period base, m3 def by_copying(v, t): wide = array("d", [t[j % PERIODS] for j in range(len(v))]) return array("d", [v[j] - wide[j] for j in range(len(v))]), sys.getsizeof(wide) def by_broadcasting(v, t): return array("d", [v[j] - t[j % PERIODS] for j in range(len(v))]), 0 K1, extra1 = by_copying(READINGS, BASE) K2, extra2 = by_broadcasting(READINGS, BASE) deviating = sum(1 for a, b in zip(K1, K2) if abs(a - b) > 1e-9) w = row_fmt([26, 16, 16, 12]) print(w(["measure", "by copying", "by broadcasting", "ratio"])) print(w(["base array bytes", sys.getsizeof(BASE), sys.getsizeof(BASE), "1x"])) print(w(["extra bytes allocated", extra1, extra2, "-"])) print(w(["total bytes allocated", extra1 + sys.getsizeof(K1), extra2 + sys.getsizeof(K2), f"{(extra1+sys.getsizeof(K1))/(extra2+sys.getsizeof(K2)):.1f}x"])) print(f"\nboth paths give the same result: {deviating} elements deviate above the 1e-9 m3 threshold") print(f"broadcasting applies the 3-value base to {len(READINGS)} elements without copying; " f"in the copied version the base takes {extra1} bytes instead of {sys.getsizeof(BASE)}") ``` ``` measure by copying by broadcasting ratio base array bytes 104 104 1x extra bytes allocated 30320 0 - total bytes allocated 60640 30320 2.0x both paths give the same result: 0 elements deviate above the 1e-9 m3 threshold broadcasting applies the 3-value base to 3780 elements without copying; in the copied version the base takes 30320 bytes instead of 104 ``` The gain of the rule stands in this table, and it has nothing to do with the computed result — both paths give the same 3,780 values, and there is not a single element deviating above the 1e-9 m³ threshold. The difference is in **copying**. The most direct way of applying the period base by hand is to expand the 3 values to 3,780 elements and subtract the two equal shapes element by element; this expansion takes 30,320 bytes, and the total allocated bytes rise to 60,640. Broadcasting never builds the expanded array; it **computes** the index of the small shape from the large shape, and the base stays at 104 bytes. The total allocated bytes fall by half, to 30,320. The ratio here is 2 times because the expanded side is the same size as the result array. The smaller the ratio of the small array to the large one, the greater the gain: if the same operation were done with 3 tariff tiers instead of 3 periods and a table of 100,000 rows, the copied version would hold two full tables and broadcasting one. **What broadcasting gains is not the calculation but the array that is not built**, and it is worth noting that the cost in the previous section comes from the same place: because the array that is not built is cheap, the array of unexpected shape is also built cheaply and grows without drawing attention. #### Summary - The broadcasting rule aligns shapes from the right and looks in every dimension for equality or 1; the rule checks shape compatibility, not the meaning of the operation. - Of the eight shape pairs, 4 give the correct result, 2 give an error, 2 silently produce an array of unexpected shape. - When the household count is written as a row, the result is 1,587,600 elements instead of 1,260; the allocated bytes grow from 10,160 to 12,700,880, that is, 1,250 times. - Of the 1,587,600 elements of the silent array only 1,260 are correct; the remaining 1,586,340 elements have no real counterpart, and the reported per-person mean comes out 36.497564 instead of 36.938180 m³. - The period base adjustment produces 9 values instead of 3 and carries the adjusted base total from 83.901904 m³ to 251.846368 m³, to 3.0017 times. - What the rule gains is the array that is not built: applying the period base by copying asks for 30,320 bytes of extra space, broadcasting asks for 0 bytes, and the total allocated bytes fall from 60,640 to 30,320. #### Next Step Broadcasting matches two arrays of the same shape element by element, and this is the simplest multi-array operation there is. The tariff calculation itself, however, is not an element-by-element match: when the unit prices of three tiers and the consumption of three periods are multiplied and summed, every result value is the joint calculation of **a whole row and a whole column**. The next lesson writes this operation — matrix multiplication and the solution problem derived from it — with a hand loop and with the abstraction, compares the pass counts and measures where numerical precision comes into play. Shape compatibility is decisive there too, but the rule changes: in broadcasting compatibility is sought by right alignment, while in multiplication a single equality is sought, and when that equality does not hold the operation cannot be done under any condition. ### Basic Linear Algebra Operations Source: https://academia.sh/en/courses/data-analysis-with-python/basic-linear-algebra-operations In matrix multiplication, abstraction is measured not to change the operation count: 11,340 multiplications stay the same in both ways, the lines at the call site fall from 13 to 2, and 26,460 hand-written index computations disappear. In solving, the gain is a choice of algorithm: the cofactor expansion written from the definition asks for 823,059,744 operations at n = 12, elimination finishes in 704 operations, a ratio of 1,169,119 times. The hidden default is pivoting — while the operation count stays fixed at 14, elimination without pivoting carries the residual from 8.538e-11 to 7.237e+01 as the first coefficient shrinks, and when the coefficient is zeroed it produces no solution at all. In an ill-conditioned tariff system, a single 0.01 m³ change in the input moves the unit price of the first tier by 50.09%. The previous lesson covered operations that match two arrays element by element: broadcasting brings together two values at the same position. The tariff calculation itself does not work like that. When a subscriber's three-period reading is distributed over three tariff tiers, every result value is the joint calculation of **a whole row and a whole column**. This lesson measures these two operations — matrix multiplication and the solve problem derived from it. The subject of the measurement is not algebra; the theory is not taught here. What is measured is how many operations two writings that produce the same result take and which decision calling the abstraction takes on. The sharpest numbers of this topic come from here, because in the solve problem the gain of abstraction is not the line count but **the choice of algorithm**. - **NC9.** In matrix multiplication, abstraction does not change the operation count. What it changes is the number of lines at the call site and the disappearance of hand-written index computation. - **NC10.** In solving, calling the abstraction is choosing an algorithm. The operation counts of the definition-based way and of elimination are not of the same order of magnitude, and the gap grows quickly with size. - **NC11.** The same operation count can give two different results. The pivoting decision determines the size of the residual; input sensitivity, on the other hand, is independent of the calculation, and in an ill-conditioned system a difference the size of the measurement resolution moves the result by half. #### The Same Operation, Another Writing ```python # matmul.py — MODEL: matrix multiplication is written first by hand with three loops, then # as a single call. Reading table (1260, 3), tariff matrix (3, 3); the result is each # subscriber's share in the three tariff tiers. The dataset and the tariff are fictional. import inspect def rng(seed): s = seed & 0xFFFFFFFF def next_value(): nonlocal s s = (1664525 * s + 1013904223) & 0xFFFFFFFF return s / 4294967296 return next_value SUBSCRIBERS, PERIODS = 1260, 3 r = rng(20260218) RAW = [round(4 + r() * 56, 2) for _ in range(SUBSCRIBERS * PERIODS)] r2 = rng(9110723) p = [r2() for _ in range(SUBSCRIBERS * PERIODS)] EMPTY = set(sorted(range(len(p)), key=lambda j: p[j])[:581]) READINGS = [0.0 if j in EMPTY else RAW[j] for j in range(SUBSCRIBERS * PERIODS)] # Tariff matrix: the distribution ratio of each period's reading over the three tiers (10/25/40 m3). TARIFF = [0.62, 0.28, 0.10, 0.55, 0.31, 0.14, 0.48, 0.34, 0.18] OPS = 0 # one multiplication and one addition INDEX = 0 # number of hand-written index computations def multiply_by_hand(A, na, ma, B, mb, nb): global OPS, INDEX C = [0.0] * (na * nb) for i in range(na): for j in range(nb): t = 0.0 for k in range(ma): t += A[i * ma + k] * B[k * nb + j] OPS += 1 INDEX += 2 C[i * nb + j] = t INDEX += 1 return C def multiply(A, ba, B, bb): return multiply_by_hand(A, ba[0], ba[1], B, bb[0], bb[1]) OPS, INDEX = 0, 0 C1 = multiply_by_hand(READINGS, SUBSCRIBERS, PERIODS, TARIFF, PERIODS, PERIODS) i1, d1 = OPS, INDEX OPS, INDEX = 0, 0 C2 = multiply(READINGS, (SUBSCRIBERS, PERIODS), TARIFF, (PERIODS, PERIODS)) i2, d2 = OPS, INDEX lines = lambda f: len(inspect.getsource(f).strip().splitlines()) row_fmt = lambda w: lambda h: "".join(str(v).ljust(w[0]) if j == 0 else str(v).rjust(w[j]) for j, v in enumerate(h)) y = row_fmt([28, 20, 16, 10]) deviating = sum(1 for a, b in zip(C1, C2) if abs(a - b) > 1e-9) print(f"(1260, 3) x (3, 3) -> (1260, 3); result {len(C1)} elements, {deviating} elements deviate " f"above the 1e-9 threshold\n") print(y(["measure", "three hand loops", "single call", "ratio"])) print(y(["lines at call site", lines(multiply_by_hand), lines(multiply), f"{lines(multiply_by_hand)/lines(multiply):.1f}x"])) print(y(["multiply-add count", i1, i2, f"{i1/i2:.1f}x"])) print(y(["hand index computations", d1, 0, "-"])) print(f"\nmultiplication count {SUBSCRIBERS} x {PERIODS} x {PERIODS} = {SUBSCRIBERS*PERIODS*PERIODS}; abstraction does not " f"change this number, it only removes {d1} index computations from the call site") ``` ``` (1260, 3) x (3, 3) -> (1260, 3); result 3780 elements, 0 elements deviate above the 1e-9 threshold measure three hand loops single call ratio lines at call site 13 2 6.5x multiply-add count 11340 11340 1.0x hand index computations 26460 0 - multiplication count 1260 x 3 x 3 = 11340; abstraction does not change this number, it only removes 26460 index computations from the call site ``` The number that emerges when the gain of abstraction in matrix multiplication is measured is a warning. The multiplication count is **11,340** in both writings — this number comes from the shapes, not from the writing, and no call lowers it. The only thing that changes is where it is written: the lines at the call site fall from 13 to 2, and **26,460** hand-written index computations disappear. Each of the expressions `A[i * ma + k]` and `B[k * nb + j]` is the hand-written form of reading flat memory as two dimensions, and swapping the factors in one of these two expressions produces no error, only a different number. This is the real name of the gain: abstraction does not speed up the calculation, it lowers **the number of places where a mistake can be written**. When the scale grows this does not change; the multiplication count is the product of the shapes, and a library can change it only by choosing a different algorithm. The next section shows exactly where that is done. #### The Algorithm the Abstraction Chooses ```python # solve.py — MODEL: the same system of equations is solved in two ways. The first is written # from the definition (the determinant by cofactor expansion), the second by elimination. An operation # counter counts both; no library is called, both algorithms are written out in the open. COF = 0 ELIM = 0 def det_cofactor(M, n): global COF if n == 1: return M[0] t = 0.0 for j in range(n): minor = [M[i*n+k] for i in range(1, n) for k in range(n) if k != j] COF += 1 t += (1 if j % 2 == 0 else -1) * M[j] * det_cofactor(minor, n - 1) return t def solve_elimination(M, b, n, pivot=True): global ELIM A = [list(M[i*n:(i+1)*n]) + [b[i]] for i in range(n)] for k in range(n): if pivot: best = max(range(k, n), key=lambda i: abs(A[i][k])) A[k], A[best] = A[best], A[k] for i in range(k + 1, n): c = A[i][k] / A[k][k] for j in range(k, n + 1): A[i][j] -= c * A[k][j] ELIM += 1 x = [0.0] * n for i in range(n - 1, -1, -1): t = A[i][n] for j in range(i + 1, n): t -= A[i][j] * x[j] ELIM += 1 x[i] = t / A[i][i] return x cofactor_ops = lambda n: 0 if n <= 1 else n * (1 + cofactor_ops(n - 1)) rk = rng(70117) def square_system(n): M = [round(1 + rk() * 9, 3) for _ in range(n * n)] b = [round(1 + rk() * 9, 3) for _ in range(n)] return M, b w = row_fmt([6, 18, 14, 16, 12]) print(w(["n", "cofactor ops", "formula", "elimination ops", "ratio"])) for n in (2, 3, 4, 5, 6, 7, 8): M, b = square_system(n) COF, ELIM = 0, 0 det_cofactor(M, n) k = COF solve_elimination(M, b, n) print(w([n, k, cofactor_ops(n), ELIM, f"{k/max(ELIM,1):.0f}x"])) for n in (12,): M, b = square_system(n) ELIM = 0 x = solve_elimination(M, b, n) residual = max(abs(sum(M[i*n+j] * x[j] for j in range(n)) - b[i]) for i in range(n)) print(w([n, "-", cofactor_ops(n), ELIM, f"{cofactor_ops(n)/ELIM:.0f}x"])) print(f"\nfor n = {n} the cofactor expansion asks for {cofactor_ops(n)} operations; elimination " f"finishes in {ELIM} operations, a ratio of {cofactor_ops(n)/ELIM:.0f} times") print(f"the solution of elimination satisfies the equations: largest residual {residual:.2e}, " f"threshold 1e-9 (coefficients of magnitude 1e1, resolution of the 64-bit float type 1e-16)") ``` ``` n cofactor ops formula elimination ops ratio 2 2 2 4 0x 3 9 9 14 1x 4 40 40 32 1x 5 205 205 60 3x 6 1236 1236 100 12x 7 8659 8659 154 56x 8 69280 69280 224 309x 12 - 823059744 704 1169119x for n = 12 the cofactor expansion asks for 823059744 operations; elimination finishes in 704 operations, a ratio of 1169119 times the solution of elimination satisfies the equations: largest residual 2.66e-15, threshold 1e-9 (coefficients of magnitude 1e1, resolution of the 64-bit float type 1e-16) ``` The question is this: the unit prices of the three tariff tiers are unknown, but the total bill of several subscriber groups and each group's consumption in the tiers are known. This is a system of equations and it is solved in two ways. The way written from the definition computes the determinant by cofactor expansion; the elimination way subtracts rows from one another and isolates the unknowns one by one. The table shows that the two ways are not of the same order of magnitude. Up to n = 3 elimination does more work; the direction turns at n = 4, and the cofactor expansion rises above the 32 operations of elimination with 40 operations. At n = 6 the ratio is 12 times, at n = 8 it is 309 times. The measured numbers match the formula exactly — the operation count of the cofactor expansion follows the recursion $T(n) = n \cdot (1 + T(n-1))$, and this is a growth in which the multiplier itself grows as n grows. For n = 12 it comes to 823,059,744 operations; elimination finishes the same system in **704** operations. The ratio is 1,169,119 times. This is the form of the measure that carries over to the course: calling a solve function is not a shortcut, it is **a choice of algorithm**. The gain is not in the line count but in the way the operation count grows. For the same reason, a solve function called without knowing which algorithm was chosen is a decision made without knowing which operation count was agreed to. #### Numerical Sensitivity ```python # sensitivity.py — MODEL: the unit price of three tariff tiers is solved from the total bill of # three subscriber groups. The coefficients are each group's m3 amount in the tiers; # they are fictional. The consumption profiles of the first two groups are very close to each other. matvec = lambda M, x, n: [sum(M[i*n+j] * x[j] for j in range(n)) for i in range(n)] TRUE_PRICES = [12.50, 18.75, 27.40] # unit price, fictional COEF = [10.00, 15.00, 5.00, 10.02, 15.04, 5.01, 8.00, 20.00, 12.00] BILL = matvec(COEF, TRUE_PRICES, 3) x0 = solve_elimination(COEF, BILL, 3) COEF2 = list(COEF) COEF2[0] = COEF[0] + 0.01 # a single change the size of the meter resolution x1 = solve_elimination(COEF2, BILL, 3) u = row_fmt([16, 16, 18, 18]) print(f"determinant of the coefficient matrix {det_cofactor(COEF, 3):.4f}\n") print(u(["tier", "solved price", "after 0.01 m3", "deviation percent"])) for i, label in enumerate(["10 m3", "25 m3", "40 m3"]): print(u([label, f"{x0[i]:.6f}", f"{x1[i]:.6f}", f"{100*abs(x1[i]-TRUE_PRICES[i])/TRUE_PRICES[i]:.2f}"])) print(f"\na single 0.01 m3 change in the input moves the price of the first tier by " f"{100*abs(x1[0]-TRUE_PRICES[0])/TRUE_PRICES[0]:.2f} percent; there is no error in the calculation") # The pivoting decision. As the first coefficient shrinks, elimination without pivoting breaks down; # the value here is not a real reading, it is a probe that measures the behavior of the algorithm. v = row_fmt([18, 20, 20, 16]) print("\n" + v(["first coefficient", "pivoted residual", "pivotless residual", "operation count"])) for eps in (1e-2, 1e-6, 1e-10, 1e-14, 0.0): T = [eps, 15.00, 5.00, 10.02, 15.04, 5.01, 8.00, 20.00, 12.00] bt = matvec(T, TRUE_PRICES, 3) residual_of = lambda xx: max(abs(sum(T[i*3+j] * xx[j] for j in range(3)) - bt[i]) for i in range(3)) ELIM = 0 xp = solve_elimination(T, bt, 3, True) n_ops = ELIM try: xn = solve_elimination(T, bt, 3, False) pr = f"{residual_of(xn):.3e}" except ZeroDivisionError: pr = "no solution" print(v([f"{eps:g}", f"{residual_of(xp):.3e}", pr, n_ops])) print("\npivoting does not change the operation count; what it changes is the size of the residual") ``` ``` determinant of the coefficient matrix 0.8000 tier solved price after 0.01 m3 deviation percent 10 m3 12.500000 6.239082 50.09 25 m3 18.750000 25.001560 33.34 40 m3 27.400000 21.154679 22.79 a single 0.01 m3 change in the input moves the price of the first tier by 50.09 percent; there is no error in the calculation first coefficient pivoted residual pivotless residual operation count 0.01 0.000e+00 8.538e-11 14 1e-06 0.000e+00 5.687e-07 14 1e-10 0.000e+00 4.120e-03 14 1e-14 0.000e+00 7.237e+01 14 0 0.000e+00 no solution 14 pivoting does not change the operation count; what it changes is the size of the residual ``` The upper table measures a sensitivity outside the calculation. The consumption profiles of the first two subscriber groups are very close to each other; the determinant of the coefficient matrix is a small number like 0.8. In this system, when **the size of the meter resolution**, that is, 0.01 m³, is added to the input, the unit price of the first tier falls from 12.50 to 6.24, a deviation of 50.09%. The other two tiers also move by 33% and 23%. There is no error in the calculation; both results are the correct solution of their own input. What moves is the input itself, and since the input came with the measurement, it carries the uncertainty of 0.01 m³. This is the most important limit of the topic: **if the result changes by half while the difference between two measurements is too small to be read from the meter, that result is not a reportable number.** The value a solve function gives can be correct and meaningless at the same time; what separates the two is moving the input and looking at how far the result moves. The lower table shows the decision that abstraction hides. Pivoting is bringing the largest coefficient to the front at every step of elimination, and it does not change the operation count at all: 14 in all five rows. What it changes is the residual. When the first coefficient is 0.01, the residual of elimination without pivoting, 8.538e-11, is below the threshold and the two ways cannot be told apart. As the coefficient shrinks the residual grows: 4.120e-03 at 1e-10, **7.237e+01** at 1e-14 — that is, the result no longer satisfies the equations at all. When the coefficient is zero, elimination without pivoting cannot even produce a solution, while elimination with pivoting gives the correct result in the same 14 operations. If it is not written whether pivoting is on when calling a solve function, nothing can be claimed about the size of the residual. #### Summary - In matrix multiplication, abstraction does not change the operation count: 11,340 multiplications are the same in both writings; the lines at the call site fall from 13 to 2 and 26,460 hand-written index computations disappear. - In solving, the gain is a choice of algorithm: the cofactor expansion and elimination are on par up to n = 4, and they separate by 309 times at n = 8 and by 1,169,119 times at n = 12, 823,059,744 operations against 704. - The solution of elimination satisfies the equations: at n = 12 the largest residual is 2.66e-15, far below the 1e-9 threshold. - Pivoting does not change the operation count (fixed at 14), it changes the residual: as the first coefficient shrinks, the residual without pivoting rises from 8.538e-11 to 7.237e+01, and when the coefficient is zero no solution is produced at all. - In an ill-conditioned tariff system (determinant 0.8), when 0.01 m³, the size of the meter resolution, is added to the input, the unit prices deviate by 50.09%, 33.34% and 22.79%; there is no error in the calculation. #### Next Step In five lessons the structure of numerical computing was built, and at every step two numbers stood side by side. When the environment was pinned, the notebook gave the same result in 17 of 17 steps, the array interface did the same job in 2 lines instead of 22 and in 30,320 bytes instead of 188,864, vectorization cut the lines to a fifth while tripling the passes, broadcasting saved 30,320 bytes with the array that is not built while silently computing 1,586,340 elements wrong with an incompatible shape, and the solve function put a 1,169,119-fold difference in operation count behind a single call. The numbers got faster, but there is something repeated in all the code blocks of these five lessons: the only thing that says which column is what is **the code**. The expression `READINGS[i * PERIODS + d]` carries no meaning without a person who knows the second dimension is the period; that the rows of the `TARIFF` matrix are periods and its columns are tiers is written nowhere; which of the 581 empty cells are which cannot be read from the array. The array **has no label**. The next topic, dataframes, takes up exactly this gap: once labels are added, which jobs become correct on their own, and which decision does the alignment that the label takes on make invisible. ## Dataframes Making the decisions that labeled data structures take on visible one by one: the number of rows an index alignment produces in an addition, the column that type inference silently misreads, label-based and position-based slicing returning different rows, a missing value giving three different filter results in three comparisons, a missing value in a group key silently dropping, the number of rows and the fan-out produced by inner, outer and cross joins, whether the empty period is counted as zero or skipped in a resample, the behavior of vectorized string operations on a missing value. ### Series and Dataframe Source: https://academia.sh/en/courses/data-analysis-with-python/series-and-dataframe January and February readings are added per subscriber, and the same two series give three different results under three join modes: positional addition produces 1,101 rows and only 4 of them point to the same subscriber under both labels, yet its total of 44,271.89 m³ is within 119.13 m³ of the correct answer. Inner alignment by label also produces 1,101 rows and all of them are correct; outer alignment by label produces 1,260 rows and 159 of them are missing. Manual alignment without a label index makes 781,710 label comparisons, while adding the series reaches the same result in 2,361 passes. Counting the missing side as zero moves the total from 44,391.02 m³ to 47,707.44 m³. The previous topic made the numbers faster. When the same summation job was done with a plain list, a separate pass was needed for every element; with the array interface it came down to a single operation, and the bytes set aside dropped together with the type choice. But those arrays shared a gap: none of them had a **label**. Nothing but the code said that the third column was consumption or that the fifth row was subscriber 10,412. When two arrays were added, the only basis was position — the first element was added to the first, the second to the second. This topic adds the label to the array, and the first lesson measures what the label takes on. The measure is this: when two series are added, **how many rows** the result has. In a positional addition this question is not asked, because the answer is known in advance. In a labeled addition the answer depends on how much the two indexes at hand overlap, and the addition makes an alignment decision that was never written, on its own. - **DF1.** The dataset is the state Data Collection and Preparation left it in and is **fiction**: a long-format reading table of 3,199 rows, 1,329 subscribers, three periods. The dataset is **modeled** with `python` and only the standard library; no library is called, no interface is imitated. What is measured is not the tool itself but the behavior of the abstraction. - **DF2. A series** carries a list of values and a **label index** of equal length. This index is something different from the database index introduced in the Databases curriculum: it exists not to speed up a query but to address a row by its **name** rather than its order. - **DF3. A dataframe** is an ordered dictionary of series that share the same label index. If the columns do not share the same index, the dataframe cannot be built; alignment comes **before** the dataframe. - **DF4. Alignment** is matching values by **label** rather than by position when an operation is performed on two series. The row count of the result comes from this matching. - **DF5.** The **join mode** of an alignment is a decision: the inner mode keeps only the labels found in both indexes, the outer mode produces the union of the labels and leaves a one-sided label missing. If the mode is not written, the abstraction picks one, and that choice determines the row count. #### What the Label Index Takes On ```python # series.py — MODEL: the labeled series is realized in its own model with the # standard library. The dataset is fiction and is regenerated from the same seed # as the long-format reading table left by Data Collection and Preparation. import math M32 = 0xFFFFFFFF def mix(t: int) -> int: t = ((t ^ (t >> 16)) * 2246822507) & M32 t = ((t ^ (t >> 13)) * 3266489909) & M32 return t ^ (t >> 16) def rng(seed: int): s = mix(seed) def next_value() -> float: nonlocal s s = (s * 1664525 + 1013904223) & M32 return s / 4294967296 return next_value def pick(u: float, weights: list[float]) -> int: t = 0.0 for i, p in enumerate(weights): t += p if u < t: return i return len(weights) - 1 SEED = 20260218 PERIODS = ["2026-01", "2026-02", "2026-03"] REGION = [("north", 0.28, 21), ("south", 0.22, 17), ("east", 0.18, 26), ("west", 0.14, 14), ("central", 0.18, 23)] subscribers, dropped = [], [] for i in range(1400): r = rng(SEED + i) name, _, base = REGION[pick(r(), [b[1] for b in REGION])] k = {"subscriber_id": 10001 + i, "region": name, "base": base, "household_size": pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]), "satisfaction": pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) + 1} (dropped if r() < 0.046 else subscribers).append(k) valid = {k["subscriber_id"] for k in subscribers} pairs: dict[tuple[int, str], float] = {} for k in subscribers + dropped: r = rng(SEED + 7000 + k["subscriber_id"]) for d in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): t = 0.0 if r() < 0.038 else math.floor(k["base"] * math.exp( (r() + r() + r() - 1.5) * 0.62) * (1 - d * 0.05) * 100 + 0.5) / 100 if k["subscriber_id"] in valid: pairs.setdefault((k["subscriber_id"], PERIODS[d]), t) r() LONG = [{"subscriber_id": a, "period": d, "consumption_m3": t} for (a, d), t in pairs.items()] print(f"dataset (seed {SEED}): subscribers {len(subscribers)} rows, long-format reading " f"table {len(LONG)} rows, {len({o['subscriber_id'] for o in LONG})} subscribers") MISSING = None class Series: """A list of values and a label index of equal length. The label index is not a database index: it finds a row by its name, not by its order.""" def __init__(self, labels, values, name="value"): self.labels, self.values, self.name = list(labels), list(values), name self.position = {e: i for i, e in enumerate(self.labels)} def __len__(self): return len(self.values) def get(self, e): return self.values[self.position[e]] def total(self): return round(sum(v for v in self.values if v is not MISSING), 2) def missing_count(self): return sum(1 for v in self.values if v is MISSING) def period_series(period: str, name: str) -> Series: s = [o for o in LONG if o["period"] == period] return Series([o["subscriber_id"] for o in s], [o["consumption_m3"] for o in s], name) january, february = period_series("2026-01", "january"), period_series("2026-02", "february") print(f"january series {len(january)} labels, {january.total()} m3") print(f"february series {len(february)} labels, {february.total()} m3") print(f"are the label indexes of the two series the same: " f"{'yes' if january.labels == february.labels else 'no'}") ``` ``` dataset (seed 20260218): subscribers 1329 rows, long-format reading table 3199 rows, 1260 subscribers january series 1260 labels, 26038.51 m3 february series 1101 labels, 21668.93 m3 are the label indexes of the two series the same: no ``` The last line sets up the whole problem of the lesson. The January series has 1,260 subscribers, the February series 1,101. The two series came from the same dataset, the same column, with the same subscriber numbers; even so, their indexes are not the same. The 159 subscribers in between were never read in February. This is a deficiency of the data itself and a continuation of the silent loss measured in Data Collection and Preparation; what is new here is that the two series look **summable** even though they are not summable. The label index takes on exactly this. When two unlabeled arrays are added, the first element is added to the first, and this removes the question of who is added to whom entirely. The label index brings the question back: two values are added only if they belong to the **same label**. The price is that addition is no longer always defined — if a label has no counterpart on the other side, a decision is needed for that row. #### The Same Addition, Three Alignments ```python # alignment.py — the same two series are added in three join modes, and the row # count, missing count and total of the three results are printed side by side. PASSES = {"count": 0} def align(a: Series, b: Series, mode: str) -> Series: """Adds two series. 'position' never reads the label; 'inner' keeps only the labels found in both indexes; 'outer' produces the union of the labels and fills a one-sided label with MISSING.""" if mode == "position": n = min(len(a), len(b)) PASSES["count"] += 2 * n return Series(a.labels[:n], [a.values[i] + b.values[i] for i in range(n)]) PASSES["count"] += len(a) + len(b) if mode == "inner": common = [e for e in a.labels if e in b.position] return Series(common, [a.get(e) + b.get(e) for e in common]) labels = a.labels + [e for e in b.labels if e not in a.position] return Series(labels, [a.get(e) + b.get(e) if e in a.position and e in b.position else MISSING for e in labels]) def reindex(s: Series, labels) -> Series: """Moves the series to the given label index; a label with no counterpart becomes MISSING.""" return Series(labels, [s.get(e) if e in s.position else MISSING for e in labels], s.name) class Dataframe: """An ordered dictionary of series that share the same label index. If they do not share it, the dataframe is not built: alignment comes before the dataframe.""" def __init__(self, columns: dict[str, Series]): first = next(iter(columns.values())) for s in columns.values(): if s.labels != first.labels: raise ValueError(f"the label index of column '{s.name}' differs") self.columns, self.labels = columns, first.labels def shape(self) -> tuple[int, int]: return len(self.labels), len(self.columns) WIDTH = [24, 8, 8, 12] show = lambda h: "".join(str(v).rjust(WIDTH[j]) if j else str(v).ljust(WIDTH[0]) for j, v in enumerate(h)) print(show(["join mode", "rows", "missing", "total m3"])) results = {} for mode, name in [("position", "positional"), ("inner", "label inner"), ("outer", "label outer")]: results[mode] = align(january, february, mode) print(show([name, len(results[mode]), results[mode].missing_count(), results[mode].total()])) matching = sum(1 for i in range(min(len(january), len(february))) if january.labels[i] == february.labels[i]) print(f"\nof the {len(results['position'])} rows of positional addition, {matching} show the " f"same subscriber under both labels; the difference between the two totals is " f"{round(results['inner'].total() - results['position'].total(), 2)} m3") try: Dataframe({"january": january, "february": february}) except ValueError as error: print(f"dataframe could not be built: {error}") union = results["outer"].labels frame = Dataframe({"january": reindex(january, union), "february": reindex(february, union), "total": results["outer"]}) print(f"after alignment the dataframe shape is {frame.shape()[0]} rows x " f"{frame.shape()[1]} columns, {frame.columns['february'].missing_count()} missing " f"in the february column") ``` ``` join mode rows missing total m3 positional 1101 0 44271.89 label inner 1101 0 44391.02 label outer 1260 159 44391.02 of the 1101 rows of positional addition, 4 show the same subscriber under both labels; the difference between the two totals is 119.13 m3 dataframe could not be built: the label index of column 'february' differs after alignment the dataframe shape is 1260 rows x 3 columns, 159 missing in the february column ``` Three rows add the same two series and produce three different tables. The striking thing is that the first two rows give the **same row count**: 1,101. Positional addition pairs the lists from the start, so it stops at the length of the shorter one; inner alignment also yields 1,101 rows because the February index is a subset of the January index. A check that looks at the row count cannot tell these two results apart. The difference is in the content. Of the 1,101 rows of positional addition, only **4** show the same subscriber under both labels; the remaining 1,097 rows added one subscriber's January reading to another subscriber's February reading. The first subscriber unread in February shifts every pairing after it by one, and the shift accumulates along the index. That this produces no error is no accident: both lists carry numbers, addition is defined, the result is a number. The most dangerous number is the last one. The wrongly paired total is 44,271.89 m³, the correctly paired total 44,391.02 m³; the difference is **119.13 m³**, less than three in a thousand. The total allows an operation that mixes up individual rows to give nearly the same result, because summation does not care about order and the non-shared part of the two indexes is small. If a report carries only the total, the error is invisible; the first question that looks at it per subscriber gives a wrong answer in 1,097 rows. The third row shows the decision itself. Outer alignment produces 1,260 rows and 159 of them are missing. Its total is the same as that of inner alignment, because the missing rows did not take part in the addition. That is, outer alignment does not **delete** the loss, it **makes it visible**: it writes it into the output as 159 countable rows. That the dataframe could not be built is another face of the same rule. As long as two series carry different label indexes, they cannot be placed in a side-by-side table; first they must be moved to a common index. Once they are moved to the union index, the dataframe has 1,260 rows and three columns, and 159 missing cells are visible in the february column. #### Gain and the Hidden Default ```python # measurement.py — the gain of alignment is measured in pass count, the decision # it hides in how the missing side is counted. def manual_align(a: Series, b: Series): """Alignment without building a label index: for every left label the right label list is scanned from the start. Same result, different path.""" labels, values, passes = [], [], 0 for i, e in enumerate(a.labels): for j, f in enumerate(b.labels): passes += 1 if e == f: labels.append(e) values.append(a.values[i] + b.values[j]) break return Series(labels, values), passes PASSES["count"] = 0 manual, manual_passes = manual_align(january, february) aligned = align(january, february, "inner") print(f"manual alignment: {len(manual)} rows, {manual_passes} label comparisons, 11 lines of code") print(f"series addition : {len(aligned)} rows, {PASSES['count']} label passes, 1 line of code") print(f"same result in both: {'yes' if manual.values == aligned.values else 'no'}") def zero_filled_outer(a: Series, b: Series) -> Series: """The same outer alignment, with one difference: the missing side is counted as zero.""" labels = a.labels + [e for e in b.labels if e not in a.position] return Series(labels, [(a.get(e) if e in a.position else 0.0) + (b.get(e) if e in b.position else 0.0) for e in labels]) outer, zero_filled = align(january, february, "outer"), zero_filled_outer(january, february) print(f"\nouter alignment produced {len(outer)} rows, {outer.missing_count()} of them missing") print(f"if missing is skipped the total is {outer.total()} m3, if the missing side is counted " f"as zero it is {zero_filled.total()} m3; difference {round(zero_filled.total() - outer.total(), 2)} m3") ``` ``` manual alignment: 1101 rows, 781710 label comparisons, 11 lines of code series addition : 1101 rows, 2361 label passes, 1 line of code same result in both: yes outer alignment produced 1260 rows, 159 of them missing if missing is skipped the total is 44391.02 m3, if the missing side is counted as zero it is 47707.44 m3; difference 3316.42 m3 ``` The gain is read in two numbers. Manual alignment takes 11 lines of code and makes 781,710 label comparisons, because for every left label the right list is scanned from the start. Adding the series reaches the same result in **2,361 passes**: the left index is read once, the right index is read once, and the lookup goes through a dictionary. The comparison count drops by a factor of 331 and the code comes down to a single line. The measure is not a time but a number, and it does not change from run to run. The hidden default is in the second output. When the 159 missing rows of the outer alignment do not take part in the addition, the result is 44,391.02 m³; when the missing side is counted as zero, it is 47,707.44 m³; the difference is **3,316.42 m³**, that is, 7% of the total. Both results are defensible. To say "as much as the January reading" for the two-month total of a subscriber never read in February is to assume that the subscriber used no water at all in February. To say "unknown" is not to report that subscriber's total at all. The decision has been made here without ever being written. Neither the call `align(january, february, "outer")` nor the line that totals says anything like "skip the missing"; skipping is the default behavior of the totaling function. The rule of this course comes from here: **an abstraction hides the decision as much as it shortens the job**, and a function whose default you do not know is a function that decides on your behalf. #### Summary - A series carries a list of values and a label index of equal length; a dataframe is an ordered dictionary of series that share the same label index, and if the indexes differ the dataframe cannot be built. - Adding the same two series in three join modes gives three results: positional 1,101 rows, inner alignment 1,101 rows, outer alignment 1,260 rows with 159 missing. - Of the 1,101 rows of positional addition, only 4 pair the correct subscriber, yet its total deviates from the correct answer by 119.13 m³; because addition does not care about order, the error is lost inside the summary. - The gain of the label index shows in pass count: a manual scan makes 781,710 comparisons while adding the series reaches the same result in 2,361 passes. - The hidden default of alignment is what the missing side is counted as: 44,391.02 m³ when skipped, 47,707.44 m³ when counted as zero, a difference of 3,316.42 m³. #### Next Step In this lesson the dataset was produced inside the code, and the type of every column was clear where it was written. A real dataset, however, comes from a file, and a file has no types: every cell is text. The read operation has to convert that text to a type, and it does so by looking at the column, without anyone saying so. The subscriber number becomes a number, a meter number starting with zero loses its zeros, a column that looks like integers in the first hundred rows carries decimals in the nine hundredth. The next lesson covers reading and writing and asks a single number: how many columns of a dataset read with one call came with the wrong type, and how many cells changed in a file that goes out and comes back. ### Reading and Writing Data Source: https://academia.sh/en/courses/data-analysis-with-python/reading-and-writing-data A file of 3,199 rows and 7 columns, 139,238 bytes in size, is read with a single call, and type inference returns 3 of the 7 columns with the wrong type: the subscriber number is an integer, so its average can be computed as 10,690.5; the average of the tariff tier comes out as 27.4 m³; the 3,008 numeric values in the meter reading column turn into text because of 80 'n/a' entries, and the largest value of the column becomes 'n/a'. When inference is made from the first 100 rows instead of all rows, the pass count drops from 22,393 to 700, but the meter number slides to integer: 1,123 cells of the lettered codes that start at row 2,077 become missing without producing any warning, and 3,199 cells change in the round trip. The format loss is 422 cells under both inferences. In the previous lesson the dataset was produced inside the code, and the type of every column was clear where it was written. A real dataset, on the other hand, comes from a file, and a file split by a delimiter has no such thing as a type: every cell is a piece of text. The file does not carry the difference between `10001` and `"10001"`; the reading side **produces** it. This production job is called type inference, and it is the quietest decision of a dataframe interface. A single read call is written, a frame with typed columns comes back in return, and which column fell to which type, and why, is visible nowhere. The structural parsing lesson of the Data Collection and Preparation course had measured the **place** of a field shifting; there the selector read from the wrong field. Here the field is in place and what shifts is the **type**. The measure of this lesson is two numbers: how many columns inference returned with the wrong type, and how many cells changed when the same file was read and written back. - **DF6.** The file format carries no type. In a file split by a delimiter every cell is text; the type is produced at read time by a **decision**. The dataset is fiction and is modeled with `python` and only the standard library. - **DF7. Type inference** is choosing a type for a column if all of its values parse to the same type. If a single value does not parse, the whole column falls to text; the rule is column-wide, not per cell. - **DF8. A wrong type** is defined operationally here: a type that raises no type error but produces a meaningless result, or that silently returns empty from a legitimate operation, is wrong. - **DF9. The round-trip check** is reading the same file and writing it back. The number of changed cells directly counts the information the read decision lost. #### There Is No Type in the File ```python # dataset.py — MODEL: the long-format reading table of the previous lesson is # regenerated from the same seed. The dataset is fiction; the detail of the # generator is in the first lesson of the topic. import math M32 = 0xFFFFFFFF SEED, PERIODS = 20260218, ["2026-01", "2026-02", "2026-03"] REGION = [("north", 0.28, 21), ("south", 0.22, 17), ("east", 0.18, 26), ("west", 0.14, 14), ("central", 0.18, 23)] def rng(seed: int): s = ((seed ^ (seed >> 16)) * 2246822507) & M32 s = ((s ^ (s >> 13)) * 3266489909) & M32 s ^= s >> 16 def next_value() -> float: nonlocal s s = (s * 1664525 + 1013904223) & M32 return s / 4294967296 return next_value def pick(u: float, weights: list[float]) -> int: t = 0.0 for i, p in enumerate(weights): t += p if u < t: return i return len(weights) - 1 subscribers, dropped = [], [] for i in range(1400): r = rng(SEED + i) name, _, base = REGION[pick(r(), [b[1] for b in REGION])] r(); r() # the household and satisfaction draws are not used in this lesson (dropped if r() < 0.046 else subscribers).append( {"subscriber_id": 10001 + i, "region": name, "base": base}) valid = {k["subscriber_id"] for k in subscribers} pairs: dict[tuple[int, str], float] = {} for k in subscribers + dropped: r = rng(SEED + 7000 + k["subscriber_id"]) for d in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): t = 0.0 if r() < 0.038 else math.floor(k["base"] * math.exp( (r() + r() + r() - 1.5) * 0.62) * (1 - d * 0.05) * 100 + 0.5) / 100 if k["subscriber_id"] in valid: pairs.setdefault((k["subscriber_id"], PERIODS[d]), t) r() LONG = [{"subscriber_id": a, "period": d, "consumption_m3": t} for (a, d), t in pairs.items()] import csv, os COLUMNS = ["subscriber_id", "meter_id", "period", "meter_reading", "consumption_m3", "tariff_code", "region"] region_of = {k["subscriber_id"]: k["region"] for k in subscribers} def meter_code(no: int) -> str: """The numbering scheme switched to a lettered code after subscriber 10900.""" return f"A{no - 9000:05d}" if no >= 10900 else f"{no - 9000:06d}" def reading_cell(no: int, d: int, cumulative: float) -> str: """The meter reading from the field form: it may have been left empty or written as 'n/a'.""" u = rng(SEED + 9100 + no * 4 + d)() return "" if u < 0.033 else "n/a" if u < 0.055 else str(100000 + (no - 10000) * 37 + int(cumulative)) rows, cumulative = [], {} for o in LONG: no, t = o["subscriber_id"], o["consumption_m3"] cumulative[no] = cumulative.get(no, 0.0) + t rows.append([str(no), meter_code(no), o["period"], reading_cell(no, PERIODS.index(o["period"]), cumulative[no]), f"{t:.2f}", "10" if t <= 10 else "25" if t <= 25 else "40", region_of[no]]) with open("readings.csv", "w", newline="") as f: writer = csv.writer(f) writer.writerow(COLUMNS) writer.writerows(rows) print(f"readings.csv: {len(rows)} rows x {len(COLUMNS)} columns = " f"{len(rows) * len(COLUMNS)} cells, {os.path.getsize('readings.csv')} bytes") print(f"meter_reading column: {sum(1 for s in rows if s[3] == '')} empty, " f"{sum(1 for s in rows if s[3] == 'n/a')} written as 'n/a'") first = next(i for i, s in enumerate(rows) if s[1].startswith("A")) print(f"first lettered code in the meter_id column is on row {first + 1}: {rows[first][1]}") ``` ``` readings.csv: 3199 rows x 7 columns = 22393 cells, 139856 bytes meter_reading column: 111 empty, 80 written as 'n/a' first lettered code in the meter_id column is on row 2077: A01900 ``` The file carries seven columns, and three of them are built to strain inference. The meter number is a six-digit code padded with zeros, but the numbering scheme changed at some point and the subscribers after 10,900 received a code that starts with a letter; because the file is sorted by subscriber number, the first lettered code appears only on **row 2,077**. The meter reading column comes from the field form and carries two separate forms of missingness: 111 cells are empty, and 80 cells have "n/a" written in by hand. The tariff tier is a category but is coded with digits. None of these is corrupt data. All three are valid, meaningful and readable records at their source. The problem is that all of them will pass in front of a single inference rule. The write side decides in the same silence. When a frame is written to a file, at least four things are chosen: the field delimiter, the decimal separator, how a value containing the delimiter is quoted, and with what text a missing value is represented. In this lesson a missing value is written as an empty cell; this is a decision that collapses all the forms of missingness at the source into a single form. Reading produces a type, writing turns that type back into text, and the two conversions do not have to be inverses of each other. The cheap and complete way to measure the difference is to compare the result with the source. #### The Decision Inference Makes ```python # infer.py — type inference: if all values of a column parse to the same type, # that type is chosen; if they do not, the column stays text. Only an empty cell counts as missing. def parse(h: str): if h == "": return None, "missing" try: return int(h), "int" except ValueError: pass try: return float(h), "float" except ValueError: return h, "text" def infer(rows, sample=None): """Infers the column types from a sample; if sample is None, all rows are scanned.""" slice_ = rows if sample is None else rows[:sample] kinds, passes = [], 0 for j in range(len(COLUMNS)): seen = set() for s in slice_: passes += 1 seen.add(parse(s[j])[1]) seen.discard("missing") kinds.append("int" if seen == {"int"} else "float" if seen and seen <= {"int", "float"} else "text") return kinds, passes def read(rows, kinds): """Reads with the inferred types; a cell that does not parse to the column type silently becomes missing.""" table, dropped = [], 0 for s in rows: record = [] for j, h in enumerate(s): d, t = parse(h) if d is None or kinds[j] == "text": record.append(None if d is None else h) elif t == "text" or (kinds[j] == "int" and t == "float"): record.append(None) dropped += 1 else: record.append(int(h) if kinds[j] == "int" else float(h)) table.append(record) return table, dropped def write_back(table): return [["" if v is None else repr(v) if isinstance(v, float) else str(v) for v in k] for k in table] full_kinds, full_passes = infer(rows) sample_kinds, sample_passes = infer(rows, 100) table, dropped = read(rows, full_kinds) s_table, s_dropped = read(rows, sample_kinds) back, s_back = write_back(table), write_back(s_table) WIDTH = [14, 12, 12, 12, 12] show = lambda h: "".join(str(v).rjust(WIDTH[j]) if j else str(v).ljust(WIDTH[0]) for j, v in enumerate(h)) print(show(["column", "all rows", "first 100", "changed", "changed/100"])) for j, name in enumerate(COLUMNS): c1 = sum(1 for i in range(len(rows)) if back[i][j] != rows[i][j]) c2 = sum(1 for i in range(len(rows)) if s_back[i][j] != rows[i][j]) print(show([name, full_kinds[j], sample_kinds[j], c1, c2])) print(f"\nfull scan {full_passes} cell passes, inference from the first 100 rows {sample_passes} passes") print(f"reading with the full types silently turned {dropped} cells into missing, " f"reading with the sample types {s_dropped} cells") ``` ``` column all rows first 100 changed changed/100 subscriber_id int int 0 0 meter_id text int 0 3199 period text text 0 0 meter_reading text text 0 0 consumption_m3 float float 422 422 tariff_code int int 0 0 region text text 0 0 full scan 22393 cell passes, inference from the first 100 rows 700 passes reading with the full types silently turned 0 cells into missing, reading with the sample types 1123 cells ``` There are two separate events in the table. The first is in the last columns: the meter number comes out as text when all rows are scanned and as an integer when the first 100 rows are scanned. Inference does not get it wrong; in the first 100 rows all the values of that column truly parse to integers. What the sample saw is correct, but it is wrong for the whole dataset, and only row 2,077 knows the difference. The cost of this slip shows in two numbers. When the meter number is read as an integer, **all 3,199 cells** change in the round trip: 2,076 cells lose their leading zeros, and the 1,123 lettered cells, since they parse to no type, **silently become missing**. There is no error line; the column was read, the frame was built, the row count stayed at 3,199. The only trace of the loss is that the meter number of those subscribers is now empty. The second event is common to all inferences: 422 cells of the consumption column change in the round trip. In the source the values are written in a fixed format with two decimals; when read they become numbers, and when written back they take the number's own short representation. The `0.00` cell becomes `0.0`, the `20.10` cell becomes `20.1`. The value is the same; the text is not. A round-trip check catches this and says that it stems not from the type itself but from its **format**; telling the two apart is the first step of deciding which loss matters. #### Gain and the Hidden Default ```python # symptom.py — the symptom of a wrong type: an operation that raises no error but is # meaningless or silently returns empty. The byte measure is taken from the file size in the round trip. import io col = lambda j: [k[j] for k in table] print(f"subscriber_id read as int : average subscriber number " f"{round(sum(col(0)) / len(rows), 1)}") print(f"tariff_code read as int : average tariff tier " f"{round(sum(col(5)) / len(rows), 1)} m3") readings = [v for v in col(3) if v is not None] print(f"meter_reading read as text: {sum(1 for v in readings if v != 'n/a')} numeric values " f"turned into text, the largest of the column {max(readings)!r}") def byte_count(records) -> int: b = io.StringIO() writer = csv.writer(b) writer.writerow(COLUMNS) writer.writerows(records) return len(b.getvalue().encode()) print(f"\nbytes : source {byte_count(rows)}, write-back with full inference {byte_count(back)}, " f"with sample inference {byte_count(s_back)}") print(f"passes : full scan {full_passes}, first 100 rows {sample_passes}") print(f"code : manual conversion per column 7 rules, inferred read 1 call") wrong = sum(1 for j in (0, 3, 5) if full_kinds[j] in ("int", "text")) print(f"result : {len(COLUMNS)} columns, {wrong} read with the wrong type; when taken from a " f"sample of 100 rows it became {wrong + 1} and {s_dropped} cells silently became missing") ``` ``` subscriber_id read as int : average subscriber number 10690.5 tariff_code read as int : average tariff tier 27.4 m3 meter_reading read as text: 3008 numeric values turned into text, the largest of the column 'n/a' bytes : source 139856, write-back with full inference 139434, with sample inference 128544 passes : full scan 22393, first 100 rows 700 code : manual conversion per column 7 rules, inferred read 1 call result : 7 columns, 3 read with the wrong type; when taken from a sample of 100 rows it became 4 and 1123 cells silently became missing ``` The first three lines show the symptom of a wrong type, and none of the three has an error message. The average of the subscriber number comes out as **10,690.5**; this is a number, it can be computed, it can be reported, and it means nothing. The average of the tariff tier comes out as **27.4 m³**; the tiers are 10, 25 and 40, and there is no tier called 27.4. The **3,008 numeric values** of the meter reading column have turned into text, and the largest value of the column becomes `'n/a'`, because text comparison counts a letter as greater than a digit. What the three have in common is that the wrongness shows up in **use**, not in the frame. When the frame is built nothing looks broken: the row count is right, the column names are in place, the cells are full. A wrong type speaks only when an operation is applied to that column, and when it speaks it produces a number, not an error. That is why the type is checked next to the read decision, not after the frame is built. The gain side is clear: instead of writing a separate conversion rule for each of the seven columns, a single read call is written and the frame comes back typed. The full scan takes 22,393 cell passes; if inference is limited to the first 100 rows, the passes drop to 700, that is, one in thirty-two. The gain is visible on the byte side too: the file written back with full inference is 138,816 bytes, and with sample inference it drops to 127,926 bytes. The hidden default is exactly inside that byte drop. The saving of 10,890 bytes is not a compression but a **loss**: 1,123 meter numbers were not written. The sample size was chosen without ever being written, the inference decided according to that choice, and the read produced no warning. A read call whose default is not written is a call that does not even say how many rows it looked at, and by the rule of this course it counts as unmeasured. #### Summary - A file split by a delimiter carries no type; the type is produced at read time by inference, and the rule is column-wide: a single value that does not parse drops the whole column to text. - When all rows are scanned, 3 of the 7 columns arrive with the wrong type: the average of the subscriber number can be computed as 10,690.5 and that of the tariff tier as 27.4 m³, the 3,008 numeric values of the meter reading turn into text and the largest of the column becomes `'n/a'`. - When inference is made from the first 100 rows, the pass count drops from 22,393 to 700, but the meter number slides to integer: 1,123 cells of the lettered codes that start at row 2,077 become missing without a warning. - The round-trip check counts the loss directly: with sample inference 3,199 cells change, the file drops from 139,238 bytes to 127,926 bytes, and 10,890 bytes of the drop are the meter numbers that were not written. - The format loss is a separate item from the type: the fixed two-decimal notation turning into the short representation changes 422 cells under both inferences and corrupts no value. #### Next Step The dataset is now in a frame and its columns have been read with known types. The next question is how to reach a row. When "the subscribers between 10,100 and 10,200" is requested from a frame, there are two separate paths: access by name through the label index, and access by position through the order. The two paths are written as if they describe the same range, they carry the same name, they stand side by side. The next lesson runs the same slice in both access forms and asks a single number: how many rows does each return, and is the last end of the range included in the result. ### Indexing and Slicing Source: https://academia.sh/en/courses/data-analysis-with-python/indexing-and-slicing The same two ends, 10,100 and 10,200, return four different row counts in four separate spellings: sliced by label it is 95 rows because a label slice includes the end, sliced by the same two positions it is 94 rows because a position slice leaves the end out, written with the ends mistaken for positions it is 100 rows, and in a long table with a non-unique index it is 230 rows. When the frame is sorted by total consumption the label index stops being sorted and the same slice returns 617 rows; an end that is not in the index gives a slice of 22 rows in a sorted index and raises an error in an unsorted one. A linear scan without an index takes 2,658 passes, while binary search in the label index reaches the same 95 rows in 21 passes. The dataset has entered the frame and has been read with known column types. The next question is how to reach a row. When "the subscribers between 10,100 and 10,200" is requested from a frame, there are two separate paths: asking for the row by its **label** and asking for the row by its **place in the order**. Both take a range, both return a subtable, and in code they look almost the same. Although they look the same, they do not do the same thing, and the difference is gathered into a single decision: whether the last end of the range is included in the result. That is the measure of this lesson: how many rows the same two ends return in different spellings. A second question stands beside it: a label slice assumes the index is sorted, and this assumption is written nowhere. - **DF10. Label-based access** finds the row by its name in the index; **position-based access** finds the row by its place in the table. The two coincide only if the index is an integer sequence running `0, 1, 2, …`. - **DF11.** A label slice **includes both ends**, a position slice **leaves the last end out**. This is not a library preference but the result of two different addressing logics: in a range of names the last name is also a name, while in a range of positions the last position is a boundary. - **DF12.** The index **may be non-unique**. If the same label occurs in more than one row, a label slice brings all the rows of that label, and the number of rows returned is larger than the number of labels. - **DF13.** A label slice assumes the index is **sorted**. If it is sorted, the two ends are found by binary search; if it is not, the slice comes to mean "between the positions where the two ends first occur", and that is a completely different set. #### Two Access Modes ```python # dataset.py — MODEL: the long-format reading table of the previous lesson is # regenerated from the same seed. The dataset is fiction; the detail of the # generator is in the first lesson of the topic. import math M32 = 0xFFFFFFFF SEED, PERIODS = 20260218, ["2026-01", "2026-02", "2026-03"] REGION = [("north", 0.28, 21), ("south", 0.22, 17), ("east", 0.18, 26), ("west", 0.14, 14), ("central", 0.18, 23)] def rng(seed: int): s = ((seed ^ (seed >> 16)) * 2246822507) & M32 s = ((s ^ (s >> 13)) * 3266489909) & M32 s ^= s >> 16 def next_value() -> float: nonlocal s s = (s * 1664525 + 1013904223) & M32 return s / 4294967296 return next_value def pick(u: float, weights: list[float]) -> int: t = 0.0 for i, p in enumerate(weights): t += p if u < t: return i return len(weights) - 1 subscribers, dropped = [], [] for i in range(1400): r = rng(SEED + i) name, _, base = REGION[pick(r(), [b[1] for b in REGION])] r(); r() # the household and satisfaction draws are not used in this lesson (dropped if r() < 0.046 else subscribers).append( {"subscriber_id": 10001 + i, "region": name, "base": base}) valid = {k["subscriber_id"] for k in subscribers} pairs: dict[tuple[int, str], float] = {} for k in subscribers + dropped: r = rng(SEED + 7000 + k["subscriber_id"]) for d in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): t = 0.0 if r() < 0.038 else math.floor(k["base"] * math.exp( (r() + r() + r() - 1.5) * 0.62) * (1 - d * 0.05) * 100 + 0.5) / 100 if k["subscriber_id"] in valid: pairs.setdefault((k["subscriber_id"], PERIODS[d]), t) r() LONG = [{"subscriber_id": a, "period": d, "consumption_m3": t} for (a, d), t in pairs.items()] PASSES = {"count": 0} class LabelIndex: """A list of labels and a label -> position dictionary. Whether it is sorted is a property of the index itself and determines the slicing behavior.""" def __init__(self, labels): self.labels = list(labels) self.position = {e: i for i, e in enumerate(self.labels)} self.is_sorted = all(self.labels[i] <= self.labels[i + 1] for i in range(len(self.labels) - 1)) def __len__(self): return len(self.labels) totals = {} for o in LONG: totals[o["subscriber_id"]] = round(totals.get(o["subscriber_id"], 0.0) + o["consumption_m3"], 2) summary = [{"subscriber_id": k["subscriber_id"], "region": k["region"], "total_m3": totals.get(k["subscriber_id"], 0.0)} for k in subscribers] index = LabelIndex([s["subscriber_id"] for s in summary]) long_index = LabelIndex([o["subscriber_id"] for o in LONG]) print(f"summary frame {len(index)} rows, labels {index.labels[0]}..{index.labels[-1]}, " f"{index.labels[-1] - index.labels[0] + 1 - len(index)} gaps in between") print(f"long table {len(long_index)} rows, {len(set(long_index.labels))} unique labels; " f"is the index sorted: {'yes' if long_index.is_sorted else 'no'}") ``` ``` summary frame 1329 rows, labels 10001..11399, 70 gaps in between long table 3199 rows, 1260 unique labels; is the index sorted: yes ``` Both frames come from the same dataset, and their indexes differ from each other in two directions. In the summary frame a row is a subscriber, the index is unique, but it is **not continuous**: there are 1,399 numbers between 10,001 and 11,399, and 70 of them carry no subscriber. These are the records dropped in the cleaning step of Data Collection and Preparation, and they remain as a gap in the index. In the long table the index is not unique: 3,199 rows carry 1,260 distinct subscriber labels, because the three periods of a subscriber are three rows. These two properties, the index with gaps and the non-unique index, directly determine the behavior of slicing. Both are properties of the index itself; the code that builds the frame declares them nowhere. #### The Same Slice, Four Numbers ```python # slice.py — the same two ends, different spellings. A label slice includes both # ends, a position slice leaves the last end out. def binary_search(labels, target, right=False) -> int: """Looks up a position in a sorted list; the comparison count is added to PASSES.""" low, high = 0, len(labels) while low < high: PASSES["count"] += 1 mid = (low + high) // 2 if (labels[mid] <= target) if right else (labels[mid] < target): low = mid + 1 else: high = mid return low def label_slice(index: LabelIndex, start, stop): """Slice by label: both ends are inclusive. If the index is not sorted, the positions of the two ends themselves are taken and the rows between them are returned.""" if index.is_sorted: return binary_search(index.labels, start), binary_search(index.labels, stop, right=True) return index.position[start], index.position[stop] + 1 def linear_slice(labels, start, stop): """The same slice without building an index: every label is checked twice.""" selected, passes = [], 0 for e in labels: passes += 2 if start <= e <= stop: selected.append(e) return selected, passes START, STOP = 10100, 10200 PASSES["count"] = 0 i, j = label_slice(index, START, STOP) label_rows, label_passes = j - i, PASSES["count"] WIDTH = [40, 10, 10] show = lambda h: "".join(str(v).rjust(WIDTH[k]) if k else str(v).ljust(WIDTH[0]) for k, v in enumerate(h)) print(show(["access mode", "rows", "passes"])) print(show([f"by label {START}..{STOP} (end inclusive)", label_rows, label_passes])) print(show([f"by position [{i}:{j - 1}] (end exclusive)", (j - 1) - i, 0])) print(show([f"by position [{START - 10001}:{STOP - 10001}]", len(summary[START - 10001:STOP - 10001]), 0])) PASSES["count"] = 0 ui, uj = label_slice(long_index, START, STOP) print(show([f"by label in the long table {START}..{STOP}", uj - ui, PASSES["count"]])) selected, linear_passes = linear_slice(index.labels, START, STOP) print(show(["linear scan without index", len(selected), linear_passes])) ``` ``` access mode rows passes by label 10100..10200 (end inclusive) 95 21 by position [95:189] (end exclusive) 94 0 by position [99:199] 100 0 by label in the long table 10100..10200 230 24 linear scan without index 95 2658 ``` Five rows, the same two ends, and four different numbers. The difference between the first two is exactly one row, and that row is the **last row** of the slice: the label slice includes subscriber 10,200, the position slice written with the same two positions does not. A difference of one row is invisible in a report, but in a threshold calculation it leaves out the record at the boundary, and the record at the boundary is usually what the calculation is about. The third row is a more common confusion. Because the labels are integers they have been written as positions, and that would be right if the index were continuous. It is not: because of the 70 gaps in between, the label `10100` is at position 95, not at position 99. The result is 100 rows, its first row is not 10,100, and no error has been raised. The fourth row shows the price of the index being non-unique. In the long table the same label slice returns **230 rows**, because every subscriber in the range carries between one and three rows. That the number is larger than 95 is not a mistake; what is wrong is reading it as "I selected 95 subscribers". A label slice does not select subscribers, it selects **rows**. The last row gives the gain. Reaching the same 95 rows without building an index takes 2,658 comparisons, because each of the 1,329 labels is checked twice. In the label index two binary searches finish the job in **21 passes**: 126 times fewer comparisons and a one-line slice in place of a six-line loop. #### Row and Column Together ```python # axis.py — label-based access works on two axes: first the row, then the column. # The measure is the number of cells that enter the result. COLUMNS = ["subscriber_id", "region", "total_m3"] wanted = ["region", "total_m3"] part = summary[i:j] print(f"frame {len(summary)} rows x {len(COLUMNS)} columns = {len(summary) * len(COLUMNS)} cells") print(f"row slice first, then column selection: {len(part)} x {len(wanted)} = " f"{len(part) * len(wanted)} cells") print(f"column selection first, then row slice: {len(summary)} x {len(wanted)} = " f"{len(summary) * len(wanted)} cells in the intermediate table, then {len(part)} rows") single = [s["total_m3"] for s in part] print(f"if a single column is selected the result is a series: {len(single)} values, " f"total {round(sum(single), 2)} m3") ``` ``` frame 1329 rows x 3 columns = 3987 cells row slice first, then column selection: 95 x 2 = 190 cells column selection first, then row slice: 1329 x 2 = 2658 cells in the intermediate table, then 95 rows if a single column is selected the result is a series: 95 values, total 4348.3 m3 ``` Label-based access does not work on a single axis: the same expression names both the row and the column. This shortens things because it combines two separate selections in one step, but it slips an order decision in between. If the row slice is applied first, **190 cells** enter the result; if the column selection is applied first, an intermediate table of 2,658 cells is built and the row slice is taken from it. The result is the same, the table built is not. In a small frame this difference is immaterial; when the scale grows, the item measured by the `scale-and-performance` topic is exactly this. One detail also changes the **kind** of the result. If two columns are requested, what comes back is a frame; if a single column is requested, what comes back is a series. When the same spelling takes a one-element list of columns instead of a column name, the type of the result changes, and the code that follows has been written against either the series interface or the frame interface. How many columns the selection returns is therefore not only a number but a type decision. #### When the Order Breaks ```python # broken_order.py — the same label slice, in an index whose order has been broken. # What breaks the order is not an error but a legitimate sorting decision. sorted_by_total = sorted(summary, key=lambda s: -s["total_m3"]) broken = LabelIndex([s["subscriber_id"] for s in sorted_by_total]) print(f"frame sorted by total: is the label index sorted: " f"{'yes' if broken.is_sorted else 'no'}") bi, bj = label_slice(broken, START, STOP) print(f"the same slice {START}..{STOP}: {START} is at position {bi}, {STOP} is at position {bj - 1}, " f"slice {max(bj - bi, 0)} rows") absent = next(e for e in range(START, STOP) if e not in index.position) gi, gj = label_slice(index, START, absent) print(f"\nend {absent} not in the index: in the sorted index the slice returns {gj - gi} rows") try: label_slice(broken, START, absent) except KeyError as error: print(f"the same end, in the unsorted index: KeyError {error}") print(f"\npasses: scan without index {linear_passes}, binary search in the label index {label_passes}") print(f"code : linear slice 6 lines, label slice 1 line") print(f"rows : the same two ends return {label_rows}, {(j - 1) - i}, " f"{len(summary[START - 10001:STOP - 10001])} and {max(bj - bi, 0)} rows in four spellings") ``` ``` frame sorted by total: is the label index sorted: no the same slice 10100..10200: 10100 is at position 492, 10200 is at position 1108, slice 617 rows end 10122 not in the index: in the sorted index the slice returns 22 rows the same end, in the unsorted index: KeyError 10122 passes: scan without index 2658, binary search in the label index 21 code : linear slice 6 lines, label slice 1 line rows : the same two ends return 95, 94, 100 and 617 rows in four spellings ``` Sorting the frame by total consumption is a legitimate request and brings the biggest consumer to the top. The side effect it has is that the label index stops being sorted. In an unsorted index a label slice can no longer mean "the labels between 10,100 and 10,200", because such a range does not sit together anywhere in the table. Its meaning is forced to shift to this: the rows between the positions where the two ends are found. The result is **617 rows**, and the labels of most of these rows lie outside the range. The same expression, the same two ends, 617 rows instead of 95. There is no error message between the two, because both conform to the definition of the slice; what changed is the index. This is one of the clearest examples of the rule of the course: an abstraction shortens the operation but does not make visible the assumption it works on. An end that is not found produces two different behaviors in the two indexes, and here the difference reverses direction. Subscriber 10,122 is not in the index. In the sorted index this is not a problem: binary search finds the position where that label **would be** and a slice of 22 rows is returned. In the unsorted index that label has no position and the slice raises an error. So the abstraction does not stay silent here, it speaks; but the case in which it behaves loudly is the case of **missing information**, not of a wrong result. In the 617-row case, where it stays silent, it has produced an answer. #### Summary - Label-based access finds the row by its name, position-based access by its order; a label slice includes the last end, a position slice leaves it out. - The same two ends return 95, 94, 100 and 617 rows in four spellings; the one-row difference between the first two is the record at the boundary of the slice. - If the index is not continuous, mistaking the label for a position slides silently: because of 70 gaps the label 10,100 is at position 95, and the slice written by position returns the wrong 100 of 100 rows. - In a non-unique index a label slice selects rows, not subscribers: in the long table of 3,199 rows the same range brings 230 rows. - The gain of the index is read in pass count: a linear scan takes 2,658 comparisons while binary search reaches the same 95 rows in 21 passes; the hidden default is that a label slice assumes the index is sorted. #### Next Step Slicing gives a range, but what is wanted in analysis is not a range but a condition: "the readings whose consumption is above 25 m³". A conditional selection arises from a comparison, and a comparison assumes that every value in the dataset is comparable. It had been seen that the meter reading column has 111 empty cells and 80 cells with "n/a" written in; a value whose being below or above a threshold is unknown falls on which side of the filter. The next lesson runs the same condition in three separate spellings and counts how many rows each of the three returns. ### Filtering and Querying Source: https://academia.sh/en/courses/data-analysis-with-python/filtering-and-querying Readings with a meter reading above 120,000 are filtered with three separate spellings and three different row counts come out: the numeric comparison gives 1,835 rows and 36,435.83 m³, the negation of the same condition gives 2,026 rows and 40,356.71 m³, and the same comparison with the column read as text gives 1,915 rows and 38,035.93 m³. The difference is in the missing values: the negation takes in all 191 missing rows, the text comparison takes in the 80 'n/a' cells because a letter counts as greater than a digit. When two conditions are combined the difference grows, and two expressions written as equivalent in logic return 446 and 502 rows. A manual filter chain builds an intermediate table of 16,184 bytes in 5,034 passes, while the mask makes 6,398 passes and holds 3,399 bytes. Slicing gives a range, but what is wanted in analysis is not a range but a condition: "the readings whose meter reading is above 120,000". A conditional selection arises from a comparison, and a comparison carries a silent assumption: the assumption that every value in the dataset is comparable. In the previous lesson the meter reading column was seen to have 111 empty cells and 80 cells with "n/a" written in. A value whose being below or above a threshold is unknown falls on which side of the filter. The answer to this lesson is not a single sentence, because it changes with the spelling. The same question can be written in three separate forms, all three look correct, none of them raises an error, and the three return different row counts. That is the measure: the same condition, three spellings, three numbers. The difference always comes from the same place, the missing values, but in each spelling it falls to a different side. - **DF14.** A comparison made with a missing value returns **false**. This does not mean that the missing value is small; because the comparison cannot produce an answer, it has chosen an answer on its own. - **DF15.** Because the comparison maps "unknown" to "false", this mapping is **not preserved under negation**: in a row where `x > e` is false, `x <= e` is also false, and therefore `not(x <= e)` is true. The same row is outside in one spelling and inside in the other. - **DF16. A boolean mask** is an array that carries one truth value per row. The mask does not copy rows; conditions are combined on the mask and the result is produced once. - **DF17. Querying** is writing the filter condition as an expression. The form of the spelling does not change the result; what changes it is what the comparisons inside the condition do on a missing value. #### A Missing Value Cannot Be Compared ```python # dataset.py — MODEL: the long-format reading table of the previous lesson is # regenerated from the same seed. The dataset is fiction; the detail of the # generator is in the first lesson of the topic. import math M32 = 0xFFFFFFFF SEED, PERIODS = 20260218, ["2026-01", "2026-02", "2026-03"] REGION = [("north", 0.28, 21), ("south", 0.22, 17), ("east", 0.18, 26), ("west", 0.14, 14), ("central", 0.18, 23)] def rng(seed: int): s = ((seed ^ (seed >> 16)) * 2246822507) & M32 s = ((s ^ (s >> 13)) * 3266489909) & M32 s ^= s >> 16 def next_value() -> float: nonlocal s s = (s * 1664525 + 1013904223) & M32 return s / 4294967296 return next_value def pick(u: float, weights: list[float]) -> int: t = 0.0 for i, p in enumerate(weights): t += p if u < t: return i return len(weights) - 1 subscribers, dropped = [], [] for i in range(1400): r = rng(SEED + i) name, _, base = REGION[pick(r(), [b[1] for b in REGION])] r(); r() # the household and satisfaction draws are not used in this lesson (dropped if r() < 0.046 else subscribers).append( {"subscriber_id": 10001 + i, "region": name, "base": base}) valid = {k["subscriber_id"] for k in subscribers} pairs: dict[tuple[int, str], float] = {} for k in subscribers + dropped: r = rng(SEED + 7000 + k["subscriber_id"]) for d in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): t = 0.0 if r() < 0.038 else math.floor(k["base"] * math.exp( (r() + r() + r() - 1.5) * 0.62) * (1 - d * 0.05) * 100 + 0.5) / 100 if k["subscriber_id"] in valid: pairs.setdefault((k["subscriber_id"], PERIODS[d]), t) r() LONG = [{"subscriber_id": a, "period": d, "consumption_m3": t} for (a, d), t in pairs.items()] # the meter_reading column comes from the field form as in the previous lesson: some # of it is empty, and "n/a" has been written by hand in some of it. import sys cumulative: dict[int, float] = {} records = [] for o in LONG: no, t = o["subscriber_id"], o["consumption_m3"] cumulative[no] = cumulative.get(no, 0.0) + t u = rng(SEED + 9100 + no * 4 + PERIODS.index(o["period"]))() raw = "" if u < 0.033 else "n/a" if u < 0.055 else str( 100000 + (no - 10000) * 37 + int(cumulative[no])) records.append({"subscriber_id": no, "period": o["period"], "consumption_m3": t, "meter_reading": raw}) MISSING = None def to_number(h: str): """Converts text to a number; every cell that cannot be converted becomes MISSING.""" try: return int(h) except ValueError: return MISSING numeric = [to_number(k["meter_reading"]) for k in records] text_values = [k["meter_reading"] for k in records] print(f"{len(records)} rows: {sum(1 for v in numeric if v is not MISSING)} numeric meter readings, " f"{sum(1 for k in records if k['meter_reading'] == '')} empty, " f"{sum(1 for k in records if k['meter_reading'] == 'n/a')} 'n/a'") print(f"numeric meter reading range {min(v for v in numeric if v is not MISSING)}.." f"{max(v for v in numeric if v is not MISSING)}") ``` ``` 3199 rows: 3008 numeric meter readings, 111 empty, 80 'n/a' numeric meter reading range 100067..151812 ``` Of the three thousand one hundred ninety-nine rows, 3,008 carry a numeric meter reading and 191 do not. The loss of these 191 rows dates from the recording stage: the meter could not be read, or the form was filled in incompletely. Both come to the same result: for that row the meter reading is unknown. "Unknown" is not a value, it is an **information state**, and a comparison operator cannot express it. A comparison has to give one of two options, and what it chooses is "false". This choice is a defensible default, because it is reasonable for a filter not to select a row it knows nothing about. The problem is not in the default itself but in its being **unwritten**. The distinction has to be drawn once more here: a missing value and zero are not the same thing. There are readings with zero consumption in the dataset, and these are measured records; the meter was read, and no consumption came out. They enter the comparison, they fall to one side, and they carry no uncertainty. The 191 rows with a missing meter reading, however, were not measured. A dataset that keeps the two cases in the same column has to write down, while writing the filter, which of them is being compared. #### The Same Question, Three Spellings ```python # three_spellings.py — the same question in three forms: readings with a meter reading above 120000. THRESHOLD = 120000 def mask_numeric(v): """A missing value cannot enter the comparison, the result returns false.""" return v is not MISSING and v > THRESHOLD def mask_negation(v): """The negation of the same condition: in a missing row the inner comparison is also false.""" return not (v is not MISSING and v <= THRESHOLD) s1 = [mask_numeric(v) for v in numeric] s2 = [mask_negation(v) for v in numeric] s3 = [h > str(THRESHOLD) for h in text_values] WIDTH = [42, 8, 12] show = lambda h: "".join(str(v).rjust(WIDTH[k]) if k else str(v).ljust(WIDTH[0]) for k, v in enumerate(h)) total_of = lambda m: round(sum(k["consumption_m3"] for k, b in zip(records, m) if b), 2) print(show(["spelling", "rows", "total m3"])) print(show([f"numeric meter_reading > {THRESHOLD}", sum(s1), total_of(s1)])) print(show([f"negation not(meter_reading <= {THRESHOLD})", sum(s2), total_of(s2)])) print(show([f"text meter_reading > '{THRESHOLD}'", sum(s3), total_of(s3)])) print(f"\nthe second spelling has {sum(s2) - sum(s1)} rows more than the first: all of the missing") print(f"the third spelling has {sum(s3) - sum(s1)} rows more than the first: cells written as 'n/a', " f"because text comparison counts a letter greater than a digit") ``` ``` spelling rows total m3 numeric meter_reading > 120000 1835 36435.83 negation not(meter_reading <= 120000) 2026 40356.71 text meter_reading > '120000' 1915 38035.93 the second spelling has 191 rows more than the first: all of the missing the third spelling has 80 rows more than the first: cells written as 'n/a', because text comparison counts a letter greater than a digit ``` Three rows ask the same question and give three separate numbers: 1,835, 2,026 and 1,915 rows. The consumption total that enters the report is also three separate values, and between the lowest and the highest there are 3,920.88 m³. The second spelling is the most insidious one, because negation looks natural when read: "those not below the threshold" is taken to be the same as "those that exceed the threshold". It is not. In a row with a missing meter reading the comparison `meter_reading <= 120000` returns false, because that is the answer chosen for a value that cannot be compared; the negation of that false is true. So 191 rows, although nothing at all is known about them, enter the set "those that exceed the threshold", and 3,920.88 m³ of consumption is added to the reported total. The third spelling is a direct continuation of the previous lesson. The meter reading column had been read as text because of 80 "n/a" entries; in a column read as text the `>` operator still works, but it works in dictionary order. Because all the numeric meter readings have six digits, the text comparison gives the same result as the numeric comparison on them; the only place they part is the `'n/a'` cells, and the letter "n" is greater than the digit "1". The result is exactly 80 rows more, and none of these rows has a meter reading. #### Combined Condition and the Cost of the Filter ```python # combined.py — when two conditions are combined the effect of the missing multiplies: # two expressions written as equivalent in logic do not give the same row count. def and_both(v, t): return (v is not MISSING and v > THRESHOLD) and t > 25.0 def not_or(v, t): return not ((v is not MISSING and v <= THRESHOLD) or t <= 25.0) b1 = [and_both(v, k["consumption_m3"]) for v, k in zip(numeric, records)] b2 = [not_or(v, k["consumption_m3"]) for v, k in zip(numeric, records)] print(f"(meter_reading > {THRESHOLD}) and (consumption > 25) : {sum(b1)} rows, {total_of(b1)} m3") print(f"not((meter_reading <= {THRESHOLD}) or (consumption <= 25)) : {sum(b2)} rows, {total_of(b2)} m3") print(f"the two expressions are equivalent in logic, the result differs by {sum(b2) - sum(b1)} rows") def manual_chain(): """For every condition the dataset is scanned from the start and the intermediate table is copied.""" middle = [(k, v) for k, v in zip(records, numeric) if v is not MISSING and v > THRESHOLD] final = [(k, v) for k, v in middle if k["consumption_m3"] > 25.0] return middle, final, len(records) + len(middle) middle, final, manual_passes = manual_chain() mask = bytearray(b1) print(f"\nmanual chain: {len(final)} rows, {manual_passes} passes, intermediate table " f"{sys.getsizeof([k for k, _ in middle])} bytes") print(f"mask : {sum(mask)} rows, {2 * len(records)} passes, mask " f"{sys.getsizeof(mask)} bytes, 1 line of code") print(f"the mask does not win in pass count, it wins in the intermediate table: " f"{sys.getsizeof([k for k, _ in middle]) - sys.getsizeof(mask)} bytes") ``` ``` (meter_reading > 120000) and (consumption > 25) : 446 rows, 13906.21 m3 not((meter_reading <= 120000) or (consumption <= 25)) : 502 rows, 15675.3 m3 the two expressions are equivalent in logic, the result differs by 56 rows manual chain: 446 rows, 5034 passes, intermediate table 16184 bytes mask : 446 rows, 6398 passes, mask 3399 bytes, 1 line of code the mask does not win in pass count, it wins in the intermediate table: 12785 bytes ``` When two conditions are combined the difference does not shrink, it changes direction. `A and B` and `not(not A or not B)` are the same thing in two-valued logic and can be exchanged in a textbook. Here they cannot be exchanged, because the atomic comparison is not two-valued: it squeezes three states, that is true, false and unknown, into two values and maps "unknown" to "false". This mapping is not preserved under negation. The result is 502 rows instead of 446, and the reported total is 15,675.30 m³ instead of 13,906.21 m³. The cost of the filter is in the second output, and here the gain of the abstraction is not where it is expected. The manually written chain applies every condition in a separate pass and **copies** a table in between: 5,034 passes, an intermediate table of 16,184 bytes. The mask evaluates every condition on all rows, makes 6,398 passes, that is, it takes more passes; where it gains is memory. The mask is 3,399 bytes and no row is copied; the difference is 12,785 bytes and it is preserved proportionally as the dataset grows. That querying writes the condition as an expression does not change this table either. Whether the condition is written as a string or the masks are combined with an operator, the result comes out of the same three items: which comparisons there are, what they return on a missing value, and how they are combined. The form of the spelling is a readability decision; what determines the row count is the behavior of the comparison on a missing value. #### The Third Set of the Filter ```python # three_sets.py — the defensible spelling: the filter produces not two but three sets # and the sum of the three gives the whole dataset. above = sum(1 for v in numeric if v is not MISSING and v > THRESHOLD) below = sum(1 for v in numeric if v is not MISSING and v <= THRESHOLD) unknown = sum(1 for v in numeric if v is MISSING) print(f"above the threshold {above}, below {below}, unknown {unknown}, " f"total {above + below + unknown}") print(f"unknown share: {round(100 * unknown / len(records), 2)} percent; " f"if all of it is written onto the above group, {above} becomes {above + unknown}, " f"that is a {round(100 * unknown / above, 1)} percent swing") ``` ``` above the threshold 1835, below 1173, unknown 191, total 3199 unknown share: 5.97 percent; if all of it is written onto the above group, 1835 becomes 2026, that is a 10.4 percent swing ``` When a filter spelling reports these three numbers separately, all the uncertainty of the lesson disappears. 1,835 rows are above the threshold, 1,173 rows are below it, no decision could be made about 191 rows, and the sum of the three is the whole dataset. This sum check cannot be made in any of the spellings that report a single number: which number is right between `1,835` and `2,026` is determined not by looking at the dataset but only by deciding where the missing are to be written. The size of the uncertainty can also be read here. The unknown rows are 5.97% of the dataset, but their effect on the reported number is 10.4%, because the missing are proportioned not to the whole dataset but to the part that passes the filter. The effect on the report of a missing rate that looks small depends on where the threshold is put, and it grows as the threshold narrows. That is why the missing rate alone is not a sufficient check; what has to be measured is the ratio of the missing to the **filter result**. #### Summary - A comparison made with a missing value returns false; this does not mean that the missing value is small, it means that the comparison chose an answer where it could not produce one. - The same condition gives three row counts in three spellings: numeric 1,835, negation 2,026, text comparison 1,915; the reported total moves between 36,435.83 and 40,356.71 m³. - The negation takes in all 191 missing rows; the text comparison takes in the 80 "n/a" cells, because in dictionary order a letter is greater than a digit. - In a combined condition two expressions written as equivalent in logic return 446 and 502 rows, because the comparison maps "unknown" to "false" and this mapping is not preserved under negation. - The gain of the mask is in memory, not in passes: the manual chain builds an intermediate table of 16,184 bytes in 5,034 passes while the mask makes 6,398 passes and holds 3,399 bytes. #### Next Step A filter selects rows, but a report does not ask for individual rows; it asks for a summary by breakdown. The question "average consumption per region" first splits the dataset, applies a calculation to each part, and combines the results in a single table. This three-step job fits into a single call, and that call carries two separate decisions: which group the rows that carry a missing value in the group key enter, and whether the aggregation calculation includes the missing values in the denominator. The next lesson measures these two decisions separately and counts how many rows are lost without appearing in any group. ### Grouping and Aggregation Source: https://academia.sh/en/courses/data-analysis-with-python/grouping-and-aggregation The split, apply and combine steps fit into a single call, and that call carries two decisions at once: the 160 rows whose group key is empty appear in no group under the default grouping, and the total of the five-row summary comes out as 60,025.56 m³ instead of 63,060.63 m³, that is, 4.81% of the dataset drops out without producing any warning. When the missing key is kept, the summary has 6 rows and the dropped part becomes visible as 3,035.07 m³. The second decision is the denominator of the aggregation: the north group holds 878 rows but the meter reading column has 831 filled values, and the average comes out as 126,247.9 or 119,489.8 depending on the denominator. A separate loop per measure costs 15,995 key lookups, while a single split feeds five measures with 3,199 lookups. A filter selects rows, but a report does not ask for individual rows; it asks for a summary by breakdown. The question "average meter reading per region" is a job of three steps: the dataset is **split** by the key, a calculation is **applied** to each part, and the results are **combined** in a single table. The grouping lesson of the Data Collection and Preparation course ran these three steps with hand-written loops and measured that the grouping decision determines the number of rows that come out; that calculation is the input of this lesson and is not repeated. What changes here is that the three steps fit into a single call. That call carries two separate decisions at once, and neither of them is written: which group the row with an empty group key enters, and what the aggregation calculation takes as its denominator. The measure of this lesson is how many rows and how many cubic meters these two decisions move. - **DF18. Split–apply–combine** is the three steps of grouping. The split sets up one bucket per key, the apply step makes a calculation in each bucket, and the combine step produces one row per bucket. The number of rows that come out is **the number of distinct values of the key**, not the number of rows of the dataset. - **DF19.** A row whose group key is missing does not belong to any bucket. The default behavior is to **drop** it; collecting the missing value in a bucket of its own is also a legitimate decision, and the difference between the two decisions is a row count and a total. - **DF20. The denominator of the aggregation** is a separate decision. An average can be calculated by counting the **filled** values of the aggregated column or by counting **all** the rows of the group. Two denominators give two averages, and the `n` column in the report has to say which one it writes. - **DF21.** If the split step is done once, the same buckets are shared by many measures. The measure of the gain is **the number of key lookups**, not the number of value reads. #### A Row With No Key ```python # dataset.py — MODEL: the long-format reading table of the previous lesson is # regenerated from the same seed. The dataset is fiction; the detail of the # generator is in the first lesson of the topic. import math M32 = 0xFFFFFFFF SEED, PERIODS = 20260218, ["2026-01", "2026-02", "2026-03"] REGION = [("north", 0.28, 21), ("south", 0.22, 17), ("east", 0.18, 26), ("west", 0.14, 14), ("central", 0.18, 23)] def rng(seed: int): s = ((seed ^ (seed >> 16)) * 2246822507) & M32 s = ((s ^ (s >> 13)) * 3266489909) & M32 s ^= s >> 16 def next_value() -> float: nonlocal s s = (s * 1664525 + 1013904223) & M32 return s / 4294967296 return next_value def pick(u: float, weights: list[float]) -> int: t = 0.0 for i, p in enumerate(weights): t += p if u < t: return i return len(weights) - 1 subscribers, dropped = [], [] for i in range(1400): r = rng(SEED + i) name, _, base = REGION[pick(r(), [b[1] for b in REGION])] r(); r() # the household and satisfaction draws are not used in this lesson (dropped if r() < 0.046 else subscribers).append( {"subscriber_id": 10001 + i, "region": name, "base": base}) valid = {k["subscriber_id"] for k in subscribers} pairs: dict[tuple[int, str], float] = {} for k in subscribers + dropped: r = rng(SEED + 7000 + k["subscriber_id"]) for d in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): t = 0.0 if r() < 0.038 else math.floor(k["base"] * math.exp( (r() + r() + r() - 1.5) * 0.62) * (1 - d * 0.05) * 100 + 0.5) / 100 if k["subscriber_id"] in valid: pairs.setdefault((k["subscriber_id"], PERIODS[d]), t) r() LONG = [{"subscriber_id": a, "period": d, "consumption_m3": t} for (a, d), t in pairs.items()] # The group key is the subscriber's region; for some subscribers the address record # did not match, so the region is empty. The meter_reading column carries missing # values as in the previous lesson. MISSING = None def region_cell(no: int, name: str): return MISSING if rng(SEED + 61000 + no)() < 0.048 else name region_of = {k["subscriber_id"]: region_cell(k["subscriber_id"], k["region"]) for k in subscribers} cumulative: dict[int, float] = {} records = [] for o in LONG: no, t = o["subscriber_id"], o["consumption_m3"] cumulative[no] = cumulative.get(no, 0.0) + t u = rng(SEED + 9100 + no * 4 + PERIODS.index(o["period"]))() meter_reading = MISSING if u < 0.055 else 100000 + (no - 10000) * 37 + int(cumulative[no]) records.append({"subscriber_id": no, "period": o["period"], "consumption_m3": t, "meter_reading": meter_reading, "region": region_of[no]}) print(f"{len(records)} rows; the group key is empty in " f"{sum(1 for k in records if k['region'] is MISSING)} rows, the column to aggregate " f"is empty in {sum(1 for k in records if k['meter_reading'] is MISSING)} rows") print(f"the true total of the dataset is {round(sum(k['consumption_m3'] for k in records), 2)} m3") ``` ``` 3199 rows; the group key is empty in 160 rows, the column to aggregate is empty in 191 rows the true total of the dataset is 63060.63 m3 ``` The two missing patterns sit in the same dataset and are independent of each other. In 160 rows the **group key** is empty: the address record of the subscriber did not match a region. In 191 rows the **column to aggregate** is empty: the meter could not be read and the meter reading was not recorded. Both of the missing patterns pass through the same grouping call, and the call treats each of them differently. The true total of the dataset is 63,060.63 m³, and this number is the audit anchor of this lesson. As long as the total of a summary table can be compared with it, what the grouping lost can be read; when it is not compared, the summary looks consistent within itself and says nothing. #### Split, Apply, Combine ```python # split_apply_combine.py — grouping is three steps: split, apply, combine. # It fits into a single call and carries two decisions at once. def split(rows, key: str, drop_missing=True): """Split step: rows are distributed into buckets by the key.""" bucket, lost = {}, 0 for k in rows: if k[key] is MISSING and drop_missing: lost += 1 continue bucket.setdefault(k[key], []).append(k) return bucket, lost def apply(bucket, column: str): """Apply step: in each bucket a count, a total and an average with two separate denominators.""" out = [] for name, members in bucket.items(): filled = [k[column] for k in members if k[column] is not MISSING] out.append({ "group": "(missing)" if name is MISSING else name, "rows": len(members), "filled": len(filled), "total_m3": round(sum(k["consumption_m3"] for k in members), 2), "avg_filled": round(sum(filled) / len(filled), 1) if filled else 0.0, "avg_rows": round(sum(filled) / len(members), 1)}) return sorted(out, key=lambda c: c["group"]) WIDTH = [12, 8, 8, 12, 12, 12] show = lambda h: "".join(str(v).rjust(WIDTH[k]) if k else str(v).ljust(WIDTH[0]) for k, v in enumerate(h)) bucket, lost = split(records, "region") summary = apply(bucket, "meter_reading") print(show(["group", "rows", "filled", "total m3", "avg/filled", "avg/rows"])) for c in summary: print(show([c["group"], c["rows"], c["filled"], c["total_m3"], c["avg_filled"], c["avg_rows"]])) print(show(["combined", sum(c["rows"] for c in summary), sum(c["filled"] for c in summary), round(sum(c["total_m3"] for c in summary), 2), "", ""])) print(f"\nthe default grouping produced a summary of {len(summary)} rows and showed {lost} rows " f"in no group") ``` ``` group rows filled total m3 avg/filled avg/rows central 570 533 12796.0 125262.2 117131.2 east 533 508 13203.17 124854.4 118998.2 north 878 831 17749.3 126247.9 119489.8 south 614 574 10153.66 125303.5 117140.4 west 444 415 6123.43 126006.5 117776.3 combined 3039 2861 60025.56 the default grouping produced a summary of 5 rows and showed 160 rows in no group ``` A summary of five rows came out, and looked at on its own it seems flawless. The region names are complete, the counts are plausible, the averages are close to each other. The flaw shows only in the combined row: 3,039 rows have been counted, whereas the dataset is 3,199 rows; the total is 60,025.56 m³, whereas the dataset is 63,060.63 m³. The 160 rows in between are reported nowhere. There is no error message, no warning, not even an empty row; because no bucket was opened for those rows, there was no place in the table where they could appear. This differs in structure from the silent loss measured in the previous lesson: there the filter made its decision by a condition, here what makes the decision is **the non-existence of the bucket**. The last two columns of the table show the second decision. The north group has 878 rows but the meter reading column has 831 filled values. If the average counts the filled values it comes out as 126,247.9, if it counts all the rows of the group it comes out as 119,489.8. Both numbers are an average and there is a difference of 6,758.1 between them. #### Two Decisions, Two Numbers ```python # decision.py — the two decisions are measured separately: the missing in the group key # and the denominator of the aggregation. bucket2, lost2 = split(records, "region", drop_missing=False) summary2 = apply(bucket2, "meter_reading") miss = next(c for c in summary2 if c["group"] == "(missing)") total1 = round(sum(c["total_m3"] for c in summary), 2) total2 = round(sum(c["total_m3"] for c in summary2), 2) print(f"if the missing key is kept the summary has {len(summary2)} rows instead of {len(summary)}; " f"rows in the buckets {sum(c['rows'] for c in summary2)}, total {total2} m3") print(f"the (missing) group carries {miss['rows']} rows and {miss['total_m3']} m3; the default " f"grouping drops it and the total stays {total1} m3") print(f"share lost: {round(100 * (total2 - total1) / total2, 2)} percent, without producing any warning") north = next(c for c in summary if c["group"] == "north") print(f"\nin the north group there are two separate counts: {north['rows']} rows and {north['filled']} " f"filled meter readings; the average is {north['avg_filled']} or {north['avg_rows']}") print(f"a report that writes 'n' on the same line has to say which one it wrote") MEASURES = ["count", "sum", "mean", "max", "min"] print(f"\npasses: a separate loop per measure {len(MEASURES) * len(records)} key lookups, " f"a single split {len(records)} key lookups and {len(MEASURES) * len(records)} value reads") print(f"code : a separate loop per measure {len(MEASURES)} blocks, a single split " f"{len(MEASURES)} applications 1 call") ``` ``` if the missing key is kept the summary has 6 rows instead of 5; rows in the buckets 3199, total 63060.63 m3 the (missing) group carries 160 rows and 3035.07 m3; the default grouping drops it and the total stays 60025.56 m3 share lost: 4.81 percent, without producing any warning in the north group there are two separate counts: 878 rows and 831 filled meter readings; the average is 126247.9 or 119489.8 a report that writes 'n' on the same line has to say which one it wrote passes: a separate loop per measure 15995 key lookups, a single split 3199 key lookups and 15995 value reads code : a separate loop per measure 5 blocks, a single split 5 applications 1 call ``` The change of a single option takes the summary from five rows to six, and the sixth row carries the whole of the dropped part: 160 rows, 3,035.07 m³, 4.81% of the dataset. The total of the six-row summary is equal to the true total of the dataset, and this equality is the only check needed for a grouping to be verifiable. Both decisions have a defensible side. Putting the 160 rows whose region is unknown into the region breakdown is meaningless, because it is not known which region they belong to. But not writing them into the table at all means that the region totals do not give the dataset total and that this is not visible. The difference in between is not a preference but a **reporting debt**: as long as the dropped rows are counted and written down, the report remains readable whichever decision is made. The decision on the denominator has the same structure. The average meter reading of the north group is 126,247.9 as the average of 831 measured values; in the calculation that also adds the 47 unmeasured rows to the count it falls to 119,489.8. The second is the same as saying "the meter reading of the unreadable meters is zero", and this is an assumption. When `n` is not written next to the average, the two numbers cannot be told apart. The gain side is in the last two lines. Writing five separate loops for five measures costs 15,995 key lookups, because every measure scans the dataset from the start and rebuilds the buckets. Split–apply–combine does the split step once: 3,199 key lookups, and then the five measures are calculated on the same buckets with 15,995 value reads. The key lookups fall to one fifth, the code falls from five blocks to a single call, and it is exactly this shortening that makes the two decisions invisible. #### The Summary Is Also a Frame ```python # share.py — the summary is also a frame and its label index is the group key. # The share calculation gives two separate tables depending on which total it takes as the denominator. W2 = [12, 12, 14, 14] show2 = lambda h: "".join(str(v).rjust(W2[k]) if k else str(v).ljust(W2[0]) for k, v in enumerate(h)) print(show2(["group", "total m3", "share/summary", "share/dataset"])) for c in summary: print(show2([c["group"], c["total_m3"], round(100 * c["total_m3"] / total1, 2), round(100 * c["total_m3"] / total2, 2)])) print(show2(["combined", total1, round(sum(100 * c["total_m3"] / total1 for c in summary), 2), round(sum(100 * c["total_m3"] / total2 for c in summary), 2)])) print(f"\nthe table that takes the summary total as the denominator closes at 100 percent and hides the " f"{lost} dropped rows; the table that takes the dataset total as the denominator leaves " f"{round(100 - sum(100 * c['total_m3'] / total2 for c in summary), 2)} percent open") ``` ``` group total m3 share/summary share/dataset central 12796.0 21.32 20.29 east 13203.17 22.0 20.94 north 17749.3 29.57 28.15 south 10153.66 16.92 16.1 west 6123.43 10.2 9.71 combined 60025.56 100.0 95.19 the table that takes the summary total as the denominator closes at 100 percent and hides the 160 dropped rows; the table that takes the dataset total as the denominator leaves 4.81 percent open ``` The output of the aggregation is a new frame and its label index is the group key. This applies the rule of the first lesson of the topic to the result of the grouping as well: the next operation is aligned over this index, and a group that is not in the index silently becomes missing. The share calculation is the most common form of this, because choosing the denominator is exactly choosing an index. The two share columns come out of the same five numbers and they differ only in the denominator. The column that takes the summary total as the denominator closes at 100%; someone looking at the table sees no shortfall, because a percentage table closing at 100 is taken as a sign of correctness. The column that takes the dataset total as the denominator closes at 95.19% and the remaining **4.81%** stands in the table itself as a question. That the second table closes short is not a defect, it is a finding. The table that closes at 100% is more dangerous, because it can hide the existence of the 160 dropped rows without breaking the table: the closing check has passed, and what the check measured is not that there is no loss but that the shares are consistent within themselves. #### Summary - Split–apply–combine is three steps and the number of rows that come out is the number of distinct values of the key; doing the split step once brings the key lookups down from 15,995 to 3,199. - The 160 rows whose group key is empty enter no bucket under the default grouping and there is no place in the table where they could appear; the summary counts 3,039 rows, the dataset is 3,199 rows. - The dropped part is 3,035.07 m³, that is, 4.81% of the dataset; when the missing key is kept the summary has 6 rows and its total equals the true total of the dataset. - The denominator of the aggregation is the second decision: the north group holds 878 rows and 831 filled meter readings, and the average comes out as 126,247.9 or 119,489.8 depending on the denominator. - The only mandatory check of a summary table is that the group totals equal the dataset total; when the equality is not established, the summary looks consistent within itself and does not report the dropped rows. #### Next Step Grouping stays within a single table. Most questions of analysis, however, ask for more than one table: the subscriber information has to come next to the reading table, and the field visits next to the subscriber information. In the Data Collection and Preparation course this job was done with a hand-written matching loop and the row calculation was kept item by item. In a dataframe interface the same job is a single call, and that call chooses a join type. The next lesson applies the inner, outer and cross join to the same two tables and counts how many rows each of them produces and how rows multiply where the key is not unique. ### Join Operations Source: https://academia.sh/en/courses/data-analysis-with-python/join-operations The reading table of 3,199 rows and the visit table of 818 rows are joined on the same key with four modes and four different row counts come out: inner join 1,965, left outer 3,820, full outer 3,856, cross join 2,616,782 rows. The inner join drops 1,855 readings and opens the remaining 1,344 rows into 1,965 rows; this fan-out of 621 rows moves the total from 26,450.81 m³ to 38,701.46 m³, that is, it produces 12,250.65 m³ of excess counting. Although the relationship of the key is many-to-many, it is verified nowhere: the key of 468 rows on the left side corresponds to more than one row on the right, and these 468 rows silently open into 1,089 rows. A hand-written nested loop makes 2,616,782 comparisons, while an indexed join finishes in 4,017 passes. Grouping stays within a single table. Most questions of analysis, however, ask for more than one table: the field visits have to come next to the reading table. The join lesson of the Data Collection and Preparation course did this job with a hand-written matching loop and kept the row calculation item by item; 3,471 readings and 1,329 subscribers gave 3,302 rows, 169 readings did not match, and 69 subscribers never appeared in the output. That calculation is the input of this lesson and is not repeated. In a dataframe interface the same job is a single call, and that call chooses a **join type**. The measure of this lesson is this: the same two tables, the same key, four modes and four different row counts. A second number comes next to it: when the key grain is not verified, how many rows multiply and how much the total inflates. - **DF22. The join type** is the decision that determines the row count of the output. The **inner** join keeps only the matching rows, the **left outer** also keeps the unmatched left rows, the **full outer** also keeps the unmatched ones on both sides. The **cross** join never reads the key and produces all the pairs of the two tables. - **DF23. Key grain** is how many rows a key value corresponds to on each side. The relationship can be one-to-one, one-to-many or many-to-many, and the join does this **without verifying** it. - **DF24. Fan-out** is a left row being repeated as many times as it has matches on the right. Fan-out is not an error; the error is taking a total over a multiplied table. - **DF25.** Even when the output row count is lower than the input, no elimination may have been made: within the same operation some rows may drop while others multiply, and the net change hides both. #### Two Tables and Three Sets ```python # dataset.py — MODEL: the long-format reading table of the previous lesson is # regenerated from the same seed. The dataset is fiction; the detail of the # generator is in the first lesson of the topic. import math M32 = 0xFFFFFFFF SEED, PERIODS = 20260218, ["2026-01", "2026-02", "2026-03"] REGION = [("north", 0.28, 21), ("south", 0.22, 17), ("east", 0.18, 26), ("west", 0.14, 14), ("central", 0.18, 23)] def rng(seed: int): s = ((seed ^ (seed >> 16)) * 2246822507) & M32 s = ((s ^ (s >> 13)) * 3266489909) & M32 s ^= s >> 16 def next_value() -> float: nonlocal s s = (s * 1664525 + 1013904223) & M32 return s / 4294967296 return next_value def pick(u: float, weights: list[float]) -> int: t = 0.0 for i, p in enumerate(weights): t += p if u < t: return i return len(weights) - 1 subscribers, dropped = [], [] for i in range(1400): r = rng(SEED + i) name, _, base = REGION[pick(r(), [b[1] for b in REGION])] r(); r() # the household and satisfaction draws are not used in this lesson (dropped if r() < 0.046 else subscribers).append( {"subscriber_id": 10001 + i, "region": name, "base": base}) valid = {k["subscriber_id"] for k in subscribers} pairs: dict[tuple[int, str], float] = {} for k in subscribers + dropped: r = rng(SEED + 7000 + k["subscriber_id"]) for d in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): t = 0.0 if r() < 0.038 else math.floor(k["base"] * math.exp( (r() + r() + r() - 1.5) * 0.62) * (1 - d * 0.05) * 100 + 0.5) / 100 if k["subscriber_id"] in valid: pairs.setdefault((k["subscriber_id"], PERIODS[d]), t) r() LONG = [{"subscriber_id": a, "period": d, "consumption_m3": t} for (a, d), t in pairs.items()] # Field visits table: one row is one visit and it is not tied to a period. visits = [] for k in subscribers: r = rng(SEED + 31000 + k["subscriber_id"]) for z in range(pick(r(), [0.58, 0.27, 0.11, 0.04])): visits.append({"subscriber_id": k["subscriber_id"], "outcome": ["positive", "repeat", "unreachable"][ pick(r(), [0.60, 0.28, 0.12])]}) readers = {o["subscriber_id"] for o in LONG} visited = {z["subscriber_id"] for z in visits} print(f"reading table {len(LONG)} rows, {len(readers)} subscribers") print(f"visit table {len(visits)} rows, {len(visited)} subscribers") print(f"{len(readers & visited)} subscribers on both sides; only in readings " f"{len(readers - visited)}, only in visits {len(visited - readers)}") ``` ``` reading table 3199 rows, 1260 subscribers visit table 818 rows, 559 subscribers 534 subscribers on both sides; only in readings 726, only in visits 25 ``` Three numbers are known before the join, and these three numbers explain in advance all the row counts that will come out. 534 subscribers are in both tables, 726 subscribers only in the reading table, 25 subscribers only in the visit table. The sum of these three sets is 1,285, and choosing a join type is in fact choosing which of these three sets will be represented in the output. One more detail directly affects the row count: the two tables do not carry the same **grain**. In the reading table a row is one period of a subscriber, in the visit table a row is one visit and it is not tied to a period. When only the subscriber number is used as the key, three readings and two visits of a subscriber produce six rows. This is a property that can be read from the tables themselves and it is known before the join call. #### The Same Two Tables, Four Modes ```python # join_tables.py — the same two tables with four join types. The mode is the # decision that determines the row count of the output. PASSES = {"count": 0} def join_tables(left, right, field: str, mode: str = "inner"): """Puts the two sides next to each other on the key. mode: inner, left_outer, full_outer, cross.""" if mode == "cross": PASSES["count"] += len(left) * len(right) return len(left) * len(right), 0, 0 index: dict = {} for s in right: index.setdefault(s[field], []).append(s) PASSES["count"] += len(right) + len(left) produced, empty_left, matched = 0, 0, set() for l in left: e = index.get(l[field]) if not e: empty_left += 1 produced += 1 if mode in ("left_outer", "full_outer") else 0 continue produced += len(e) matched.add(l[field]) empty_right = len({s[field] for s in right} - matched) if mode == "full_outer": produced += sum(1 for s in right if s[field] not in matched) return produced, empty_left, empty_right WIDTH = [22, 12, 12, 12, 12] show = lambda h: "".join(str(v).rjust(WIDTH[k]) if k else str(v).ljust(WIDTH[0]) for k, v in enumerate(h)) print(show(["join type", "left in", "right in", "output", "unmatched"])) result = {} for mode, name in [("inner", "inner"), ("left_outer", "left outer"), ("full_outer", "full outer"), ("cross", "cross")]: PASSES["count"] = 0 produced, el, er = join_tables(LONG, visits, "subscriber_id", mode) result[mode] = (produced, el, er, PASSES["count"]) print(show([name, len(LONG), len(visits), produced, el if mode != "cross" else 0])) print(f"\ninner join drops {result['inner'][1]} of {len(LONG)} readings and produces " f"{result['inner'][0]} rows") print(f"cross join never reads the key: {len(LONG)} x {len(visits)} = " f"{result['cross'][0]} rows") ``` ``` join type left in right in output unmatched inner 3199 818 1965 1855 left outer 3199 818 3820 1855 full outer 3199 818 3856 1855 cross 3199 818 2616782 0 inner join drops 1855 of 3199 readings and produces 1965 rows cross join never reads the key: 3199 x 818 = 2616782 rows ``` Four modes, the same two tables, four different row counts: 1,965, 3,820, 3,856 and 2,616,782. The ratio between the smallest and the largest is 1,332 times, and the only thing that makes the choice is a mode name that is written or not written in the call. The number of the inner join is interesting in two directions at once. The 1,965 rows that come out are fewer than what went in, and at first sight this looks like an elimination. It is not: 1,855 readings have dropped, and the remaining 1,344 readings have opened into 1,965 rows. Because the net change is negative, the fan-out is not visible inside the number. This is the same behavior that was measured in the Data Collection and Preparation course; what is new here is that the whole operation fits into a single call and that none of the numbers 1,855 and 621 comes back from the call. The left outer join gives 3,820 rows and keeps all the rows of the reading table: 1,965 matched rows and 1,855 with an empty visit side. The full outer, with 3,856 rows, also adds the 36 visit rows of the 25 subscribers who are only in the visit table. The cross join, on the other hand, never reads the key; the 2,616,782 rows it produces are not the answer to a question, they are all the pairs of the two tables. #### Fan-Out and the Key Grain ```python # fan_out.py — if the uniqueness of the key is not verified, the join counts the same # row more than once. The measure is the multiplied rows and the inflated total. counts: dict[int, int] = {} for z in visits: counts[z["subscriber_id"]] = counts.get(z["subscriber_id"], 0) + 1 matched = [o for o in LONG if o["subscriber_id"] in counts] produced = sum(counts[o["subscriber_id"]] for o in matched) total_of = lambda a: round(sum(x["consumption_m3"] for x in a), 2) inflated = round(sum(o["consumption_m3"] * counts[o["subscriber_id"]] for o in matched), 2) print(f"dataset {len(LONG)} rows, {total_of(LONG)} m3") print(f"matched {len(matched)} rows, {total_of(matched)} m3") print(f"inner join output {produced} rows, {inflated} m3; " f"{produced - len(matched)} rows of fan-out, {round(inflated - total_of(matched), 2)} m3 excess counting") def relationship(left, right, field: str) -> str: """The key grain: is it unique on both sides, or is it many on one side.""" left_unique = len({s[field] for s in left}) == len(left) right_unique = len({s[field] for s in right}) == len(right) return ("one-to-one" if left_unique and right_unique else "one-to-many" if right_unique or left_unique else "many-to-many") many = sum(1 for o in matched if counts[o["subscriber_id"]] > 1) print(f"\nkey grain: {relationship(LONG, visits, 'subscriber_id')}; on the left side " f"the key of {many} rows corresponds to more than one row on the right") print(f"when no verification is made, these {many} rows silently open into {produced - len(matched) + many} " f"rows") print(f"\npasses: hand-written nested loop {len(LONG) * len(visits)}, indexed join " f"{len(LONG) + len(visits)}") print(f"rows : the same two tables produce {result['inner'][0]}, {result['left_outer'][0]}, " f"{result['full_outer'][0]} and {result['cross'][0]} rows in four modes") ``` ``` dataset 3199 rows, 63060.63 m3 matched 1344 rows, 26450.81 m3 inner join output 1965 rows, 38701.46 m3; 621 rows of fan-out, 12250.65 m3 excess counting key grain: many-to-many; on the left side the key of 468 rows corresponds to more than one row on the right when no verification is made, these 468 rows silently open into 1089 rows passes: hand-written nested loop 2616782, indexed join 4017 rows : the same two tables produce 1965, 3820, 3856 and 2616782 rows in four modes ``` The price of the fan-out is read on the second line. The total of the 1,344 matched readings is 26,450.81 m³; the total of the same readings in the join output is 38,701.46 m³. The **12,250.65 m³** in between is not new consumption, it is the same readings being counted a second and a third time. The January reading of a subscriber has entered the total twice because that subscriber has two visits. The danger of this number is not in its size but in its **plausibility**. 38,701.46 m³ is smaller than the true total of the dataset, that is, 63,060.63 m³; if an audit looked at "the total does not exceed the dataset total", it would pass this table. Yet the table carries a total that is both undercounted and overcounted: 1,855 readings have not been counted at all, 468 readings have been counted more than once. The last two lines put the gain and the hidden default side by side. The gain is clear: while a hand-written nested loop makes 2,616,782 comparisons, a join that builds a key index on the right side and walks the left side once finishes in 4,017 passes, which is 651 times fewer. The hidden default, on the other hand, is that the relationship is **many-to-many** and that this is verified nowhere. The join call does not ask what the relationship is; had it been asked, it would have been known before the call that 468 rows would open into 1,089 rows. #### Fixing the Key Grain ```python # grain_fix.py — what prevents the fan-out is not the join type but bringing the right side # to the grain: aggregate first, then join. visit_summary = [{"subscriber_id": no, "visit_count": n} for no, n in counts.items()] unique = len({z["subscriber_id"] for z in visit_summary}) == len(visit_summary) print(f"visit table fell from {len(visits)} rows to {len(visit_summary)} rows; " f"is the key unique: {'yes' if unique else 'no'}") print(f"key grain is now {relationship(LONG, visit_summary, 'subscriber_id')}") inner2, empty_left2, empty_right2 = join_tables(LONG, visit_summary, "subscriber_id", "inner") left2, _, _ = join_tables(LONG, visit_summary, "subscriber_id", "left_outer") print(f"\ninner join {inner2} rows (not {result['inner'][0]}), total {total_of(matched)} m3") print(f"left outer join {left2} rows, total {total_of(LONG)} m3: the incoming rows and the incoming " f"total did not change") print(f"fan-out fell from {result['inner'][0] - len(matched)} rows to {inner2 - len(matched)} rows, " f"excess counting from {round(inflated - total_of(matched), 2)} m3 to 0.0 m3") ``` ``` visit table fell from 818 rows to 559 rows; is the key unique: yes key grain is now one-to-many inner join 1344 rows (not 1965), total 26450.81 m3 left outer join 3199 rows, total 63060.63 m3: the incoming rows and the incoming total did not change fan-out fell from 621 rows to 0 rows, excess counting from 12250.65 m3 to 0.0 m3 ``` What removes the fan-out is not changing the join type but bringing the right side to the **grain** of the left side. When the visit table is aggregated per subscriber and falls from 818 rows to 559 rows, the key becomes unique, the relationship stops being many-to-many and returns to one-to-many, and the same join call no longer multiplies any row. The excess counting falls from 12,250.65 m³ to zero, and this fall is not a correction; it never occurred. The last line gives the check that this lesson was looking for. The left outer join made with the right grain closes at 3,199 rows and 63,060.63 m³: the incoming row count and the incoming total have not changed. This is the only cheap sign that a join was done correctly: if the row count and the total of the left table are preserved in the output, the left side has neither dropped nor multiplied. If they are not preserved, the difference is written in two items, and these two items are numbers that the call itself never reports. Aggregating first and joining afterwards also has a price, and it has to be written down: the detail that the 818 visit rows carried, that is, the outcome of each visit, is lost inside the summary. The grain decision therefore depends on the question to be asked. The question "how many visits per subscriber" is answered with the summary table, while the question "how many visits closed as unreachable" is asked of the visit table itself and is never joined with the reading table. #### Summary - The join type determines the row count of the output: the same two tables give 1,965 rows in the inner join, 3,820 in the left outer, 3,856 in the full outer and 2,616,782 in the cross join. - That the inner join gives fewer rows than went in does not mean an elimination: 1,855 readings have dropped, the remaining 1,344 readings have opened into 1,965 rows, and the net change hides both. - Fan-out inflates the total: while the true total of the matched readings is 26,450.81 m³, the total in the output is 38,701.46 m³, and the 12,250.65 m³ in between is readings that have been counted a second time. - Because the inflated total stays below the dataset total, it passes an upper-bound check; the table has undercounting and overcounting at the same time. - The relationship of the key is many-to-many and is not verified: 468 rows silently open into 1,089 rows. The indexed join finishes in 4,017 passes, the nested loop makes 2,616,782 comparisons. #### Next Step The join put the tables next to each other and the row calculation was written down. One axis has not been handled in the dataset yet: time. In the reading table the period is a text column and it can be sorted, but a question between periods such as "how many weeks have passed in between" or "in which week is there no reading at all" cannot be asked in this form. When the time axis is moved to the label index these questions become askable, and a new decision arises: in a resample, is a period that carries no record counted as zero, or counted as never having existed. The next lesson calculates these two answers separately and measures how they change the total and the average. ### Time Series Operations Source: https://academia.sh/en/courses/data-analysis-with-python/time-series-operations 3,199 readings are reduced to seven-day periods and two answers come out for the 5 periods that carry no record: if they are counted as zero the series has 13 rows, if they are skipped it has 8 rows. The total is 63,060.63 m³ in both and does not change, but every calculation after the total does. The weekly average moves between 4,850.82 m³ and 7,882.58 m³; of the 12 period-to-period percent change ratios, 5 are left undefined because they fall on a zero denominator, while none of the 7 ratios of the skipped series is undefined; the three-period moving average splits between 708.03 m³ and 9,387.54 m³ in the period of 2026-02-12. The same dataset gives 3 periods with a thirty-day step and none of them is empty: what creates the empty period is not the data but the step decision. The join put the tables next to each other and the row calculation was written down. There is an axis in the dataset that has not been handled yet: time. In the reading table the period is a text column. It can be sorted, grouped, put into an equality; but questions such as "how many weeks have passed in between", "in which week is there no reading at all" or "what is the average of the last three periods" cannot be asked in this form, because a text column does not know the **distance** between two values. When the time axis is moved to the label index these questions become askable, and a new decision comes along with them. A time axis is **continuous**, while the data is discrete: every period on the axis does not have to have a counterpart in the dataset. For a period that carries no record there are two answers, and both are defensible: the consumption was zero in that period, or that period was never measured. The measure of this lesson is the place where these two answers part. - **DF26. The time axis** is the labels of the index being dates. A date index knows the distance between two labels, and this knowledge makes sorting, taking differences and resampling possible. - **DF27. Resampling** is reducing the records to a coarser step on the time axis. What determines the number of periods that come out is not the data but **the step decision**. - **DF28. The empty period** is a period that is on the axis but carries no record. It can be counted as zero or not produced in the output at all; the difference between the two is a row count. - **DF29.** The empty period decision **does not change the total**, because zero adds nothing to a sum. It changes every calculation that comes after the total: the denominator of the average, the denominator of the change ratio and the periods covered by the moving window. #### The Time Axis ```python # dataset.py — MODEL: the long-format reading table of the previous lesson is # regenerated from the same seed. The dataset is fiction; the detail of the # generator is in the first lesson of the topic. import math M32 = 0xFFFFFFFF SEED, PERIODS = 20260218, ["2026-01", "2026-02", "2026-03"] REGION = [("north", 0.28, 21), ("south", 0.22, 17), ("east", 0.18, 26), ("west", 0.14, 14), ("central", 0.18, 23)] def rng(seed: int): s = ((seed ^ (seed >> 16)) * 2246822507) & M32 s = ((s ^ (s >> 13)) * 3266489909) & M32 s ^= s >> 16 def next_value() -> float: nonlocal s s = (s * 1664525 + 1013904223) & M32 return s / 4294967296 return next_value def pick(u: float, weights: list[float]) -> int: t = 0.0 for i, p in enumerate(weights): t += p if u < t: return i return len(weights) - 1 subscribers, dropped = [], [] for i in range(1400): r = rng(SEED + i) name, _, base = REGION[pick(r(), [b[1] for b in REGION])] r(); r() # the household and satisfaction draws are not used in this lesson (dropped if r() < 0.046 else subscribers).append( {"subscriber_id": 10001 + i, "region": name, "base": base}) valid = {k["subscriber_id"] for k in subscribers} pairs: dict[tuple[int, str], float] = {} for k in subscribers + dropped: r = rng(SEED + 7000 + k["subscriber_id"]) for d in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): t = 0.0 if r() < 0.038 else math.floor(k["base"] * math.exp( (r() + r() + r() - 1.5) * 0.62) * (1 - d * 0.05) * 100 + 0.5) / 100 if k["subscriber_id"] in valid: pairs.setdefault((k["subscriber_id"], PERIODS[d]), t) r() LONG = [{"subscriber_id": a, "period": d, "consumption_m3": t} for (a, d), t in pairs.items()] # Reading date: the field team works in the reading window of each month. The # period text is moved to a date axis this way. from datetime import date, timedelta START, END = date(2026, 1, 1), date(2026, 3, 31) MONTH = {"2026-01": 1, "2026-02": 2, "2026-03": 3} def reading_date(no: int, period: str) -> date: """The reading window is between the 18th and the 28th of the month.""" r = rng(SEED + 77000 + no * 4 + MONTH[period]) return date(2026, MONTH[period], 18 + pick(r(), [0.09] * 10 + [0.10])) records = [{"subscriber_id": o["subscriber_id"], "date": reading_date(o["subscriber_id"], o["period"]), "consumption_m3": o["consumption_m3"]} for o in LONG] print(f"{len(records)} readings, date range {min(k['date'] for k in records)} .. " f"{max(k['date'] for k in records)}") print(f"axis between {START} and {END} is {(END - START).days + 1} days, " f"{(END - START).days // 7 + 1} seven-day periods") print(f"total {round(sum(k['consumption_m3'] for k in records), 2)} m3") ``` ``` 3199 readings, date range 2026-01-18 .. 2026-03-28 axis between 2026-01-01 and 2026-03-31 is 90 days, 13 seven-day periods total 63060.63 m3 ``` The mismatch between the two numbers sets up the whole subject of the lesson. The axis is 90 days long and is divided into 13 periods with a seven-day step. The data, on the other hand, consists of 3,199 readings, and all of the readings are gathered between the 18th and the 28th of the month, because the field team works in a certain reading window each month. This is not a flaw of the dataset, it is the way the work is done. Its consequence is this: some periods of the axis carry no reading at all. These periods are empty because no data was collected, not because there was no consumption. The water kept flowing, the meter kept turning, but that week nobody went to read it. #### What an Empty Period Is ```python # resample.py — reduces the time axis to seven-day periods. For a period that # carries no record there are two answers: zero or nonexistent. def resample(rows, step: int, empty_as_zero=True): """If empty_as_zero is true a period that carries no record is produced with 0.0; if false that period does not appear in the output at all.""" bucket: dict[int, list[float]] = {} for k in rows: bucket.setdefault((k["date"] - START).days // step, []).append(k["consumption_m3"]) out = [] for i in range((END - START).days // step + 1): v = bucket.get(i, []) if not v and not empty_as_zero: continue out.append({"period": START + timedelta(days=i * step), "readings": len(v), "total_m3": round(sum(v), 2)}) return out skipped = resample(records, 7, empty_as_zero=False) zero_filled = resample(records, 7, empty_as_zero=True) WIDTH = [14, 10, 14, 10, 14] show = lambda h: "".join(str(v).rjust(WIDTH[k]) if k else str(v).ljust(WIDTH[0]) for k, v in enumerate(h)) print(show(["period start", "readings", "total m3", "readings", "total m3"])) d = {c["period"]: c for c in skipped} for c in zero_filled: e = d.get(c["period"]) print(show([str(c["period"]), c["readings"], c["total_m3"], e["readings"] if e else "-", e["total_m3"] if e else "-"])) total_zero = round(sum(c["total_m3"] for c in zero_filled), 2) total_skip = round(sum(c["total_m3"] for c in skipped), 2) print(f"\nrows : {len(zero_filled)} when the empty period is counted as zero, {len(skipped)} when skipped") print(f"total: {total_zero} m3 and {total_skip} m3 — the total does not change") print(f"average: {round(total_zero / len(zero_filled), 2)} m3 and {round(total_skip / len(skipped), 2)} m3") ``` ``` period start readings total m3 readings total m3 2026-01-01 0 0 - - 2026-01-08 0 0 - - 2026-01-15 460 9260.91 460 9260.91 2026-01-22 800 16777.6 800 16777.6 2026-01-29 0 0 - - 2026-02-05 0 0 - - 2026-02-12 106 2124.1 106 2124.1 2026-02-19 690 13715.37 690 13715.37 2026-02-26 305 5829.46 305 5829.46 2026-03-05 0 0 - - 2026-03-12 87 1665.23 87 1665.23 2026-03-19 521 9477.26 521 9477.26 2026-03-26 230 4210.7 230 4210.7 rows : 13 when the empty period is counted as zero, 8 when skipped total: 63060.63 m3 and 63060.63 m3 — the total does not change average: 4850.82 m3 and 7882.58 m3 ``` Two series come out of the same dataset, and at first sight both say the same thing: the total is 63,060.63 m³. This equality gives a misleading confidence, because zero adds nothing to a sum and the empty period decision leaves no trace in the total. The difference starts in the row count: 13 rows against 8 rows. In the series filled with zeros, five periods carry `0` consumption, and these five rows are drawn on a chart as five points that sink to the floor. Whoever reads the chart sees this: the consumption was zero in the first two weeks of the year, then it rose, and at the end of January it fell to zero again. None of this happened. What is zero is not the consumption but the number of readings made that week, and the second column of the table already says so. The difference in the average is the numerical counterpart of this reading. The same total gives 4,850.82 m³ when divided by 13 and 7,882.58 m³ when divided by 8; the second number is 1.63 times the first. The number to be reported as "weekly average consumption" is one of these two, and which one it is is determined by an option that is never written down. #### Every Calculation After the Total ```python # change.py — counting the empty period as zero does not change the total, but it # changes every calculation that comes after the total. def change(series): """Percent change from period to period; if the denominator is zero the ratio is undefined.""" o, undefined = [], 0 for i in range(1, len(series)): previous = series[i - 1]["total_m3"] if previous == 0: undefined += 1 o.append(None) else: o.append(round(100 * (series[i]["total_m3"] - previous) / previous, 1)) return o, undefined def moving(series, window=3): """For each period the average of the last window periods.""" o = {} for i in range(len(series)): v = [c["total_m3"] for c in series[max(0, i - window + 1):i + 1]] o[series[i]["period"]] = round(sum(v) / len(v), 2) return o c1, u1 = change(zero_filled) c2, u2 = change(skipped) print(f"zero-filled series: {len(c1)} change ratios, {u1} of them undefined (denominator zero)") print(f"skipped series: {len(c2)} change ratios, {u2} of them undefined") m1, m2 = moving(zero_filled), moving(skipped) common = [g for g in m1 if g in m2] farthest = max(common, key=lambda g: abs(m1[g] - m2[g])) print(f"three-period moving average is compared in {len(common)} common periods; " f"the largest difference is {farthest}: zero-filled {m1[farthest]} m3, skipped {m2[farthest]} m3") thirty_day = resample(records, 30, empty_as_zero=True) print(f"\nthe same dataset gives {len(thirty_day)} periods with a thirty-day step and none of them is empty; " f"what creates the empty period is not the data itself, it is the step decision") print(f"\npasses: a dataset scan per period {13 * len(records)}, a single bucket pass " f"{len(records) + 13}") print(f"code : hand-written period loop 9 lines, resample 1 call") ``` ``` zero-filled series: 12 change ratios, 5 of them undefined (denominator zero) skipped series: 7 change ratios, 0 of them undefined three-period moving average is compared in 8 common periods; the largest difference is 2026-02-12: zero-filled 708.03 m3, skipped 9387.54 m3 the same dataset gives 3 periods with a thirty-day step and none of them is empty; what creates the empty period is not the data itself, it is the step decision passes: a dataset scan per period 41587, a single bucket pass 3212 code : hand-written period loop 9 lines, resample 1 call ``` The change ratio is the harshest counterpart of the empty period decision. In the series filled with zeros, **5 of the 12 ratios are undefined**, because there is a zero in the denominator. In a spreadsheet these cells either stay empty or carry an error value; both are noise that has to be explained in the middle of a report, and it comes not from the data itself but from a crude default. In the skipped series none of the 7 ratios of the same calculation is undefined. The moving average, on the other hand, drifts silently. The three-period window gives 708.03 m³ in the period of 2026-02-12 in the zero-filled series and 9,387.54 m³ in the skipped series. The ratio between them is thirteen times, and the reason is simple: in the zero-filled series the window covers two empty periods and divides the average by three, while in the skipped series the window covers three real reading periods. Both numbers are calculated correctly, both carry the same window name, and which one will be reported is not written even in a sentence. The last two lines show where the decision comes from. The same dataset gives three periods with a thirty-day step and none of them is empty. The empty period is not a property of the data; it is **a product of the step decision**. The moment the seven-day step is chosen, five empty periods have been created, and this comes from the reading window not carrying a meaningful signal at weekly resolution. The gain side has to be written down as well. An approach that scans the dataset from the start for every period costs 41,587 passes; the resample that walks the records once and throws them into the period bucket finishes in 3,212 passes, and the nine-line loop falls to a single call. The two decisions stand exactly inside this shortened call: the step and what the empty period is counted as. #### A Shift Does Not Know the Group Boundary ```python # shift.py — to take a difference the series is shifted by one step. A shift knows # the table order, it does not know the group boundary. ordered = sorted(LONG, key=lambda o: (o["subscriber_id"], o["period"])) raw_diff = [(ordered[i]["subscriber_id"], ordered[i - 1]["subscriber_id"], round(ordered[i]["consumption_m3"] - ordered[i - 1]["consumption_m3"], 2)) for i in range(1, len(ordered))] same = [f for f in raw_diff if f[0] == f[1]] boundary = [f for f in raw_diff if f[0] != f[1]] print(f"{len(raw_diff)} differences were produced from {len(ordered)} rows") print(f"same subscriber's two periods: {len(same)} differences, average " f"{round(sum(f[2] for f in same) / len(same), 2)} m3") print(f"between two subscribers : {len(boundary)} differences, average " f"{round(sum(f[2] for f in boundary) / len(boundary), 2)} m3") print(f"a shift that does not know the group boundary calculates {len(boundary)} of " f"{len(raw_diff)} differences from two separate subscribers") ``` ``` 3198 differences were produced from 3199 rows same subscriber's two periods: 1939 differences, average -1.12 m3 between two subscribers : 1259 differences, average 1.72 m3 a shift that does not know the group boundary calculates 1259 of 3198 differences from two separate subscribers ``` Taking a difference means shifting the series by one step and subtracting it from itself, and in a long-format table this operation follows the table order. The table is in subscriber and period order, so most of the differences come out right: 1,939 differences compare two consecutive periods of the same subscriber. The remaining **1,259 differences** are between the last period of one subscriber and the first period of the next subscriber, and they measure nothing. The difference between the two averages gives the sign of this contamination: while the average of the real period differences is -1.12 m³, the average of the boundary differences is +1.72 m³. If the two are reported together, the period-by-period downward trend in consumption weakens. A shift does not know the group boundary by itself; for it to know, the shift has to be done per subscriber, that is, a grouping decision has to be written before taking the difference. #### Summary - A date index knows the distance between two labels; a text column does not. Resampling, sorting and taking differences become possible with this knowledge. - The same dataset gives 13 periods with a seven-day step and 3 periods with a thirty-day step; what creates the empty period is not the data but the step decision. - When the empty period is counted as zero the series has 13 rows, when skipped it has 8 rows; the total is 63,060.63 m³ in both and the decision leaves no trace in the total. - Every calculation after the total changes: the average moves between 4,850.82 and 7,882.58 m³, 5 of the 12 change ratios are left undefined because of a zero denominator, and the three-period moving average splits between 708.03 and 9,387.54 m³ in the period of 2026-02-12. - A series filled with zeros produces points that sink to the floor on a chart and is read as "consumption stopped"; what is zero is not the consumption but the number of readings made in that period. #### Next Step Numeric columns, dates and keys have been handled. The last kind of field left in the dataset is text: the region name, the visit outcome, the address code, the field note. Text columns stand in the frame in the same place as numeric columns and seem to be processable in the same way; a string method is applied to the whole column with a single call and there is no need to write a loop. What that call does on a missing cell is visible nowhere. The next lesson handles vectorized string operations, counts what they return on a missing value, and closes the topic. ### Text Operations Source: https://academia.sh/en/courses/data-analysis-with-python/text-operations Of the 3,199 rows of the address code column, 2,943 are complete, 160 are empty and 96 do not carry the third part. The hand-written loop fails at row 20 and stops, producing 0 rows; the vectorized chain that does the same job produces all 3,199 rows, raises no error at all and silently turns 256 rows into missing. The filter that comes after the chain leaves these 256 rows outside on equality and takes them in on inequality: 1,004 and 2,195 rows, whose sum is exactly the dataset. Because the first step of the chain converts the code to lowercase, the uppercase filter returns 0 rows and that does not raise a warning either. A text column with three intermediate series holds 32.6 bytes of pointers per row, and the same structure reaches 0.61 gibibytes at 20 million rows. Numeric columns, dates and keys have been handled. The last kind of field left in the dataset is text: the region name, the visit outcome, the address code, the field note. Text columns stand in the frame next to numeric columns and look as if they can be processed in the same way. A string method is applied to the whole column with a single call, the result becomes a new series, and there is no need to write a loop. The price of this convenience is measured in this lesson. A string method is defined on a text; it is not defined on a missing cell. A hand-written loop reports this by raising an error. A vectorized operation does not report it: it does not pass the missing cell through the operation, it leaves it missing and keeps running. The measure is this — how many rows each produces, where each stops, and how many of the rows produced have silently turned into missing. - **DF30. A vectorized string operation** is applying a string method to the whole column with a single call. A missing cell does not pass through the operation and **stays missing** in the result; the operation does not raise an error. - **DF31.** A string chain **propagates the missing value**. A cell that turns missing at any step of the chain stays missing in all later steps, and how many rows are missing at the end of the chain is known only if they are counted separately. - **DF32.** A malformed value is different from a missing value but at the end of the chain it falls into **the same place**: both become missing and both are reported in the same way. - **DF33.** One step of the chain changes the input of the later steps. Decisions about case, whitespace and separators can make an equality filter further down the chain useless without producing any warning. #### The Text Column and the Malformed Format ```python # dataset.py — MODEL: the long-format reading table of the previous lesson is # regenerated from the same seed. The dataset is fiction; the detail of the # generator is in the first lesson of the topic. import math M32 = 0xFFFFFFFF SEED, PERIODS = 20260218, ["2026-01", "2026-02", "2026-03"] REGION = [("north", 0.28, 21), ("south", 0.22, 17), ("east", 0.18, 26), ("west", 0.14, 14), ("central", 0.18, 23)] def rng(seed: int): s = ((seed ^ (seed >> 16)) * 2246822507) & M32 s = ((s ^ (s >> 13)) * 3266489909) & M32 s ^= s >> 16 def next_value() -> float: nonlocal s s = (s * 1664525 + 1013904223) & M32 return s / 4294967296 return next_value def pick(u: float, weights: list[float]) -> int: t = 0.0 for i, p in enumerate(weights): t += p if u < t: return i return len(weights) - 1 subscribers, dropped = [], [] for i in range(1400): r = rng(SEED + i) name, _, base = REGION[pick(r(), [b[1] for b in REGION])] r(); r() # the household and satisfaction draws are not used in this lesson (dropped if r() < 0.046 else subscribers).append( {"subscriber_id": 10001 + i, "region": name, "base": base}) valid = {k["subscriber_id"] for k in subscribers} pairs: dict[tuple[int, str], float] = {} for k in subscribers + dropped: r = rng(SEED + 7000 + k["subscriber_id"]) for d in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): t = 0.0 if r() < 0.038 else math.floor(k["base"] * math.exp( (r() + r() + r() - 1.5) * 0.62) * (1 - d * 0.05) * 100 + 0.5) / 100 if k["subscriber_id"] in valid: pairs.setdefault((k["subscriber_id"], PERIODS[d]), t) r() LONG = [{"subscriber_id": a, "period": d, "consumption_m3": t} for (a, d), t in pairs.items()] # The address code is produced in the field and has the form REGION-NNNN-L. Some # of them were never assigned, in some the third part is missing. MISSING = None def address_code(no: int, region: str): r = rng(SEED + 61000 + no) if r() < 0.048: return MISSING if r() < 0.037: return f"{region.upper()}-{no - 10000:04d}" return f"{region.upper()}-{no - 10000:04d}-{'ABC'[pick(r(), [0.42, 0.35, 0.23])]}" code_of = {k["subscriber_id"]: address_code(k["subscriber_id"], k["region"]) for k in subscribers} records = [{"subscriber_id": o["subscriber_id"], "consumption_m3": o["consumption_m3"], "address_code": code_of[o["subscriber_id"]]} for o in LONG] column = [k["address_code"] for k in records] missing = sum(1 for v in column if v is MISSING) malformed = sum(1 for v in column if v is not MISSING and v.count("-") < 2) print(f"{len(records)} rows: {len(column) - missing - malformed} complete codes, {missing} missing, " f"{malformed} without a third part") print(f"sample codes: {[v for v in column[:6]]}") ``` ``` 3199 rows: 2943 complete codes, 160 missing, 96 without a third part sample codes: ['CENTRAL-0001-A', 'CENTRAL-0001-A', 'NORTH-0002-A', 'NORTH-0002-A', 'NORTH-0002-A', 'SOUTH-0003-A'] ``` There are three separate cases in the column, and two of them are the subject of this lesson. The address code of 160 rows **does not exist at all**: the address record of the subscriber was never bound to a code. The code of 96 rows exists but is **malformed**: the region and number parts are in place, the third part, that is the building letter, was not written. The two stand side by side in a dataset and they are different things. For a row that carries no code, nothing is known. For a row whose code is malformed, both the region and the number are known, only one part is missing. Whether this distinction is preserved to the very end is one of the numbers this lesson will count. #### The Loop That Stops, the Chain That Does Not ```python # strings.py — the same string operation in two ways: a hand-written loop and a # single call for the whole column. The difference shows on a missing cell. import sys def manual_lower(column): """Hand-written loop: the string method is applied directly to every cell.""" return [v.lower() for v in column] def vector(column, operation): """Vectorized operation: a missing cell does not pass through, it stays missing.""" return [MISSING if v is MISSING else operation(v) for v in column] try: manual_lower(column) except AttributeError as error: first = next(i for i, v in enumerate(column) if v is MISSING) print(f"manual loop stopped at row {first + 1}: AttributeError {error}") print(f"rows produced: 0 ({len(column) - first - 1} remaining rows were never looked at)") lower = vector(column, str.lower) parts = vector(lower, lambda v: v.split("-")) letter = [MISSING if p is MISSING or len(p) < 3 else p[2] for p in parts] print(f"\nvectorized chain: produced {len(lower)} rows, raised 0 errors") print(f"after the third part is extracted {sum(1 for h in letter if h is MISSING)} rows are missing: " f"{sum(1 for v in column if v is MISSING)} carrying no code at all, " f"{sum(1 for p in parts if p is not MISSING and len(p) < 3)} without a third part") print(f"bytes: three intermediate series {sys.getsizeof(lower) + sys.getsizeof(parts) + sys.getsizeof(letter)}, " f"source column {sys.getsizeof(column)}") print(f"code : manual safe loop 9 lines, vectorized chain 3 calls") ``` ``` manual loop stopped at row 20: AttributeError 'NoneType' object has no attribute 'lower' rows produced: 0 (3179 remaining rows were never looked at) vectorized chain: produced 3199 rows, raised 0 errors after the third part is extracted 256 rows are missing: 160 carrying no code at all, 96 without a third part bytes: three intermediate series 78120, source column 26040 code : manual safe loop 9 lines, vectorized chain 3 calls ``` The two approaches do the same job and fail in exactly opposite ways. The manual loop **stops at row 20**, produces zero rows and never looks at the remaining 3,179 rows. In a production line this is an outage and work cannot continue until it is fixed; but the outage itself is information, because it announces within the first twenty rows that the dataset has rows that carry no code. The vectorized chain never stops. It produces all 3,199 rows, raises zero errors and at the end of the chain **256 rows** are missing. These 256 rows do not show up in the row count of the output; the output is still 3,199 rows and its length is the same as the source. They have to be counted separately to be seen, and the count in this lesson is that separate line itself. The account of the loss has two items and both have merged at the end of the chain: 160 rows never had a code, 96 rows had a code but no third part. In the output of the chain the two are the same thing, that is, missing. The distinction preserved in the previous section is lost here, and the reason for losing it is not an error but that the step extracting the third part cannot tell a missing part from a missing record. The gain side has to be written down as well. The manual safe loop, together with the error handling, takes nine lines; the vectorized chain is three calls. The price is in memory: the three intermediate series of the chain hold 78,120 bytes of pointers, while the source column itself is 26,040 bytes. Every step produces a new series and this triples as the chain gets longer. #### The Filter After the Chain ```python # filter.py — the missing that comes out of the string operation silently falls # to one side in the filter after it. An earlier step of the chain also affects # the filter. T = lambda m: round(sum(k["consumption_m3"] for k, b in zip(records, m) if b), 2) upper = [h == "B" for h in letter] lower_b = [h == "b" for h in letter] not_equal = [h != "b" for h in letter] unknown = [h is MISSING for h in letter] WIDTH = [30, 10, 14] show = lambda h: "".join(str(v).rjust(WIDTH[k]) if k else str(v).ljust(WIDTH[0]) for k, v in enumerate(h)) print(show(["filter", "rows", "total m3"])) print(show(["letter == 'B'", sum(upper), T(upper)])) print(show(["letter == 'b'", sum(lower_b), T(lower_b)])) print(show(["letter != 'b'", sum(not_equal), T(not_equal)])) print(show(["letter unknown", sum(unknown), T(unknown)])) print(show(["dataset", len(records), T([True] * len(records))])) print(f"\nbecause the chain code converted to lowercase first, the 'B' filter returns {sum(upper)} rows " f"and produces no warning") print(f"the equality filter leaves {sum(unknown)} unknown rows outside, the inequality filter takes " f"the same {sum(unknown)} rows in: " f"{sum(lower_b)} + {sum(not_equal)} = {sum(lower_b) + sum(not_equal)}") row_bytes = (sys.getsizeof(column) + sys.getsizeof(lower) + sys.getsizeof(parts) + sys.getsizeof(letter)) / len(records) print(f"\na single text column and three intermediate series hold {round(row_bytes, 1)} bytes of " f"pointers per row; {round(row_bytes * len(records) / 1024)} kibibytes at {len(records)} rows") print(f"the same structure needs {round(row_bytes * 20_000_000 / 1024 ** 3, 2)} " f"gibibytes of pointers at 20 million rows and the strings themselves are not included") ``` ``` filter rows total m3 letter == 'B' 0 0 letter == 'b' 1004 20069.73 letter != 'b' 2195 42990.9 letter unknown 256 4980.99 dataset 3199 63060.63 because the chain code converted to lowercase first, the 'B' filter returns 0 rows and produces no warning the equality filter leaves 256 unknown rows outside, the inequality filter takes the same 256 rows in: 1004 + 2195 = 3199 a single text column and three intermediate series hold 32.6 bytes of pointers per row; 102 kibibytes at 3199 rows the same structure needs 0.61 gibibytes of pointers at 20 million rows and the strings themselves are not included ``` The first line is the quietest error of the topic. The filter looks for `'B'`, there is no `'B'` in the column, because the first step of the chain has converted all the code to lowercase. The result is **zero rows**, zero cubic meters and zero warnings. An empty result set means both "there is no such record" and "you looked in the wrong place"; there is no sign in the output that separates the two. The second and third lines repeat the finding of the fourth lesson of the topic on text. The equality filter leaves the 256 unknown rows outside, the inequality filter takes the same 256 rows in, and 1,004 plus 2,195 add up to exactly 3,199. The dataset looks as if it has been split in two, yet there is a third set and it carries 4,980.99 m³ of consumption. A table split in two does not even open a row to write this third set. The last two lines are the closing measure of the topic. A single text column, together with its three intermediate series, holds 32.6 bytes of pointers per row; at 3,199 rows this is 102 kibibytes and goes completely unnoticed. The same structure reaches 0.61 gibibytes at 20 million rows, and this is the cost of the pointers alone, excluding the content of the strings themselves. All the comfort built in eight lessons rests on the assumption that the dataset fits in memory. #### Winning the Distinction Back ```python # pattern.py — using a pattern that validates the format instead of splitting on # the separator keeps the missing record and the malformed format apart at the # end of the chain. import re PATTERN = re.compile(r"^([a-z]+)-(\d{4})-([abc])$") def status_of(v) -> str: if v is MISSING: return "missing" return "complete" if PATTERN.match(v) else "malformed" counts: dict[str, int] = {} for v in lower: counts[status_of(v)] = counts.get(status_of(v), 0) + 1 print(f"parsing with a pattern: {counts['complete']} complete, {counts['malformed']} malformed format, " f"{counts['missing']} missing record; total {sum(counts.values())}") pattern_letter = [PATTERN.match(v).group(3) if v is not MISSING and PATTERN.match(v) else MISSING for v in lower] print(f"third part: {sum(1 for h in letter if h is MISSING)} missing by splitting, " f"{sum(1 for h in pattern_letter if h is MISSING)} missing by pattern; the same number, " f"the difference is in reportability") print(f"two ways that give the same result: splitting {len(lower)} passes and 2 intermediate series, " f"pattern {len(lower)} passes and 1 intermediate series") ``` ``` parsing with a pattern: 2943 complete, 96 malformed format, 160 missing record; total 3199 third part: 256 missing by splitting, 256 missing by pattern; the same number, the difference is in reportability two ways that give the same result: splitting 3199 passes and 2 intermediate series, pattern 3199 passes and 1 intermediate series ``` The distinction lost by splitting can be won back and the cost is not even a pass. When a pattern that validates the format is used instead of splitting on the separator, the column is divided into three numbers: 2,943 complete codes, 96 malformed formats, 160 missing records. The sum of the three is 3,199, and this sum check is the counterpart on text of the single check repeated throughout the topic. The third part extracted is the same on both ways: 256 rows are missing. The only thing that changes is that where the 256 come from can be **written down**. The 96 rows of malformed format are a correctable problem and can be asked back to their source; the 160 rows that have no code at all cannot be corrected, because there is no record to correct. A report that adds the two up into a single number erases this distinction and with it the work to be done. The number of passes is 3,199 on both ways, so the pattern adds no cost; the number of intermediate series falls from two to one. What lost the distinction was not vectorization itself, it was **splitting not doing any validation**. #### Summary - A vectorized string operation does not raise an error on a missing cell, it propagates the missing value: while the manual loop stops at row 20 and produces 0 rows, the chain produces all 3,199 rows and silently turns 256 of them into missing. - A missing record and a malformed format are different things in the dataset, they fall into the same place at the end of the chain: the 160 rows without a code and the 96 rows without a third part cannot be told apart in the output. - One step of the chain can make the filter after it useless: a search for `'B'` in a column converted to lowercase returns 0 rows and produces no warning. - The equality filter leaves the 256 unknown rows outside and the inequality filter leaves them inside; the sum of 1,004 and 2,195 gives the dataset but the third set of 4,980.99 m³ does not appear in any row. - The price of the chain is memory: three intermediate series hold 78,120 bytes of pointers, the source column is 26,040 bytes, and 32.6 bytes per row reaches 0.61 gibibytes at 20 million rows. #### Next Step In eight lessons the dataframe interface has been built completely, and at every step two numbers were written side by side: the passes the abstraction gained and the decision it hid. Alignment brought 781,710 comparisons down to 2,361 passes, the join brought 2,616,782 comparisons down to 4,017 passes, slicing brought 2,658 comparisons down to 21 passes. Every time, the code that got shorter left behind a default that became invisible at the same rate. All of this gain had a single condition and it was not tested in any lesson: that the dataset fits in memory. The dataset of this topic was 3,199 rows and a single text column held 102 kibibytes with its intermediate series. When the measurement network grows to cover a whole city, the row count is measured in millions and every intermediate series asks for a separate copy. The next topic tests this condition: how long the same jobs take when the whole dataset does not fit in memory, which operations can be split into pieces, how many bytes each type choice saves and how the place where the calculation is done changes the number of rows that are carried. ## Scale and Performance Counting what is left of an abstraction's convenience once the dataset no longer fits in memory: the bytes the same column holds in three types and the gain from the categorical dtype, how chunk size relates to the memory peak and the number of passes, and the operations that cannot be chunked, the number of values a row-based and a columnar read touch in the same query, the line item that partitioning and shuffle add and the job size at which moving to distribution starts to pay off, and the number of rows that cross the network once the filter and the aggregation are pushed to the source. ### Memory Usage Source: https://academia.sh/en/courses/data-analysis-with-python/memory-usage In the 3,199-row dataset of the measurement network, the number of bytes a single column holds in memory depends on the type that column is stored in: the region column takes 173,837 bytes as an object list, 19,251 bytes as fixed-width text and 3,628 bytes as a categorical code, a gain of 47.9 times. Two defensible measurements of the same column differ by 6.6 times, because the measuring function counts the pointer and does not follow the object it points to. When the reading column is cut from an object list to 1 byte, memory shrinks 31.4 times but the total drifts by 6.9 m³ and 108 rows change tariff tier; the categorical dtype gains 5.3 times on a column with 5 distinct values and loses 4.1 times on a column with 1,329 distinct values. The previous topic built the dataframe abstraction over eight lessons and wrote down the default of every call. The closing of that topic pointed to a single condition: the convenience of the dataframe holds only as long as the dataset fits in memory. This topic starts where that condition lifts, and its first question is the most concrete one: how many bytes does a column hold in memory. The answer depends not on the number of values but on the type those values are stored in. The type decision is also usually made without anyone making it: the call that reads the dataset picks a default, and the column settles into memory with that default. The unit of measure of this lesson is **allocated bytes**. The second number next to it is the decision quietly taken to win those bytes: which values can no longer be told apart, and by how many units which number was recorded wrongly. - **SP1.** The dataset is fictional and gives the long-format reading table prepared by M26/K03: 3,199 rows, 1,329 subscribers, three periods. Instead of a real dataset, a generator that produces the same structure was written; the seed is visible and the output is identical from run to run. **It is a model.** - **SP2.** Measurement is done with `sys.getsizeof`. This function returns an object's **own body** and does not follow the objects it points to; the total is therefore built by hand. - **SP3.** The figures were read from a runtime with eight-byte pointers and depend on the runtime. What does not change is the **ratios**; even if the pointer width changes, the multiple between representations stays the same. - **SP4.** Readings are reported with one decimal. For two totals to count as equal the threshold is 0.05 m³, because a difference smaller than that can come only from rounding. - **SP5.** The tariff tiers set up three tiers with limits of 10 and 25 m³, and the largest reading the generator can produce is 37.0 m³; the width of the integer representations is chosen according to this upper bound. #### The Columns of the Dataset ```python # source.py — fictional long-format reading table from the regional measurement network. # MODEL: a generator that produces the same structure was written in place of a real # dataset; the seed is visible and the output is identical from run to run. The dataset # prepared by M26/K03: 3199 rows, 1329 subscribers, three periods, tariff limits 10 and 25 m3. from collections import Counter SEED = 1246 REGION = ("north", "east", "center", "south", "west") PERIODS = (("2024-01", 1329), ("2024-02", 1035), ("2024-03", 835)) DENSITY = (0, 1, 2, 2, 2, 3, 4, 2, 2, 1) # center-weighted measurement network def rng(seed: int): s = seed & 0xFFFFFFFF def next_value() -> float: nonlocal s s = (1103515245 * s + 12345) & 0xFFFFFFFF return s / 4294967296 return next_value def tier(reading: float) -> str: return "0-10" if reading <= 10 else "10-25" if reading <= 25 else "25-40" def records() -> list[tuple]: r, out = rng(SEED), [] for period, size in PERIODS: for i in range(size): o = round(5 + r() * 32.0, 1) out.append((10000 + i, f"S-{4000 + i}", period, REGION[DENSITY[i % 10]], o, tier(o))) return out r0 = records() count = lambda j: ", ".join(f"{a} {n}" for a, n in Counter(x[j] for x in r0).most_common()) print(f"seed {SEED}; rows {len(r0)}, distinct subscribers {len({x[0] for x in r0})}, " f"periods {len(PERIODS)}") print("region:", count(3)) print("tariff:", count(5)) print(f"total {sum(x[4] for x in r0):.1f} m3, smallest {min(x[4] for x in r0)}, " f"largest {max(x[4] for x in r0)}") ``` ``` seed 1246; rows 3199, distinct subscribers 1329, periods 3 region: center 1601, east 639, north 321, south 319, west 319 tariff: 10-25 1504, 25-40 1187, 0-10 508 total 66667.9 m3, smallest 5.0, largest 37.0 ``` The dataset carries six columns, and two of them produce all of this lesson's measurements. The **region** column holds 3,199 values but contains only five distinct ones; center alone holds half of the rows. The **reading** column holds 3,199 numbers, all between 5.0 and 37.0, and each has one decimal. Both have the same number of rows, but in terms of what they cost in memory they look nothing alike. #### One Column in Three Types ```python # memory.py — the same region column is held in three types and the allocated bytes are counted. # MODEL: the column is first obtained as if read from a file, because the parser # produces a new string object for every row and the strings are not shared # the way they would be from a fixed list. import csv, io, sys from array import array buffer = io.StringIO() csv.writer(buffer, lineterminator="\n").writerows(r0) rows = list(csv.reader(io.StringIO(buffer.getvalue()))) region_col = [s[3] for s in rows] print(f"column has {len(region_col)} values, {len({id(s) for s in region_col})} distinct objects, " f"{len(set(region_col))} distinct values") container = sys.getsizeof(region_col) # pointers only objects = sum(sys.getsizeof(s) for s in {id(s): s for s in region_col}.values()) shared = sum(sys.getsizeof(s) for s in set(region_col)) longest = max(len(a) for a in set(region_col)) # fixed-width text fixed = bytearray(len(region_col) * longest) for j, a in enumerate(region_col): fixed[j * longest:(j + 1) * longest] = a.encode("ascii").ljust(longest) vocab = sorted(set(region_col)) # categorical dtype codes = array("B", [vocab.index(a) for a in region_col]) cat_vocab = sys.getsizeof(vocab) + sum(sys.getsizeof(s) for s in vocab) fmt = lambda h: "".join(str(v).rjust(14) if j else str(v).ljust(26) for j, v in enumerate(h)) print(fmt(["region column type", "list bytes", "object bytes", "total", "bytes/row"])) for label, cb, ob in [("N1 object list", container, objects), ("N1 if counted as shared", container, shared), ("N2 fixed-width text", sys.getsizeof(fixed), 0), ("N3 categorical code", sys.getsizeof(codes), cat_vocab)]: print(fmt([label, cb, ob, cb + ob, round((cb + ob) / len(region_col), 1)])) print(f"N1 -> N3 gain {round((container + objects) / (sys.getsizeof(codes) + cat_vocab), 1)}x; " f"ratio between two measurements of the same column " f"{round((container + objects) / (container + shared), 1)}x") ``` ``` column has 3199 values, 3199 distinct objects, 5 distinct values region column type list bytes object bytes total bytes/row N1 object list 26040 147797 173837 54.3 N1 if counted as shared 26040 229 26269 8.2 N2 fixed-width text 19251 0 19251 6.0 N3 categorical code 3279 349 3628 1.1 N1 -> N3 gain 47.9x; ratio between two measurements of the same column 6.6x ``` The gain lies between the two ends of the table: the same five-valued column takes 173,837 bytes as an object list and 3,628 bytes as a categorical code, which is **47.9 times**. The reason for the drop from 54.3 bytes per row to 1.1 bytes is a single observation: the column has 3,199 values but only five distinct ones. The categorical dtype turns that gap into structure; the column becomes an array of one-byte codes, and the five strings sit once in a vocabulary. The second number sits in the middle of the table and concerns the measurement itself. The first row and the second row measure **the same column**, and there are 6.6 times between them. The distinction is this: in a column read from a file, each of the 3,199 values is a separate object, because the parser produces a new string for every row. Had the same column been built from a fixed list, five objects would have been shared and the object share would have been 229 bytes instead of 147,797 bytes. The measuring function counts the pointer and does not follow the object it points to; which number is correct depends on where the column came from, and both are defensible. **If a memory report does not state which rule was used, that report is a choice.** Fixed-width text stands in the middle and carries its own decision: every row occupies as much space as the longest region name. Six bytes were chosen for the longest name; if a seven-letter region name enters the column, either the column is rebuilt or the name is truncated. The categorical dtype moves this decision into the vocabulary and removes the width problem. #### The Number That Width Choice Moves ```python # width.py — the reading column is held in four representations; for each one the # allocated bytes, the total and the number of rows whose tariff tier changed are printed side by side. THRESHOLD = 0.05 # readings are reported with one decimal, a difference below this comes from rounding reading_col = [float(s[4]) for s in rows] f64 = array("d", reading_col) # 8-byte float u16 = array("H", [round(o * 10) for o in reading_col]) # 2-byte deci-m3 u8 = array("B", [round(o) for o in reading_col]) # 1-byte whole m3 as_objects = sys.getsizeof(reading_col) + sum(sys.getsizeof(o) for o in reading_col) exact = round(sum(reading_col), 1) G = (22, 13, 12, 12, 17) fmt2 = lambda h: "".join(str(v).rjust(G[j]) if j else str(v).ljust(G[0]) for j, v in enumerate(h)) print(fmt2(["reading column type", "total bytes", "bytes/row", "total m3", "tier changed"])) for label, size, series in [("object list", as_objects, reading_col), ("8-byte float", sys.getsizeof(f64), list(f64)), ("2-byte deci-m3", sys.getsizeof(u16), [v / 10 for v in u16]), ("1-byte whole m3", sys.getsizeof(u8), [float(v) for v in u8])]: changed = sum(1 for a, b in zip(reading_col, series) if tier(a) != tier(b)) print(fmt2([label, size, round(size / len(reading_col), 1), round(sum(series), 1), changed])) drift = abs(sum(u8) - exact) print(f"the 1-byte choice shrinks memory {round(as_objects / sys.getsizeof(u8), 1)}x; " f"total drifts {drift:.1f} m3, {round(drift / THRESHOLD)}x the threshold") # The same transformation finds 1329 distinct values in the meter_id column: the code grows # to two bytes and the vocabulary becomes larger than the column itself. meter_col = [s[1] for s in rows] m_vocab = sorted(set(meter_col)) m_codes = array("H", [m_vocab.index(a) for a in meter_col]) m_categorical = sys.getsizeof(m_codes) + sys.getsizeof(m_vocab) + sum( sys.getsizeof(a) for a in m_vocab) m_fixed = sys.getsizeof(bytearray(len(meter_col) * 6)) r_fixed, r_categorical = sys.getsizeof(fixed), sys.getsizeof(codes) + cat_vocab print(f"region : {len(vocab):5d} distinct values, fixed-width {r_fixed} bytes, " f"categorical {r_categorical} bytes -> {round(r_fixed / r_categorical, 1)}x gain") print(f"meter_id: {len(m_vocab):5d} distinct values, fixed-width {m_fixed} bytes, " f"categorical {m_categorical} bytes -> {round(m_categorical / m_fixed, 1)}x loss") # The whole dataset in two layouts: row records and narrow types chosen per column. row_layout = sys.getsizeof(rows) + sum( sys.getsizeof(s) + sum(sys.getsizeof(a) for a in s) for s in rows) p_vocab, t_vocab = sorted({s[2] for s in rows}), sorted({s[5] for s in rows}) columns = (array("I", [int(s[0]) for s in rows]), m_codes, array("B", [p_vocab.index(s[2]) for s in rows]), codes, u16, array("B", [t_vocab.index(s[5]) for s in rows])) column_layout = sum(sys.getsizeof(v) for v in columns) + sum( sys.getsizeof(sz) + sum(sys.getsizeof(a) for a in sz) for sz in (m_vocab, p_vocab, vocab, t_vocab)) print(f"row layout {row_layout} bytes, column layout {column_layout} bytes -> " f"{round(row_layout / column_layout, 1)}x; {round(column_layout / len(rows))} bytes " f"per row instead of {round(row_layout / len(rows))} bytes") ``` ``` reading column type total bytes bytes/row total m3 tier changed object list 102816 32.1 66667.9 0 8-byte float 25672 8.0 66667.9 0 2-byte deci-m3 6478 2.0 66667.9 0 1-byte whole m3 3279 1.0 66661.0 108 the 1-byte choice shrinks memory 31.4x; total drifts 6.9 m3, 138x the threshold region : 5 distinct values, fixed-width 19251 bytes, categorical 3628 bytes -> 5.3x gain meter_id: 1329 distinct values, fixed-width 19251 bytes, categorical 79653 bytes -> 4.1x loss row layout 1298875 bytes, column layout 109714 bytes -> 11.8x; 34 bytes per row instead of 406 bytes ``` The first three rows are free of charge. Going from the object list to an array of eight-byte numbers shrinks memory 4 times, and going to the two-byte deci-m³ representation shrinks it 15.9 times, and the total does not change: readings are reported with one decimal, one decimal becomes an integer when multiplied by ten, and two bytes comfortably carry the value 370. Up to this point the gain of the abstraction really is free. The fourth row shows the cost. The one-byte whole m³ representation shrinks memory 31.4 times relative to the object list and moves the total by only 6.9 m³, and that difference is about one ten-thousandth of the total; looking at the report, it seems harmless. But the last column gives another number: **108 rows change tariff tier**. Rounding cancels out in the total, but at the boundary it does not; a meter that reads 10.4 m³ drops into the first tier when rounded to an integer, and that subscriber's bill changes. Comparing the 6.9 m³ drift in the total with the threshold leads to the same place: the difference is 138 times the threshold, so it cannot be counted as coming from rounding. **This is the hidden default:** narrowing the width looks like a memory decision, but it is a measurement decision that determines which row falls into which tier. The last two rows draw the limit of the categorical dtype. The same transformation gains 5.3 times on the region column relative to fixed-width text and loses 4.1 times on the meter number column. The reason is a single number: 1,329 distinct values do not fit in one byte, the code grows to two bytes and the vocabulary becomes larger than the column itself. "Switch the text column to the categorical dtype" is not a rule; it is a decision that depends on the number of distinct values, and it cannot be given without writing that number down. The final row gives the sum of the decisions. The same 3,199 records take 1,298,875 bytes as row records and 109,714 bytes with narrow types chosen per column: 34 bytes per row instead of 406 bytes, **11.8 times**. This number is the composition of six separate type decisions, and none of them comes by itself; which width and which representation was chosen for each column was written down one by one. A call that reads the dataset gives all of these decisions by default and does not report the result as a single number. #### Summary - The region column takes 173,837 bytes as an object list, 19,251 bytes as fixed-width text and 3,628 bytes as a categorical code; the gain is 47.9 times, a drop from 54.3 bytes per row to 1.1 bytes. - Two defensible measurements of the same column differ by 6.6 times, because the measuring function counts the pointer; a column read from a file holds 3,199 separate objects, a column built from a fixed list holds five. - In the reading column, going from eight bytes down to two bytes does not change the total; going down to one byte shrinks memory 31.4 times relative to the object list but moves the total by 6.9 m³, and that difference is 138 times the threshold. - The same rounding changes the tariff tier of 108 rows; the cost that does not show in the total appears at the boundary. - The categorical dtype gains 5.3 times on a column with 5 distinct values and loses 4.1 times on a column with 1,329 distinct values; the gain depends not on the number of rows but on the number of distinct values. #### Next Step This lesson narrowed the memory cost of a column and found a difference of up to 31 times between four representations. But narrowing has a floor: 3,199 rows fit in memory with every type, and 320 million rows fit with none. At that point the question is no longer choosing a type but giving up on holding the whole dataset in memory at once. The next lesson reads the dataset in chunks and measures the balance that chunk size strikes between the memory peak and the number of passes; it counts which operations can be computed chunk by chunk and which cannot, and shows the wrong average that chunking silently produces. ### Chunked Processing Source: https://academia.sh/en/courses/data-analysis-with-python/chunked-processing When the 3,199-row file of the measurement network is read chunk by chunk, the memory peak drops with chunk size but the state the operation carries does not: at a chunk of 100 rows the peak is 36,132 bytes, 32 times smaller than holding the whole dataset, and the total comes out as 66,667.9 m³ at all four chunk sizes. The total, the count and the region breakdown finish inside the chunk with 3–5 items of state; the exact median runs on a 321-bucket histogram and the exact distinct subscriber count runs on 1,329 items of state, and those 87,166 bytes do not shrink as the chunk shrinks. Chunking silently makes an order assumption: the streaming computation returns a result for 1,035 subscribers on the subscriber-ordered file and, without any error, a result for 0 subscribers on the period-ordered file; when the chunk boundary moves to the region, the mean of the chunk means departs from the overall mean by 0.1039 m³. The previous lesson narrowed the memory cost of a column by ratios of up to 31 times, but it also said that narrowing has a floor: 3,199 rows fit with every type, and as the dataset itself grows no type is enough. This lesson measures what is done when that floor is exceeded — the dataset is not taken into memory at once; it is read chunk by chunk. The promise of chunking is clear: only one chunk sits in memory at a time, and the peak usage is set by the chunk size. The measure of the lesson reads the price of this promise with two numbers. The first is the **gain**: by how many times the memory peak shrinks when the chunk size drops. The second is the **carried state**: the item the operation keeps outside the chunk and that does not shrink when the chunk size changes. This is what determines whether an operation can be computed chunk by chunk. - **SP6.** The dataset is the measurement network dataset of the previous lesson: 3,199 rows, 1,329 subscribers, three periods, a total of 66,667.9 m³. It is produced with the same seed and the output is identical from run to run. **It is a model.** - **SP7.** The file carries the dataset's five fields; the tariff tier is not included in the export because it is derived from the reading. - **SP8.** The file is written in period order: first the 1,329 rows of 2024-01, then the 1,035 rows of 2024-02, then the 835 rows of 2024-03. The rows of one subscriber are hundreds of rows apart. - **SP9.** The memory peak is the bytes of the rows held in memory at the same time plus the operation state; it is measured with `sys.getsizeof` and belongs to a runtime with eight-byte pointers. What does not change is the ratios. - **SP10.** For two means to count as equal the threshold is 0.05 m³, because readings are reported with one decimal and a difference smaller than that can come only from rounding. #### Chunk Size and Memory Peak ```python # dataset.py — the same measurement network dataset is written to a file. MODEL: # a generator that produces the same structure was written in place of a real # export; the seed is visible and the output is identical from run to run. # 3199 rows, 1329 subscribers, three periods. import csv, itertools, os, sys, tempfile SEED = 1246 REGION = ("north", "east", "center", "south", "west") PERIODS = (("2024-01", 1329), ("2024-02", 1035), ("2024-03", 835)) DENSITY = (0, 1, 2, 2, 2, 3, 4, 2, 2, 1) # center-weighted measurement network HEADER = ("subscriber_id", "meter_id", "period", "region", "reading_m3") def rng(seed: int): s = seed & 0xFFFFFFFF def next_value() -> float: nonlocal s s = (1103515245 * s + 12345) & 0xFFFFFFFF return s / 4294967296 return next_value def records() -> list[tuple]: r, out = rng(SEED), [] for period, size in PERIODS: for i in range(size): out.append((10000 + i, f"S-{4000 + i}", period, REGION[DENSITY[i % 10]], round(5 + r() * 32.0, 1))) return out r0 = records() DIR = tempfile.mkdtemp(prefix="measurement-") PATH = os.path.join(DIR, "readings.csv") with open(PATH, "w", newline="") as f: w = csv.writer(f, lineterminator="\n") w.writerow(HEADER) w.writerows(r0) nbytes = lambda p: sys.getsizeof(p) + sum( sys.getsizeof(s) + sum(sys.getsizeof(a) for a in s) for s in p) FULL = nbytes([list(map(str, s)) for s in r0]) print(f"seed {SEED}; file {os.path.getsize(PATH)} bytes, {len(r0)} data rows") print(f"if the whole dataset is taken into memory {FULL} bytes, " f"{round(FULL / len(r0))} bytes per row") print(f"total {sum(x[4] for x in r0):.1f} m3, distinct subscribers {len({x[0] for x in r0})}") ``` ``` seed 1246; file 102557 bytes, 3199 data rows if the whole dataset is taken into memory 1152229 bytes, 360 bytes per row total 66667.9 m3, distinct subscribers 1329 ``` The file takes 102,548 bytes on disk and 1,152,229 bytes in memory — eleven times. The difference is the subject of the previous lesson: once a text line is parsed, every field becomes a separate object. This lesson takes that multiplier as data and asks a single question: is it possible to produce the same numbers without holding the whole dataset in memory at the same time. ```python # chunk.py — the file is read chunk by chunk. The total, the row count, the maximum and the # per-region total are accumulated in the same pass; the chunk size changes only the # number of rows held in memory at the same time. def chunks(path: str, n: int): with open(path, newline="") as f: reader = csv.reader(f) next(reader) # header row while (p := list(itertools.islice(reader, n))): yield p def single_pass(path: str, n: int) -> dict: peak = num_chunks = count = 0 total, largest, by_region = 0.0, 0.0, {} for p in chunks(path, n): peak, num_chunks = max(peak, nbytes(p)), num_chunks + 1 for s in p: o = float(s[4]) total, count, largest = total + o, count + 1, max(largest, o) by_region[s[3]] = by_region.get(s[3], 0.0) + o return {"chunks": num_chunks, "peak": peak, "count": count, "total": round(total, 1), "largest": largest, "region": round(max(by_region.values()), 1)} G = (14, 10, 12, 11, 8, 13, 16) fmt = lambda h: "".join(str(v).rjust(G[j]) if j else str(v).ljust(G[0]) for j, v in enumerate(h)) print(fmt(["chunk size", "chunks", "peak bytes", "peak/full", "passes", "total m3", "largest region"])) for n in (100, 400, 1600, 3199): r = single_pass(PATH, n) print(fmt([n, r["chunks"], r["peak"], f"{r['peak'] / FULL:.3f}", 1, r["total"], r["region"]])) print(f"at chunk size 100 the peak is {round(FULL / single_pass(PATH, 100)['peak'])} times " f"smaller than holding the whole dataset; in all four runs the total is " f"{single_pass(PATH, 100)['total']} m3 and the rows {single_pass(PATH, 100)['count']}") ``` ``` chunk size chunks peak bytes peak/full passes total m3 largest region 100 32 36132 0.031 1 66667.9 33462.3 400 8 144085 0.125 1 66667.9 33462.3 1600 2 577624 0.501 1 66667.9 33462.3 3199 1 1152229 1.000 1 66667.9 33462.3 at chunk size 100 the peak is 32 times smaller than holding the whole dataset; in all four runs the total is 66667.9 m3 and the rows 3199 ``` The gain is linear and can be read from the table: when the chunk size drops from 3,199 to 100, the peak falls from 1,152,229 bytes to 36,132 bytes, to one thirty-second of the dataset. The number of passes is one in all four runs; the file is read once. The total also comes out the same in all four runs — 66,667.9 m³ — and the total of the largest region stays fixed at 33,462.3 m³. These four operations are **indifferent** to chunk size: each keeps a constant number of accumulators, and only those accumulators pass from chunk to chunk. The cost of chunk size is not visible in the table, because this lesson does not measure time. A smaller chunk means more calls, a larger chunk means more memory; the choice lies between these two numbers, and the peak limit comes from the hardware. The side that can be measured is the peak bytes, and it comes out proportional to the chunk size. #### Which Operation Finishes Within the Chunk ```python # operation.py — which operation finishes within the chunk and which carries state outside it. # Measure: the state items and bytes carried, independent of chunk size. THRESHOLD = 0.05 # readings are reported with one decimal, a difference below this comes from rounding N = 400 column, by_region, subscribers, bucket = [], {}, {}, {} for p in chunks(PATH, N): for s in p: o = float(s[4]) column.append(o) by_region[s[3]] = by_region.get(s[3], 0.0) + o subscribers[s[0]] = o d = round(o * 10) # readings have one decimal: 321 buckets are enough bucket[d] = bucket.get(d, 0) + 1 running, median_hist = 0, None for d in sorted(bucket): running += bucket[d] if median_hist is None and running >= (len(column) + 1) // 2: median_hist = d / 10 footprint = lambda x: sys.getsizeof(x) + sum(sys.getsizeof(a) for a in x) G2 = (36, 12, 12, 8, 12) fmt2 = lambda h: "".join(str(v).rjust(G2[j]) if j else str(v).ljust(G2[0]) for j, v in enumerate(h)) print(fmt2(["operation", "state items", "state bytes", "passes", "result"])) for label, state, state_bytes, result in [ ("total, count, max", 3, 3 * 24, round(sum(column), 1)), ("weighted mean", 2, 2 * 24, round(sum(column) / len(column), 4)), ("total per region", len(by_region), footprint(by_region), len(by_region)), ("exact median (histogram)", len(bucket), footprint(bucket), median_hist), ("exact median (holding the column)", len(column), footprint(column), sorted(column)[(len(column) - 1) // 2]), ("distinct subscriber count (exact)", len(subscribers), footprint(subscribers), len(subscribers))]: print(fmt2([label, state, state_bytes, 1, result])) chunk_bytes = max(nbytes(p) for p in chunks(PATH, 100)) print(f"at chunk size 100 the chunk holds {chunk_bytes} bytes, the subscriber state " f"{footprint(subscribers)} bytes; the actual peak is {chunk_bytes + footprint(subscribers)} " f"bytes and the state share does not shrink when the chunk size changes") ``` ``` operation state items state bytes passes result total, count, max 3 72 1 66667.9 weighted mean 2 48 1 20.8402 total per region 5 413 1 5 exact median (histogram) 321 18292 1 20.7 exact median (holding the column) 3199 102816 1 20.7 distinct subscriber count (exact) 1329 87166 1 1329 at chunk size 100 the chunk holds 36132 bytes, the subscriber state 87166 bytes; the actual peak is 123298 bytes and the state share does not shrink when the chunk size changes ``` The table divides the operations into three groups. Those with **constant state** are the easiest: the total, the count and the maximum finish with three accumulators, the weighted mean with two. For these, chunking is free. Those with **bounded state** come next: the total per region carries five items, the exact median 321 items. That the median can be computed chunk by chunk is not a general truth but the consequence of an assumption specific to this dataset — because readings have one decimal and lie between 5.0 and 37.0, the value space fits into 321 buckets and the histogram gives the same number without holding the column: 20.7 m³. If the values were continuous, this row would drop out of the table. The third group is the one that eats the gain of chunking. The **exact distinct subscriber count** carries a state of 1,329 items, and that state is 87,166 bytes. When the chunk is reduced to 100 rows, the chunk itself takes 36,132 bytes, but the actual peak is 123,298 bytes, and the state share does not shrink as the chunk shrinks. The gain of chunking reverses here: three quarters of the peak now belongs to the operation itself. The same holds for the median computation that holds the whole column — 102,816 bytes, no different from holding the whole dataset. #### The Order Assumption Chunking Makes ```python # order.py — the same dataset is written to a second file in subscriber order and the same # computation is repeated in two layouts. Chunked processing silently makes an order # assumption; when the assumption breaks no error is raised, the result empties. PATH2 = os.path.join(DIR, "readings_subscriber.csv") with open(PATH2, "w", newline="") as f: w = csv.writer(f, lineterminator="\n") w.writerow(HEADER) w.writerows(sorted(r0, key=lambda x: (x[0], x[2]))) def streaming_increase(path: str, n: int) -> dict: # state: only the open subscriber increases, active, previous = {}, None, 0.0 for p in chunks(path, n): for s in p: a, o = s[0], float(s[4]) if a != active: active, previous = a, o else: increases[a] = max(increases.get(a, 0.0), o - previous) previous = o return increases def full_increase(path: str, n: int) -> tuple[dict, int]: # state: all subscribers last, increases = {}, {} for p in chunks(path, n): for s in p: a, o = s[0], float(s[4]) if a in last: increases[a] = max(increases.get(a, 0.0), o - last[a]) last[a] = o return increases, len(last) G3 = (18, 17, 18, 14, 14) fmt3 = lambda h: "".join(str(v).rjust(G3[j]) if j else str(v).ljust(G3[0]) for j, v in enumerate(h)) print(fmt3(["file layout", "streaming state", "streaming result", "full state", "full result"])) for label, path in [("period-ordered", PATH), ("subscriber-ordered", PATH2)]: a1 = streaming_increase(path, N) a2, state = full_increase(path, N) print(fmt3([label, 1, len(a1), state, len(a2)])) d1, d2 = streaming_increase(PATH2, N), full_increase(PATH, N)[0] print(f"does the same computation give the same number in the two layouts: " f"{'yes' if d1 == d2 else 'no'}; largest increase " f"{max(d2.values()):.1f} m3") # Moving the chunk boundary to a natural key: one chunk per region. groups = {} for x in r0: groups.setdefault(x[3], []).append(x[4]) mean_of_means = sum(sum(v) / len(v) for v in groups.values()) / len(groups) weighted = sum(x[4] for x in r0) / len(r0) gap = abs(mean_of_means - weighted) print(f"chunk sizes {sorted(len(v) for v in groups.values())}; mean of the chunk means " f"{mean_of_means:.4f} m3, weighted mean {weighted:.4f} m3") print(f"gap {gap:.4f} m3, {gap / THRESHOLD:.1f}x the threshold: the equal-weight assumption " f"breaks as soon as the chunk sizes are unequal") ``` ``` file layout streaming state streaming result full state full result period-ordered 1 0 1329 1035 subscriber-ordered 1 1035 1329 1035 does the same computation give the same number in the two layouts: yes; largest increase 30.7 m3 chunk sizes [319, 319, 321, 639, 1601]; mean of the chunk means 20.7363 m3, weighted mean 20.8402 m3 gap 0.1039 m3, 2.1x the threshold: the equal-weight assumption breaks as soon as the chunk sizes are unequal ``` The first row of the table is the hidden default of the lesson. The streaming computation carries the state of a single subscription and is therefore the cheapest option; on the subscriber-ordered file it produces an increase for 1,035 subscribers, and the result is exactly the same as the result of the computation that holds all the subscribers in memory. The same code gives a result for **0 subscribers without raising any error** on the period-ordered file. An empty result does not look like a failure: the file was read, the rows were processed, the exit code is zero. Thrift in state has turned into an ordering assumption, and the assumption is not written inside the file. The second hidden default is where the chunk is cut. When the chunk boundary is moved from a row count to a natural key — one chunk per region — the chunks range from 319 to 1,601. The mean of the chunk means gives 20.7363 m³ and the weighted mean gives 20.8402 m³; the gap is 0.1039 m³ and 2.1 times the threshold, so it cannot be counted as coming from rounding. The gap looks small, but its direction is fixed: the equal-weight assumption enlarges the small chunks, and as chunk sizes move apart from one another the deviation grows. **If it is not written how a mean computed per chunk is combined, that mean counts as unmeasured.** #### Summary - When the chunk size drops from 3,199 to 100, the memory peak falls from 1,152,229 bytes to 36,132 bytes, to one thirty-second; the number of passes is one and the total comes out as 66,667.9 m³ at all four chunk sizes. - What determines whether an operation can be computed chunk by chunk is not the chunk size but the carried state: the total and the mean carry 2–3 items, the region breakdown 5 items, the exact median 321 items, the exact distinct subscriber count 1,329 items. - That the exact median fits into 321 buckets is not a general truth but an assumption that comes from readings having one decimal and lying in the 5.0–37.0 range. - When the chunk drops to 100 rows, the chunk itself takes 36,132 bytes but the subscriber state is 87,166 bytes; the actual peak rises to 123,298 bytes and the state share does not shrink as the chunk shrinks. - The streaming computation gives a correct result for 1,035 subscribers on the subscriber-ordered file and, with no error, a result for 0 subscribers on the period-ordered file; chunking silently makes an order assumption. - When the chunk boundary moves to the region, the mean of the chunk means departs from the weighted mean by 0.1039 m³, and this gap is 2.1 times the threshold. #### Next Step This lesson saved memory by reading chunk by chunk, but in every chunk it parsed **all the fields** of the file: for the region breakdown all five fields of the 3,199 rows read were converted from text to objects, and then three of them were thrown away. If five columns are read for the two columns the question asks about, chunking lowers only the peak; it does not lower the work done. The next lesson compares the layout that keeps the fields of a record adjacent with the layout that keeps a column adjacent in the same query, and ties the measure to a single number: how many values were read to produce the same answer. ### Columnar Engines Source: https://academia.sh/en/courses/data-analysis-with-python/columnar-engines In the 3,199-row, six-column dataset of the measurement network a single question is answered by four read paths, and all four give the same number: the row-based scan reads 19,194 values, columnar early materialization 9,597, late materialization with the filter moved forward 3 times fewer, 4,869, and the block-skipping path 2,835 values, and the gain reaches 6.8 times. Block skipping depends on the dataset being clustered on that column; when the same dataset is written in subscriber order two of the seven blocks are skipped instead of four, in shuffled order none is skipped, and the values read rise from 2,835 to 4,883. The gain of late materialization also depends on the selectivity of the filter: when the matching rows rise from 835 to 2,691 the gain falls from 1.97 times to 1.12 times and scattered access rises from 1,671 to 5,383. The previous lesson cut the memory peak to one thirty-second by reading the dataset chunk by chunk, but in every chunk it parsed all the fields of the record. The region breakdown looked at two columns; five columns were read, two were used, three were thrown away. Chunking lowers the peak, it does not lower the work done. The question of this lesson is that one: how many values must be touched to produce the same answer. The unit of measure is the **number of values read**, and next to it stands a second number — **access**, that is, how many times a separate contiguous region was visited. The layout that keeps the fields of a record adjacent and the layout that keeps a column adjacent work in opposite ways on these two numbers; which one is cheap depends on what the question asks. - **SP11.** The dataset is fictional and is the dataset of the previous lessons: 3,199 rows, six columns, 1,329 subscribers, three periods. It is produced with the same seed and the output is identical from run to run. **It is a model.** - **SP12.** Both store layouts are modeled in memory; what is measured is not the storage medium but the number of values touched to produce the answer. **It is a model.** - **SP13.** In the row-based layout, touching a row means reading all six fields of that row; since the fields are written adjacent, a single field cannot be fetched separately. - **SP14.** In the columnar layout each column is a separate array, and for every block of 512 values a summary holding the smallest and the largest value is stored alongside the column. - **SP15.** Access is the number of visits to a separate contiguous region: a scan from start to finish is one access, and reading the individual positions of a column is as many accesses as there are positions. #### Two Layouts ```python # store.py — the same fictional dataset is held in two layouts and every read is counted. # MODEL: two layouts were modeled in place of a real store; the measure is how many # values are touched to produce the answer of a query. 3199 rows, 1329 subscribers. SEED = 1246 REGION = ("north", "east", "center", "south", "west") PERIODS = (("2024-01", 1329), ("2024-02", 1035), ("2024-03", 835)) DENSITY = (0, 1, 2, 2, 2, 3, 4, 2, 2, 1) # center-weighted measurement network HEADER = ("subscriber_id", "meter_id", "period", "region", "reading_m3", "tier") def rng(seed: int): s = seed & 0xFFFFFFFF def next_value() -> float: nonlocal s s = (1103515245 * s + 12345) & 0xFFFFFFFF return s / 4294967296 return next_value def tier(o: float) -> str: return "0-10" if o <= 10 else "10-25" if o <= 25 else "25-40" def records() -> list[tuple]: r, out = rng(SEED), [] for period, size in PERIODS: for i in range(size): o = round(5 + r() * 32.0, 1) out.append((10000 + i, f"S-{4000 + i}", period, REGION[DENSITY[i % 10]], o, tier(o))) return out class RowStore: """All fields of a record are adjacent: touching a row means reading all the fields of that row.""" def __init__(self, rows): self.rows, self.values_read, self.accesses = [tuple(map(str, x)) for x in rows], 0, 0 def scan(self): self.accesses += 1 # one contiguous pass from start to finish for s in self.rows: self.values_read += len(HEADER) yield s def record(self, i: int) -> tuple: self.values_read, self.accesses = self.values_read + len(HEADER), self.accesses + 1 return self.rows[i] class ColumnStore: """Each column is a separate array: only the requested column, even only the requested positions of that column, can be read. The block summary is stored with the column.""" BLOCK = 512 def __init__(self, rows): self.cols = {name: [str(x[j]) for x in rows] for j, name in enumerate(HEADER)} self.values_read, self.accesses, self.n = 0, 0, len(rows) self.summary = {name: [(min(v[i:i + self.BLOCK]), max(v[i:i + self.BLOCK])) for i in range(0, len(v), self.BLOCK)] for name, v in self.cols.items()} def read_all(self, name: str) -> list: self.values_read, self.accesses = self.values_read + self.n, self.accesses + 1 return self.cols[name] def read_at(self, name: str, positions: list) -> list: self.values_read += len(positions) # scattered access: every position separate self.accesses += len(positions) return [self.cols[name][i] for i in positions] def record(self, i: int) -> tuple: self.values_read += len(HEADER) # six separate visits to six columns self.accesses += len(HEADER) return tuple(self.cols[name][i] for name in HEADER) r0 = records() rs, cs = RowStore(r0), ColumnStore(r0) print(f"seed {SEED}; {len(r0)} rows, {len(HEADER)} columns, " f"{len(r0) * len(HEADER)} values") print(f"block size {ColumnStore.BLOCK}, {len(cs.summary['period'])} block summaries " f"in the period column") print("first record:", rs.record(0)) ``` ``` seed 1246; 3199 rows, 6 columns, 19194 values block size 512, 7 block summaries in the period column first record: ('10000', 'S-4000', '2024-01', 'north', '9.4', '0-10') ``` The number draws the limit of the dataset: six columns and 3,199 rows, 19,194 values in total. The row-based layout keeps these values adjacent record by record; the six fields of the first record sit side by side. The columnar layout spreads the same values over six separate arrays and keeps one summary for each array's seven blocks. The content of the dataset is the same; its placement is different. #### One Question, Four Read Paths ```python # query.py — a single question is answered in four ways: "the total reading per region in the given # period". The answer is the same on all four paths; what changes is the number of values read and # accesses, the number of visits to a separate contiguous region. def row_path(d: RowStore, target: str) -> dict: t = {} for s in d.scan(): # all six fields of the record come along if s[2] == target: t[s[3]] = round(t.get(s[3], 0.0) + float(s[4]), 1) return t def column_early(d: ColumnStore, target: str) -> dict: # three columns end to end period, region, reading = d.read_all("period"), d.read_all("region"), d.read_all("reading_m3") return aggregate([(region[i], reading[i]) for i, v in enumerate(period) if v == target]) def column_late(d: ColumnStore, target: str) -> dict: # filter first, then positions positions = [i for i, v in enumerate(d.read_all("period")) if v == target] return aggregate(list(zip(d.read_at("region", positions), d.read_at("reading_m3", positions)))) def column_block(d: ColumnStore, target: str) -> tuple[dict, int]: summary, positions, skipped = d.summary["period"], [], 0 d.values_read, d.accesses = d.values_read + 2 * len(summary), d.accesses + 1 for bi, (lo, hi) in enumerate(summary): if not lo <= target <= hi: # the block summary does not cover the target skipped += 1 continue start = bi * d.BLOCK piece = d.cols["period"][start:start + d.BLOCK] d.values_read, d.accesses = d.values_read + len(piece), d.accesses + 1 positions += [start + i for i, v in enumerate(piece) if v == target] return aggregate(list(zip(d.read_at("region", positions), d.read_at("reading_m3", positions)))), skipped def aggregate(pairs: list) -> dict: t = {} for b, o in pairs: t[b] = round(t.get(b, 0.0) + float(o), 1) return t def run(path: str, dataset: list, target: str) -> tuple: if path == "row-based scan": d = RowStore(dataset) t = row_path(d, target) else: d = ColumnStore(dataset) t = (column_early(d, target) if "early" in path else column_late(d, target) if "late" in path else column_block(d, target)[0]) return d, t PATHS = ("row-based scan", "columnar, early materialization", "columnar, late materialization", "columnar, block skipping") G = (34, 14, 9, 9, 11, 12) fmt = lambda h: "".join(str(v).rjust(G[j]) if j else str(v).ljust(G[0]) for j, v in enumerate(h)) print(fmt(["read path", "values read", "ratio", "accesses", "center m3", "total m3"])) base = 0 for path in PATHS: d, t = run(path, r0, "2024-03") base = base or d.values_read print(fmt([path, d.values_read, f"{base / d.values_read:.1f}x", d.accesses, t["center"], round(sum(t.values()), 1)])) print(f"on the block-skipping path {column_block(ColumnStore(r0), '2024-03')[1]} of 7 blocks " f"were never opened; 2024-03 has 835 rows") ``` ``` read path values read ratio accesses center m3 total m3 row-based scan 19194 1.0x 1 8752.6 17307.2 columnar, early materialization 9597 2.0x 3 8752.6 17307.2 columnar, late materialization 4869 3.9x 1671 8752.6 17307.2 columnar, block skipping 2835 6.8x 1674 8752.6 17307.2 on the block-skipping path 4 of 7 blocks were never opened; 2024-03 has 835 rows ``` All four paths give the same answer: 8,752.6 m³ in the center region and 17,307.2 m³ in total. The values read fall from 19,194 to 2,835 — **6.8 times**. The gain does not come from a single item; it comes from the stacking of three separate decisions, and each row of the table adds one decision. That the measure is values read is not a convenience but the choice that makes the comparison portable. The time of the same four paths changes from run to run with the machine worked on, the load at that moment and the state of the cache; the number of values read derives from the layout and the question and comes out the same on every run. How many times cheaper one path is than another is a defensible sentence in this unit. The first decision is column selection. The question asks about three columns; the row-based layout brings all six, because the fields of the record are adjacent. The columnar layout reads three, and the values read are halved. This is a difference of layout, not of product: every structure that keeps a column adjacent gives the same gain. The second decision is order. Early materialization reads all three columns from start to finish; late materialization first reads the filter column, finds the 835 matching positions and takes only those positions from the other two columns. The values read fall from 9,597 to 4,869. The cost is in the accesses column: 1,671 accesses instead of three, because scattered positions are now being read instead of a contiguous array. The third decision is the block summary. The period column had been divided into seven blocks of 512, and for each block the smallest and the largest value had been stored. If the target period does not fall within a block's range, that block is never opened; four of the seven blocks were not opened and the values read dropped from 4,869 to 2,835. The block summary added fourteen values to the reading and in return saved reading 2,048 values. #### What the Gain Depends On ```python # clustering.py — block skipping depends on the data being clustered on that column. # When the same dataset is written in subscriber order, every block carries all three # periods, no summary stays outside, and the skipping gain drops to zero. by_subscriber = sorted(r0, key=lambda x: (x[0], x[2])) r = rng(99) # our own generator with a visible seed shuffled = [x for _, x in sorted((r(), x) for x in r0)] G2 = (20, 14, 14, 10, 12) fmt2 = lambda h: "".join(str(v).rjust(G2[j]) if j else str(v).ljust(G2[0]) for j, v in enumerate(h)) print(fmt2(["dataset layout", "skipped blocks", "values read", "accesses", "total m3"])) for label, dataset in [("period order", r0), ("subscriber order", by_subscriber), ("shuffled (seed 99)", shuffled)]: d = ColumnStore(dataset) t, skipped = column_block(d, "2024-03") print(fmt2([label, f"{skipped}/{len(d.summary['period'])}", d.values_read, d.accesses, round(sum(t.values()), 1)])) # The selectivity of the filter decides which path is cheap. The formulas are read from the # measured paths themselves: the early path reads three columns end to end, the late path # reads the filter column and collects the remaining m positions one by one from two columns. n = len(r0) G3 = (22, 14, 13, 15, 12) fmt3 = lambda h: "".join(str(v).rjust(G3[j]) if j else str(v).ljust(G3[0]) for j, v in enumerate(h)) print(fmt3(["matching rows", "read early", "read late", "late accesses", "late gain"])) for m in (835, 1329, 2691): print(fmt3([f"{m} ({100 * m // n} percent)", 3 * n, n + 2 * m, 1 + 2 * m, f"{(3 * n) / (n + 2 * m):.2f}x"])) d = ColumnStore(r0) column_late(d, "2024-03") print(f"the measured late path at 835 rows: {d.values_read} values and {d.accesses} accesses; " f"formula {n + 2 * 835} and {1 + 2 * 835}") # The opposite direction: fetching all of a record. SAMPLE = list(range(0, n, 320)) rs2, cs2 = RowStore(r0), ColumnStore(r0) for i in SAMPLE: rs2.record(i) cs2.record(i) print(f"all fields of {len(SAMPLE)} records: row-based {rs2.values_read} values / {rs2.accesses} " f"accesses, columnar {cs2.values_read} values / {cs2.accesses} accesses") ``` ``` dataset layout skipped blocks values read accesses total m3 period order 4/7 2835 1674 17307.2 subscriber order 2/7 4244 1676 17307.2 shuffled (seed 99) 0/7 4883 1678 17307.2 matching rows read early read late late accesses late gain 835 (26 percent) 9597 4869 1671 1.97x 1329 (41 percent) 9597 5857 2659 1.64x 2691 (84 percent) 9597 8581 5383 1.12x the measured late path at 835 rows: 4869 values and 1671 accesses; formula 4869 and 1671 all fields of 10 records: row-based 60 values / 10 accesses, columnar 60 values / 60 accesses ``` The first table is the hidden default of the lesson. Block skipping looks like a property of the structure, but it depends on **the dataset being clustered on that column**. When the same dataset is written in subscriber order, all three periods are mixed into every block, the summaries cover the target, and only two of the seven blocks are skipped; in shuffled order none is skipped. The values read rise from 2,835 to 4,244 and from there to 4,883, and the 6.8-times gain falls to four times. The total comes out as 17,307.2 m³ in all three runs; the number the decision moves is not the answer but the values read to produce it. The question of which column to sort the dataset on is a layout decision, and it is invisible when the query is written. The second table draws the limit of late materialization. When the filter leaves 835 rows, the late path reads 1.97 times fewer values than the early path; when it leaves 2,691 rows the gain falls to 1.12 times and scattered access rises from 1,671 to 5,383. As selectivity drops the position list grows, scattered reading approaches contiguous reading, and the filter moved forward does not pay for itself. The formulas are read from the measured paths themselves, and for the 835-row case they match the measured value exactly: 4,869 values, 1,671 accesses. The last line measures the opposite direction. Fetching all of ten records reads 60 values in both layouts, but the row-based layout does it with 10 accesses and the columnar layout with 60. The columnar placement wins on questions that read few columns over many rows and loses on questions that read many columns over few rows. **Layout is not a convenience preference; it is the decision of which question will be answered cheaply.** #### Summary - The same question was answered by four paths and all four gave 8,752.6 m³ for the center and 17,307.2 m³ in total; the only thing that changed is the number of values read. - The row-based scan reads 19,194 values; the columnar layout reads 9,597 with the three needed columns, 4,869 by moving the filter forward, and 2,835 by skipping blocks — 6.8 times. - Late materialization lowers the values read but raises the accesses from three to 1,671; the gain depends on selectivity and falls to 1.12 times when the matching rows rise to 2,691. - Block skipping depends on clustering: in the period-ordered dataset four of the seven blocks are skipped, in the subscriber-ordered dataset two, in the shuffled dataset none, and the values read rise from 2,835 to 4,883. - Fetching all of a single record is 10 accesses in the row-based layout and 60 accesses in the columnar layout; the gain of the layout changes sign with the direction of the question. #### Next Step This lesson cut the values read by 6.8 times, but the whole computation was still done in a single place. As the dataset grows, the limit of that single place arrives: even if the values read drop to one sixth, when 3,199 rows become 3.2 billion rows the values a single machine would read still do not fit. The next lesson distributes the work across several partitions and measures the line item that distribution itself adds — how many units partitioning, shuffle and coordination add to the computation, and at what job size that line item turns into gain. ### Parallel and Distributed Dataframes Source: https://academia.sh/en/courses/data-analysis-with-python/parallel-and-distributed-dataframes When the 3,199-row dataset of the measurement network is split into partitions, the cost of distribution itself appears and stays larger than the work of a single machine: a five-partition distribution by the region key costs 11,651 units and a four-partition distribution by the subscriber key costs 8,921 units, while a single machine spends 3,199 units. The cost has three parts — the work units of the largest partition, the rows shuffled and the coordination per partition — and the point at which it turns into gain moves with the partitioning key: 10,833 rows with a balanced key, 20,119 rows with a skewed key. A query that is not aligned never sees this threshold; in the period partitioning 66.7% of the rows do not stay in place and the shuffle item alone rises to 6.7 times the work of a single machine. The previous lesson cut the values read by 6.8 times, but the whole computation was done in a single place. The limit of that place is unavoidable: even if the values read fall to one sixth, when the dataset grows a thousandfold the values a single machine would read still do not fit. This lesson distributes the work over several partitions and asks a single question — what does distribution itself add to the computation. The answer is a list of items, and the name of the list is **the cost of the transition**. Splitting the dataset into partitions, shuffling rows between partitions and coordinating the partitions are jobs of their own, separate from the work itself. The measure of the lesson counts these items in the same unit as the work of a single machine and puts two numbers side by side: the **gain**, that is, how many times cheaper distribution brought the work, and the **hidden default**, that is, the balance and the shuffle that are decided silently when the partitioning key is chosen. - **SP16.** The dataset is fictional and is the dataset of the previous lessons: 3,199 rows, 1,329 subscribers, three periods. It is produced with the same seed and the output is identical from run to run. **It is a model.** - **SP17.** The work unit is touching a row once; work on n rows is n units on a single machine. - **SP18.** Shuffling a row between partitions costs 10 units. This ratio is an assumption of the model; when it changes all the thresholds shift, and the direction of the shift is read in the table. **It is a model.** - **SP19.** Setting up a partition and collecting its result requires 2,000 units of coordination; p partitions add p times 2,000 units. - **SP20.** Partitions run concurrently; the critical path of the distributed job is the work units of **the largest partition**. The total cost is the sum of the critical path and the shuffle and coordination items. #### Three Partitionings of the Same Dataset ```python # partition.py — the same fictional dataset is split into partitions with three separate keys. MODEL: # in place of a real distribution, the partitioning, shuffle and coordination items are counted. # The unit of measure is touching a row once. SEED = 1246 REGION = ("north", "east", "center", "south", "west") PERIODS = (("2024-01", 1329), ("2024-02", 1035), ("2024-03", 835)) DENSITY = (0, 1, 2, 2, 2, 3, 4, 2, 2, 1) # center-weighted measurement network SHUFFLE = 10 # shuffling one row to another partition costs ten local touches COORDINATION = 2000 # setting up a partition and collecting its result def rng(seed: int): s = seed & 0xFFFFFFFF def next_value() -> float: nonlocal s s = (1103515245 * s + 12345) & 0xFFFFFFFF return s / 4294967296 return next_value def records() -> list[tuple]: r, out = rng(SEED), [] for period, size in PERIODS: for i in range(size): out.append((10000 + i, f"S-{4000 + i}", period, REGION[DENSITY[i % 10]], round(5 + r() * 32.0, 1))) return out def partition_by(rows: list, key) -> dict: parts = {} for x in rows: parts.setdefault(key(x), []).append(x) return parts r0 = records() KEYS = (("region", lambda x: x[3]), ("subscriber mod 4", lambda x: x[0] % 4), ("period", lambda x: x[2])) G = (18, 12, 30, 14, 12) fmt = lambda h: "".join(str(v).rjust(G[j]) if j else str(v).ljust(G[0]) for j, v in enumerate(h)) print(f"dataset {len(r0)} rows, {len({x[0] for x in r0})} subscribers; shuffle {SHUFFLE} " f"units/row, coordination {COORDINATION} units/partition") print(fmt(["partition key", "partitions", "partition sizes", "largest", "skew"])) for name, f in KEYS: p = partition_by(r0, f) size = sorted((len(v) for v in p.values()), reverse=True) print(fmt([name, len(p), str(size), size[0], f"{size[0] / (len(r0) / len(p)):.2f}x"])) ``` ``` dataset 3199 rows, 1329 subscribers; shuffle 10 units/row, coordination 2000 units/partition partition key partitions partition sizes largest skew region 5 [1601, 639, 321, 319, 319] 1601 2.50x subscriber mod 4 4 [801, 800, 800, 798] 801 1.00x period 3 [1329, 1035, 835] 1329 1.25x ``` All three keys are defensible and the three give a different layout. Partitioning by the remainder of the subscriber number is balanced: four partitions between 798 and 801 rows, skew 1.00. Partitioning by region is skewed: the center partition carries 1,601 rows, 2.50 times the average. The period key stands in between. Skew is written directly onto the critical path. Because partitions run concurrently, the job finishes when the largest partition finishes; a dataset split into five partitions falls not to one fifth but to one half if one of the partitions carries half of the dataset. The partitioning key looks like a detail of arrangement, yet it sets the upper limit of the gain expected from distribution. #### The Items of the Transition ```python # transition.py — two queries are run over three partitionings and the items of the # transition are counted separately: the critical path (work units of the largest partition), # the rows shuffled and the coordination. Total = critical path + shuffled * SHUFFLE + partitions * COORDINATION. def build_index(p: dict) -> dict: return {name: i for i, name in enumerate(sorted(p, key=str))} def is_aligned(p: dict, identity) -> bool: where = {} for name, v in p.items(): for x in v: where.setdefault(identity(x), set()).add(name) return all(len(s) == 1 for s in where.values()) def redistribute(p: dict, identity) -> tuple[dict, int]: """The dataset is re-partitioned by the identity key and the number of partitions is kept. Every row whose target partition differs from its source partition is shuffled.""" ix, n_parts = build_index(p), len(p) new, shuffled = {i: [] for i in range(n_parts)}, 0 for name, v in p.items(): for x in v: h = identity(x) % n_parts new[h].append(x) shuffled += h != ix[name] return new, shuffled cost = lambda p, critical, shuffled: critical + shuffled * SHUFFLE + p * COORDINATION G2 = (18, 12, 10, 15, 14, 13, 9) fmt2 = lambda h: "".join(str(v).rjust(G2[j]) if j else str(v).ljust(G2[0]) for j, v in enumerate(h)) print("Query A: total reading per region (can be partially aggregated)") print(fmt2(["partitioning", "partitions", "shuffled", "critical path", "coordination", "distributed", "ratio"])) for name, f in KEYS: p = partition_by(r0, f) shuffled = sum(len({x[3] for x in v}) for v in p.values()) # partial results critical = max(len(v) for v in p.values()) m = cost(len(p), critical, shuffled) print(fmt2([name, len(p), shuffled, critical, len(p) * COORDINATION, m, f"{len(r0) / m:.2f}x"])) print("Query B: number of subscribers whose increase across periods exceeds 5 m3 (needs subscriber integrity)") print(fmt2(["partitioning", "partitions", "shuffled", "critical path", "coordination", "distributed", "ratio"])) for name, f in KEYS: p = partition_by(r0, f) if is_aligned(p, lambda x: x[0]): shuffled, critical = len(p), max(len(v) for v in p.values()) else: new, shuffled = redistribute(p, lambda x: x[0]) shuffled, critical = shuffled + len(p), max(len(v) for v in new.values()) m = cost(len(p), critical, shuffled) print(fmt2([name, len(p), shuffled, critical, len(p) * COORDINATION, m, f"{len(r0) / m:.2f}x"])) ``` ``` Query A: total reading per region (can be partially aggregated) partitioning partitions shuffled critical path coordination distributed ratio region 5 5 1601 10000 11651 0.27x subscriber mod 4 4 12 801 8000 8921 0.36x period 3 15 1329 6000 7479 0.43x Query B: number of subscribers whose increase across periods exceeds 5 m3 (needs subscriber integrity) partitioning partitions shuffled critical path coordination distributed ratio region 5 5 1601 10000 11651 0.27x subscriber mod 4 4 4 801 8000 8841 0.36x period 3 2136 1067 6000 28427 0.11x ``` Both tables run on the same dataset with the same number of partitions, and in both the distributed cost stays above the 3,199 units of the single machine. At this scale distribution does not pay; the job of the table is to show where the loss comes from. Query A can be partially aggregated: each partition computes its own region totals and sends only the partial result. The rows shuffled are between 5 and 15 and are **independent of the dataset size** — they are the product of the number of partitions and the number of groups. Almost all of the cost is coordination: 10,000 of the 11,651 units with the region key, 8,000 of the 8,921 units with the subscriber key. Query B wants all the rows of a subscriber to be in the same partition, and here the partitioning key decides the result. The region and subscriber keys are aligned — every subscriber is in a single region — and the rows shuffled stay as many as the number of partitions. The period key is not aligned: the three periods of the same subscriber are in three separate partitions, the dataset is redistributed and 2,136 rows are shuffled. The cost rises from 7,479 to 28,427 units, four times. **This is the hidden default:** the partitioning key is chosen before the query, and that choice determines the shuffle item of the query. #### The Threshold Where It Turns into Gain ```python # scale.py — the same model is solved at growing dataset sizes. The partition ratios are # taken from the measured dataset; in the query that can be partially aggregated the rows shuffled # are independent of n, in the unaligned query they grow with n. import math RATIO = {name: max(len(v) for v in partition_by(r0, f).values()) / len(r0) for name, f in KEYS} _p = partition_by(r0, dict(KEYS)["period"]) _new, _moved = redistribute(_p, lambda x: x[0]) SHARE = _moved / len(r0) # the share that does not stay in place in the period partitioning B_RATIO = max(len(v) for v in _new.values()) / len(r0) SCENARIOS = (("A / subscriber mod 4", 4, RATIO["subscriber mod 4"], 12, 0.0), ("A / region", 5, RATIO["region"], 5, 0.0), ("B / period", 3, B_RATIO, 3, SHARE)) G3 = (14, 16, 20, 12, 12, 12) fmt3 = lambda h: "".join(str(v).rjust(G3[j]) if j else str(v).ljust(G3[0]) for j, v in enumerate(h)) def distributed(n, p, ratio, fixed, share): shuffled = fixed + share * n return round(ratio * n + shuffled * SHUFFLE + p * COORDINATION) print(fmt3(["dataset n", "single machine", "A/subscriber mod 4", "A/region", "B/period", "best ratio"])) for n in (3199, 31990, 319900, 3199000): line = [distributed(n, p, o, s, share) for _, p, o, s, share in SCENARIOS] print(fmt3([n, n, line[0], line[1], line[2], f"{n / min(line):.2f}x"])) print(fmt3(["threshold n", "-", *[ (math.ceil((s * SHUFFLE + p * COORDINATION) / (1 - o)) if share * SHUFFLE + o < 1 else "never") for _, p, o, s, share in SCENARIOS], "-"])) print(f"in the period partitioning {100 * SHARE:.1f} percent of the rows do not stay in place; " f"the shuffle item alone is {SHARE * SHUFFLE:.1f} times the work of a single machine") ``` ``` dataset n single machine A/subscriber mod 4 A/region B/period best ratio 3199 3199 8921 11651 28427 0.36x 31990 31990 16130 26060 230000 1.98x 319900 319900 88220 170150 2245730 3.63x 3199000 3199000 809120 1611050 22403030 3.95x threshold n - 10833 20119 never - in the period partitioning 66.7 percent of the rows do not stay in place; the shuffle item alone is 6.7 times the work of a single machine ``` The table shows where distribution turns into gain. Coordination is a fixed item independent of the dataset size; as the dataset grows its share melts away and the critical path becomes dominant. With the balanced key the threshold is 10,833 rows: below this size distributing is expensive, above it it becomes cheap. With the skewed key the threshold rises to 20,119 rows — the same model, the same constants, only the partitioning key is different. Skew nearly doubles the threshold. The third column never sees the threshold. In the unaligned query the rows shuffled grow with the dataset size: 66.7% of the rows do not stay in place and the shuffle item alone rises to 6.7 times the work of a single machine. However large n grows the ratio stays constant and the distributed cost stays above the single machine. Scaling up does not save this query; the only thing that saves it is partitioning the dataset by the right key. The gain side is limited as well. At 3,199,000 rows the best option costs 809,120 units and gains 3.95 times over the single machine — a little under the four times expected from a job split into four partitions. The difference comes entirely from the shuffle and coordination items, and these items are not zeroed at any dataset size. **Increasing the number of partitions does not increase the gain as many times as the number of partitions; the coordination item grows with the number of partitions.** #### The Limit of the Partition Count ```python # partition_count.py — the same job is distributed with different partition counts. The critical path # shrinks with the partition count, the coordination grows with it; the best partition count # is where the two balance. A balanced key is assumed. N = 3199000 G4 = (12, 15, 12, 15, 14, 10) fmt4 = lambda h: "".join(str(v).rjust(G4[j]) if j else str(v).ljust(G4[0]) for j, v in enumerate(h)) total = lambda p: math.ceil(N / p) + (p * 5) * SHUFFLE + p * COORDINATION print(fmt4(["partitions", "critical path", "shuffled", "coordination", "distributed", "ratio"])) for p in (2, 4, 16, 40, 64, 256): print(fmt4([p, math.ceil(N / p), p * 5, p * COORDINATION, total(p), f"{N / total(p):.1f}x"])) best = min(range(1, 1001), key=total) print(f"for n = {N} the best partition count is {best}; cost {total(best)} units, " f"gain {N / total(best):.1f} times") print(f"if {4 * best} is chosen instead of {best} partitions the cost rises to {total(4 * best)} " f"units and the gain falls to {N / total(4 * best):.1f} times") ``` ``` partitions critical path shuffled coordination distributed ratio 2 1599500 10 4000 1603600 2.0x 4 799750 20 8000 807950 4.0x 16 199938 80 32000 232738 13.7x 40 79975 200 80000 161975 19.7x 64 49985 320 128000 181185 17.7x 256 12497 1280 512000 537297 6.0x for n = 3199000 the best partition count is 40; cost 161975 units, gain 19.7 times if 160 is chosen instead of 40 partitions the cost rises to 347994 units and the gain falls to 9.2 times ``` The partition count moves two items in opposite directions. The critical path is divided by the partition count and the coordination is multiplied by it; while the first falls the second rises, and the total takes its smallest value somewhere. For a dataset of 3,199,000 rows that place is 40 partitions: cost 161,975 units, gain 19.7 times. With four partitions the gain is 4.0 times and with 256 partitions it is 6.0 times — past a certain point increasing the partition count lowers the gain. The number itself depends on the constants of the model, and the constants are written in SP18 and SP19; if the shuffle cost or the coordination changes, the best partition count changes too. What does not change is the shape: the gain of distribution is not proportional to the partition count and it has a peak. When four times the best partition count is chosen the cost rises to 347,994 units and the gain falls to 9.2 times: four times as many partitions means twice the cost. The partition count too is not a default but a decision that has to be written down. #### Summary - The cost of a distributed computation is three items: the work units of the largest partition, the rows shuffled times the shuffle cost, and the coordination per partition. The partitioning key determines the partition balance: the subscriber remainder gives 1.00 skew and a critical path of 801 rows, the region key gives 2.50 skew and a critical path of 1,601 rows. - At 3,199 rows distributing does not pay; the best option costs 8,921 units while a single machine spends 3,199 units. - In the query that can be partially aggregated the rows shuffled are independent of the dataset size (5–15 rows); in the query that needs integrity an unaligned key shuffles 2,136 rows and raises the cost fourfold. - The threshold where it turns into gain is 10,833 rows with the balanced key and 20,119 rows with the skewed key; with the unaligned query there is no such threshold. - At 3,199,000 rows a four-partition distribution gains 3.95 times; on the same dataset the best partition count is 40 and the gain saturates at 19.7 times, falling back to 6.0 times at 256 partitions. #### Next Step This lesson moved the work to several partitions, but all the items were still built on **bringing** the data to the analysis environment: rows are read, shuffled, split. If the place where the data sits can also compute, the order can be reversed — the filter and the aggregation are pushed to the source and only the result comes to the analysis environment. The next lesson measures this with a single number: how many rows leave the source for the same question, and which decision is no longer in your hands once the aggregation is done at the source. ### Taking the Query to the Data Source: https://academia.sh/en/courses/data-analysis-with-python/taking-the-query-to-the-data The source of the measurement network holds 3,199 rows and a single question is answered with four strategies; all four give the same number, but the number of records transferred to the analysis environment falls from 3,199 to 5. When the filter is pushed to the source 835 rows are transferred, when column selection is pushed too 1,670 values, when aggregation is pushed as well 10 values, and the gain reaches 1,919 times. The price of the gain is visibility: the pushed total does not bring the 835 rows that passed the filter, the 3,199 rows in the source or the number of rows per group, and writing the decision next to the answer raises the values transferred from 10 to 27 and lowers the ratio to 710 times. The gain also depends on the number of groups: 3,199 times with 3 groups, 426 times with 15 groups, 11 times with 835 groups, and it is zeroed when the number of groups equals the number of filtered rows. The previous lesson moved the work to several partitions and counted the cost of distribution itself. But that whole computation was built in a single direction: the data was **brought** to the analysis environment. Rows were read, split, shuffled and in the end the answer emerged. If the place where the data sits can also compute, this direction can be reversed. The measure of this lesson is a single number: **rows transferred**. How many records leave the source for the same question, and how far does this number fall when the filter and the aggregation are pushed to the source. The second number standing beside it is the item that becomes invisible in return for that gain — a pushed computation brings the answer, it does not bring the rows that produced the answer. - **SP21.** The source is fictional and holds the dataset of the previous lessons: 3,199 rows, six columns, 1,329 subscribers, three periods. It is produced with the same seed and the output is identical from run to run. **It is a model.** - **SP22.** The source is modeled as an interface that can compute where the data sits: the filter, the column selection and the aggregation can be run there. **It is a model.** - **SP23.** What is measured is the number of rows and values passing from the source to the analysis environment; how many rows the source scans inside is outside this measure. - **SP24.** For two totals to count as the same, the threshold is 0.05 m³, because readings are reported with one decimal and a difference smaller than that can come only from rounding. - **SP25.** The question is fixed: the total reading per region in the period 2024-03. This period has 835 rows and the correct answer is 17,307.2 m³. #### The Source and the Question ```python # source.py — the place where the data sits is modeled as an interface. MODEL: # in place of a real remote store, a source was written that counts every row and value # it returns for the work asked of it. The measure is the rows and values transferred to the analysis environment. SEED = 1246 REGION = ("north", "east", "center", "south", "west") PERIODS = (("2024-01", 1329), ("2024-02", 1035), ("2024-03", 835)) DENSITY = (0, 1, 2, 2, 2, 3, 4, 2, 2, 1) # center-weighted measurement network HEADER = ("subscriber_id", "meter_id", "period", "region", "reading_m3", "tier") def rng(seed: int): s = seed & 0xFFFFFFFF def next_value() -> float: nonlocal s s = (1103515245 * s + 12345) & 0xFFFFFFFF return s / 4294967296 return next_value def tier(o: float) -> str: return "0-10" if o <= 10 else "10-25" if o <= 25 else "25-40" def records() -> list[tuple]: r, out = rng(SEED), [] for period, size in PERIODS: for i in range(size): o = round(5 + r() * 32.0, 1) out.append((10000 + i, f"S-{4000 + i}", period, REGION[DENSITY[i % 10]], o, tier(o))) return out class Source: """Returns whatever is asked and counts every row and value it returns. The filter, the column selection and the aggregation can be done at the source.""" def __init__(self, rows: list): self.rows, self.rows_out, self.values_out = rows, 0, 0 def _serve(self, rows: list, width: int) -> list: self.rows_out += len(rows) self.values_out += len(rows) * width return rows def all_rows(self) -> list: return self._serve(list(self.rows), len(HEADER)) def filter_rows(self, predicate, fields=None) -> list: s = [x for x in self.rows if predicate(x)] if fields is None: return self._serve(s, len(HEADER)) ix = [HEADER.index(a) for a in fields] return self._serve([tuple(x[i] for i in ix) for x in s], len(fields)) def count(self, predicate) -> int: return self._serve([sum(1 for x in self.rows if predicate(x))], 1)[0] def sum_wide(self, predicate, group: str, measure: str) -> list: """Puts the count, the minimum and the maximum next to the total: five values.""" gi, mi = HEADER.index(group), HEADER.index(measure) t = {} for x in self.rows: if predicate(x): a = t.setdefault(x[gi], [0.0, 0, x[mi], x[mi]]) a[0], a[1] = a[0] + x[mi], a[1] + 1 a[2], a[3] = min(a[2], x[mi]), max(a[3], x[mi]) return self._serve([(g, *v) for g, v in sorted(t.items())], 5) def sum_by(self, predicate, group: str, measure: str) -> list: gi, mi = HEADER.index(group), HEADER.index(measure) t = {} for x in self.rows: if predicate(x): t[x[gi]] = t.get(x[gi], 0.0) + x[mi] return self._serve(sorted(t.items()), 2) r0 = records() TARGET = "2024-03" print(f"seed {SEED}; {len(r0)} rows, {len(HEADER)} columns, " f"{len(r0) * len(HEADER)} values in the source") print(f"question: total reading per region in the period {TARGET}; this period has " f"{sum(a for d, a in PERIODS if d == TARGET)} rows in the source") ``` ``` seed 1246; 3199 rows, 6 columns, 19194 values in the source question: total reading per region in the period 2024-03; this period has 835 rows in the source ``` The source can do three jobs: filter rows, select columns and take the total per group. All of these can also be done in the analysis environment; the difference is only in **where** they are done. The interface counts every returned row and every returned value, because what is to be measured is not the computation itself but with how many records the result of the computation is expressed. How many rows the source scans inside is outside this number, and SP23 keeps that apart. #### Four Degrees of Pushing ```python # push.py — the same question is answered with four strategies. The difference is where the # work is done: everything at the client, the filter at the source, column narrowing at the source, # and the aggregation at the source too. THRESHOLD = 0.05 # readings are reported with one decimal, a difference below that is rounding def client_sum(rows, gi: int, mi: int) -> dict: t = {} for x in rows: t[x[gi]] = t.get(x[gi], 0.0) + x[mi] return t def s1(src: Source) -> dict: # nothing is pushed return client_sum([x for x in src.all_rows() if x[2] == TARGET], 3, 4) def s2(src: Source) -> dict: # the filter is pushed return client_sum(src.filter_rows(lambda x: x[2] == TARGET), 3, 4) def s3(src: Source) -> dict: # the filter and the column selection are pushed return client_sum( src.filter_rows(lambda x: x[2] == TARGET, ["region", "reading_m3"]), 0, 1) def s4(src: Source) -> dict: # the aggregation is pushed too return dict(src.sum_by(lambda x: x[2] == TARGET, "region", "reading_m3")) G = (40, 18, 19, 10, 12, 12) fmt = lambda h: "".join(str(v).rjust(G[j]) if j else str(v).ljust(G[0]) for j, v in enumerate(h)) print(fmt(["strategy", "rows transferred", "values transferred", "ratio", "center m3", "total m3"])) base = 0 for name, f in [("nothing is pushed", s1), ("filter is pushed", s2), ("filter and column selection are pushed", s3), ("aggregation is pushed too", s4)]: src = Source(r0) t = f(src) base = base or src.values_out print(fmt([name, src.rows_out, src.values_out, f"{base / src.values_out:.0f}x", round(t["center"], 1), round(sum(t.values()), 1)])) src1, src4 = Source(r0), Source(r0) t1, t4 = s1(src1), s4(src4) diff = max(abs(t1[g] - t4[g]) for g in t1) print(f"largest difference between the region totals of the unpushed and the pushed computation " f"{diff:.1e} m3; since the threshold is {THRESHOLD} m3 they count as the same number. Because the " f"order of floating-point summation can change, equality is tested with the threshold, not with ==") ``` ``` strategy rows transferred values transferred ratio center m3 total m3 nothing is pushed 3199 19194 1x 8752.6 17307.2 filter is pushed 835 5010 4x 8752.6 17307.2 filter and column selection are pushed 835 1670 11x 8752.6 17307.2 aggregation is pushed too 5 10 1919x 8752.6 17307.2 largest difference between the region totals of the unpushed and the pushed computation 0.0e+00 m3; since the threshold is 0.05 m3 they count as the same number. Because the order of floating-point summation can change, equality is tested with the threshold, not with == ``` All four strategies give the same answer: 8,752.6 m³ in the center region and 17,307.2 m³ in total. The rows transferred fall from 3,199 to 5 and the values transferred from 19,194 to 10 — **1,919 times**. The gain is the sum of three separate pushing decisions, and each row adds one decision. Pushing the filter lowers the rows transferred from 3,199 to 835; the ratio comes directly from the selectivity. Pushing the column selection does not change the number of rows but lowers the number of values from 5,010 to 1,670, because two fields come instead of six. Pushing the aggregation, on the other hand, changes the notion of a row: what is transferred is no longer a record but a result — five groups, five rows. The last line compares the two totals. The largest difference between the region totals of the unpushed and the pushed computation is zero, but the comparison is still made with a threshold. The reason is not the result of the run but that floating-point addition is not associative: when the same numbers are added in another order the last digit can move. In what order the source adds up inside a group is outside the interface and is not written down; so equality is not tested directly, a threshold is written. #### The Price of the Gain ```python # audit.py — a pushed computation brings the result, it does not bring the decision that produced it. # First it is counted how many more values are needed to be able to write the decision next to the answer, # then how the gain of pushing melts away with the number of groups. BASE = len(r0) * len(HEADER) src = Source(r0) wide = src.sum_wide(lambda x: x[2] == TARGET, "region", "reading_m3") filtered, total_rows = src.count(lambda x: x[2] == TARGET), src.count(lambda x: True) G2 = (34, 18, 19, 10) fmt2 = lambda h: "".join(str(v).rjust(G2[j]) if j else str(v).ljust(G2[0]) for j, v in enumerate(h)) print(fmt2(["written next to the answer", "rows transferred", "values transferred", "ratio"])) print(fmt2(["total only", 5, 10, f"{BASE // 10}x"])) print(fmt2(["total, count, minimum, maximum", len(wide), 5 * len(wide), f"{BASE // (5 * len(wide))}x"])) print(fmt2(["plus filtered and total rows", src.rows_out, src.values_out, f"{BASE // src.values_out}x"])) for g, total, count, lo, hi in wide: if g in ("center", "west"): print(f"{g:8s} total {total:8.1f} m3, {count:4d} rows, average " f"{total / count:5.2f} m3, range {lo}-{hi} m3") print(f"rows that passed the filter {filtered}, total rows in the source {total_rows}; the pushed " f"total does not bring these two numbers and does not bring the number of rows per group either") # The gain of pushing depends on the number of groups: the number of result rows equals the number of groups. print(fmt2(["grouping key", "groups", "values transferred", "ratio"])) for name, field in [("tier", "tier"), ("region", "region"), ("region + tier", None), ("subscriber_id", "subscriber_id")]: src2 = Source(r0) if field is None: t = {} for x in r0: if x[2] == TARGET: t[(x[3], x[5])] = t.get((x[3], x[5]), 0.0) + x[4] src2._serve(sorted(t.items()), 3) else: src2.sum_by(lambda x: x[2] == TARGET, field, "reading_m3") print(fmt2([name, src2.rows_out, src2.values_out, f"{BASE // src2.values_out}x"])) raw = Source(r0) raw.filter_rows(lambda x: x[2] == TARGET, ["region", "reading_m3"]) print(f"the raw slice, filtered and narrowed, carries {raw.values_out} values; the aggregation " f"pushed with the subscriber key carries {2 * filtered} values") print(f"when the number of groups equals the number of filtered rows the gain of pushing is zeroed: " f"{filtered} groups, {filtered} rows") ``` ``` written next to the answer rows transferred values transferred ratio total only 5 10 1919x total, count, minimum, maximum 5 25 767x plus filtered and total rows 7 27 710x center total 8752.6 m3, 418 rows, average 20.94 m3, range 5.0-37.0 m3 west total 1789.1 m3, 83 rows, average 21.56 m3, range 5.0-37.0 m3 rows that passed the filter 835, total rows in the source 3199; the pushed total does not bring these two numbers and does not bring the number of rows per group either grouping key groups values transferred ratio tier 3 6 3199x region 5 10 1919x region + tier 15 45 426x subscriber_id 835 1670 11x the raw slice, filtered and narrowed, carries 1670 values; the aggregation pushed with the subscriber key carries 1670 values when the number of groups equals the number of filtered rows the gain of pushing is zeroed: 835 groups, 835 rows ``` The first table is the hidden default of the lesson. The pushed aggregation brings ten values, and next to those ten values no decision is written. Whether the 8,752.6 m³ of the center region came from 418 rows or from 4 rows is not in the result; that 835 rows passed the filter is not in it either, nor that there are 3,199 rows in the source. Asking for these items too raises the values transferred from 10 to 27 and lowers the gain from 1,919 times to 710 times. **The rule of the course is paid here:** a number whose decision is not written is an unmeasured number, and the price of writing that decision at this source is 17 values. The numbers themselves carry a decision too. The center region gives an average of 20.94 m³ over 418 rows and the west region an average of 21.56 m³ over 83 rows. The totals are 8,752.6 and 1,789.1 m³ and there is a fivefold difference between them; the averages are close to each other. A report that sees only the total reads the center region as "high consumption", yet the difference comes not from consumption but from the number of rows. As long as the pushed computation does not bring the number of rows, this distinction cannot be made in the report. The second table shows what the gain depends on. The rows carried by the pushed aggregation are **as many as the number of groups**; as the number of groups grows the gain melts away. The ratio is 3,199 times with a breakdown into three tiers, 1,919 times with a breakdown into five regions, 426 times with the cross of region and tier, and 11 times with a breakdown by subscriber. The last row gives exact equality: aggregation with the subscriber key carries 1,670 values, and the filtered raw slice narrowed to two columns carries 1,670 values too. **When the number of groups equals the number of filtered rows the gain of pushing is zeroed** — 835 groups, 835 rows. Past this point it is better to pull the raw slice, because the raw rows that come at the same price can answer other questions as well. #### Summary - The same question was answered at four degrees of pushing and all four gave 17,307.2 m³; the rows transferred fell from 3,199 to 5, the values transferred from 19,194 to 10 and the gain was 1,919 times. - The items separate: pushing the filter lowers the rows to 835, pushing the column selection lowers the values to 1,670, pushing the aggregation lowers the rows to 5. - The pushed total does not bring the 835 rows that passed the filter, the 3,199 rows in the source or the number of rows per group; asking for these items too raises the values transferred to 27 and lowers the ratio to 710 times. - The 8,752.6 m³ of the center region comes from 418 rows and the 1,789.1 m³ of the west from 83 rows; the averages are 20.94 and 21.56 m³ and the difference in the totals arises not from consumption but from the number of rows. - The gain of pushing depends on the number of groups: 3,199 times with 3 groups, 426 times with 15 groups, 11 times with 835 groups; when the number of groups equals the number of filtered rows the gain is zeroed. - Even if the difference between two totals comes out zero, equality is tested with a threshold, because in what order the source adds up inside a group is outside the interface. #### Next Step This topic followed a single question through five lessons: when the dataset does not fit in memory, what remains of the comfort of abstraction. The answer came in every lesson by changing the tool. Type choice shrank the column 47.9 times, chunking lowered the memory peak 32 times, the columnar layout cut the values read 6.8 times, distribution gained up to 19.7 times on a large dataset and pushing the computation to the data cut the records transferred 1,919 times. The scale problem was solved every time by changing **the layout, the partitioning or the place where the computation is done**. The only thing that did not change was the language itself. For twenty lessons the same form of expression has been used: loops, assignments, intermediate variables and computations built step by step. The next topic questions this assumption and looks at environments that express the analysis differently — it counts in how many steps the same analysis is described, how many intermediate variables the transformation chain holds and with how many layers a chart is built. The measure is the same again: what the abstraction gains and what it hides. ## Alternative Analysis Environments Counting how the same analysis is expressed in other environments and in which criterion the choice changes: how many steps analysis-oriented languages need for the same job, the intermediate variable and step counts of verb-based transformation chains, how many layers the same chart takes in a layered visualization approach, and a numeric comparison of three environments on team, ecosystem and production-readiness criteria. ### Statistical Programming Languages Source: https://academia.sh/en/courses/data-analysis-with-python/statistical-programming-languages The same analysis — mean consumption per region and the top three regions in descending order, on the 3,199-row fiction dataset — is written in three expression styles and all three give the same number: the procedural expression holds 6 statements, 6 intermediate variables and 2 loops; the whole-object expression holds 4 statements, 3 intermediate variables and 0 loops; the declarative expression holds 1 statement, 0 intermediate variables and 0 loops. Written loops drop from 2 to 0 while executed loops rise from 2 to 3: the loop does not disappear, it moves into a helper, and one more is added. The single-statement expression writes 4 decisions and leaves 5 decisions to defaults; when the five defaults are changed one by one, the east mean becomes 25.593, 24.814, 24.120 and 25.540 m³ instead of 24.674 m³, and when the sort direction changes, the reported three regions change completely. The previous topic solved the scale problem by changing tools: the column was read, processed piece by piece, and the computation was taken to the data. In every step, what changed was where the tool was placed. The only thing that stayed the same was **the language itself**; the analysis was written from start to finish in the same style, with the same statements, and it was never asked whether this style was a choice. This topic asks that question. The same analysis is expressed in other environments in other styles, and the difference between them is not speed but **the length of the expression**. This is the first number of this lesson: how many statements, how many intermediate variables and how many loops the same analysis is written with. Beside it stands the second number of the course: as the expression gets shorter, how many decisions move out of the code and into a default. - **AE1.** The dataset is inherited from Data Collection and Preparation and is **fiction**: a long-format reading table of 3,199 rows, 1,329 subscribers, 10/25/40 m³ tariff tiers. The seed is written in the code and the numbers are the same from run to run. - **AE2.** In this lesson **no language, library or environment name appears**. Alternative environments are referred to by their **approaches**, and the approach is **modeled** in Python. The purpose of the model is not to imitate a tool but to make the expression style measurable. - **AE3.** The three approaches are as follows. In the **procedural expression** the unit of work is the row: the data is walked through one element at a time. The **whole-object expression** is the approach of analysis-oriented languages and its unit of work is the entire column: no loop is written, the operation is applied to the whole column. In the **declarative expression** the definition of the result is written, and the order of execution is not. - **AE4.** The measure is **not a comparative superiority**. No ranking is built that says a short expression is good and a long expression is bad. The only things counted are statements, intermediate variables, loops and defaults. #### The Same Analysis, Three Expressions The analysis is defined in a single sentence: mean consumption per region, the top three regions in descending order. The dataset is the input of all three expressions and the seed is fixed. ```python # environment.py — MODEL: the 3,199-row fiction dataset of Data Collection and # Preparation is regenerated from the same seed; no tool, library or # environment is imitated. import ast import inspect import math SEED = 20260218 RAW = 1400 PERIODS = ["2026-01", "2026-02", "2026-03"] TARIFF = [10, 25, 40] M32 = 0xFFFFFFFF def mix(x: int) -> int: x = ((x ^ (x >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 return (x ^ (x >> 16)) & M32 def rng(t: int): s = [mix(t)] def next_value() -> float: s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u: float, w: list[float]) -> int: t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 REGION = [("north", 0.28, 21), ("south", 0.22, 17), ("east", 0.18, 26), ("west", 0.14, 14), ("central", 0.18, 23)] _subscribers, _dropped = [], [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] record = {"subscriber_id": 10001 + i, "region": b[0], "base": b[2], "household_size": pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]), "satisfaction": pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) + 1} (_dropped if r() < 0.046 else _subscribers).append(record) _valid = {a["subscriber_id"] for a in _subscribers} READINGS = [] for k in _subscribers + _dropped: r = rng(SEED + 7000 + k["subscriber_id"]) for d in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): t = 0.0 if r() < 0.038 else math.floor( k["base"] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - d * 0.05) * 100 + 0.5) / 100 if k["subscriber_id"] not in _valid: r() continue READINGS.append({"subscriber_id": k["subscriber_id"], "region": k["region"], "period": PERIODS[d], "consumption_m3": t, "satisfaction": k["satisfaction"], "tier": 1 if t < TARIFF[0] else 2 if t < TARIFF[1] else 3 if t < TARIFF[2] else 4}) r() print(f"fictional dataset (seed {SEED}): {len(READINGS)} rows, {len(_valid)} subscribers, " f"{sum(o['consumption_m3'] for o in READINGS):.1f} m3") print(f"unread rows (zero written): {sum(1 for o in READINGS if o['consumption_m3'] == 0)}") ``` ``` fictional dataset (seed 20260218): 3199 rows, 1329 subscribers, 63060.6 m3 unread rows (zero written): 119 ``` All three expressions take the same dataset. The measurement is made not by hand but from the code's own tree: the source of each expression is parsed, and the statements in its body, the names assigned and the loop nodes are counted. This way the answer to the question "how many steps" is not a comment but a countable quantity. ```python # Helpers of the whole-object approach: the operation is applied to the entire column. def column(rows, name): return [r[name] for r in rows] def group_mean(keys, measure): total, count = {}, {} for a, v in zip(keys, measure): total[a] = total.get(a, 0.0) + v count[a] = count.get(a, 0) + 1 return {a: total[a] / count[a] for a in total} def descending(mapping): return sorted(mapping.items(), key=lambda p: -p[1]) def top_n(pairs, n): return pairs[:n] # Solver of the declarative approach: the request is a definition, the order of execution is here. def solve(request): keys = column(request["source"], request["group"]) values = column(request["source"], request["measure"]) return top_n(descending(group_mean(keys, values)), request["top"]) HELPERS = {"column": column, "group_mean": group_mean, "descending": descending, "top_n": top_n, "solve": solve} def procedural(rows): total, count = {}, {} for row in rows: region = row["region"] total[region] = total.get(region, 0.0) + row["consumption_m3"] count[region] = count.get(region, 0) + 1 mean = {} for region in total: mean[region] = total[region] / count[region] ranked = sorted(mean.items(), key=lambda p: -p[1]) return ranked[:3] def whole_object(rows): region = column(rows, "region") consumption = column(rows, "consumption_m3") mean = group_mean(region, consumption) return top_n(descending(mean), 3) def declarative(rows): return solve({"source": rows, "group": "region", "measure": "consumption_m3", "top": 3}) # The measurement is read from the code's tree: statement count, assigned name count, loop nodes. LOOPS = (ast.For, ast.While, ast.comprehension) def tree(f): return ast.parse(inspect.getsource(f)).body[0] def statement_count(f): return len(tree(f).body) def name_count(f): g = tree(f) return len({n.id for n in ast.walk(g) if isinstance(n, ast.Name) and isinstance(n.ctx, ast.Store)}) def written_loops(f): return sum(1 for n in ast.walk(tree(f)) if isinstance(n, LOOPS)) # Executed loops: the loops of the helpers the expression calls are added to its own loops. def executed_loops(f, seen=None): seen = set() if seen is None else seen count = written_loops(f) for n in ast.walk(tree(f)): if isinstance(n, ast.Call) and isinstance(n.func, ast.Name): name = n.func.id if name in HELPERS and name not in seen: seen.add(name) count += executed_loops(HELPERS[name], seen) return count def align(w): return lambda h: "".join(str(v).ljust(w[0]) if j == 0 else str(v).rjust(w[j]) for j, v in enumerate(h)) EXPRESSIONS = [("procedural", procedural), ("whole-object", whole_object), ("declarative", declarative)] y = align([16, 11, 23, 16, 15]) print("\n" + y(["expression", "statement", "intermediate variable", "written loop", "executed loop"])) for name, f in EXPRESSIONS: print(y([name, statement_count(f), name_count(f), written_loops(f), executed_loops(f)])) results = [f(READINGS) for _, f in EXPRESSIONS] same = all(s == results[0] for s in results) print(f"\nall three give the same result: {'yes' if same else 'no'}") print(" " + ", ".join(f"{b} {v:.3f}" for b, v in results[0])) ``` ``` expression statement intermediate variable written loop executed loop procedural 6 6 2 2 whole-object 4 3 0 3 declarative 1 0 0 3 all three give the same result: yes east 24.674, central 22.396, north 20.210 ``` The last line is the anchor of the lesson: all three expressions give the same three regions and the same three numbers. The difference is not in the result but **in the writing**. The procedural expression holds 6 statements and binds 6 names; four of them are intermediate results and none of them enters the report. The whole-object expression comes down to 4 statements and 3 names, because rows are no longer walked: `region` and `consumption` are taken as two columns and the mean is applied to the whole column. The declarative expression is a single statement and binds no names; what is written is the recipe of the result. The real change in the second row happens not in the number but **in the unit of work**. In the procedural expression, what the code talks about is a single row named `row`, and the total and the count grow out of the fields of that row; the object of the analysis is a record. In the whole-object expression, what the code talks about is two columns named `region` and `consumption`; not even the name of a single row appears. The approach of statistical programming languages comes from here: data is handled in the unit of the **field**, not the record, and since the operation is applied to the whole field, the walking disappears entirely from the visible part of the code. #### The Loop Did Not Disappear, It Moved The last two columns of the table go in opposite directions. **Written loops** fall from 2 to 0, **executed loops** rise from 2 to 3. This is one of the most commonly misread places of abstraction: giving up writing the loop does not eliminate the loop, it only changes its place. The procedural expression writes two loops: one scans the dataset for the total and the count, the other walks the five grouped regions for the division. The whole-object expression writes no loop, but the helpers it calls run three loops — `column` scans the dataset once each time it is called and it is called twice, `group_mean` makes one scan and one merge. As the expression gets shorter, the number of passes does **not** decrease; in this example it increases. Assuming that there is a link between brevity and the number of passes is an unmeasured claim. What is gained in return is also countable: the four intermediate names in the procedural expression — `total`, `count`, `mean`, `ranked` — come down to three in the whole-object expression and to zero in the declarative expression. An intermediate name is something that has to be read and can be misused; lowering its count lowers the reading load of the expression. The price is the subject of the next section. What the measure does not count must also be written. The table counts only the statements **the person writing the analysis writes**; the bodies of the helpers do not enter the count. The four helpers and the solver together hold nine statements, and these are statements someone wrote somewhere. The gain of abstraction is not that the amount of work decreases but **that the same statements are written once and used many times**; the two columns are taken with the same `column` call in three different analyses as well. This is why the measure gives no ranking of superiority; it only shows where the load has moved. #### Five Decisions of a Single Statement The declarative expression writes four things: the source, the grouping field, the measure field and how many rows are requested. These are not all of the decisions that determine the result of the analysis. The solver below meets the same request with a complete set of defaults, and each default is changed one by one. ```python # The same declarative request, this time in a solver that carries all the defaults by name. DEFAULTS = {"missing": "count", "weight": "row", "summary": "mean", "period": "all", "direction": "descending"} def median(v): s = sorted(v) n = len(s) return s[n // 2] if n % 2 else (s[n // 2 - 1] + s[n // 2]) / 2 def solve_full(request): a = dict(DEFAULTS) a.update(request) rows = [row for row in request["source"] if (a["period"] == "all" or row["period"] == a["period"]) and (a["missing"] == "count" or row["consumption_m3"] > 0)] if a["weight"] == "subscriber": unit = {} for row in rows: unit.setdefault(row["subscriber_id"], (row[a["group"]], []))[1].append(row[a["measure"]]) rows = [{a["group"]: g, a["measure"]: sum(v) / len(v)} for g, v in unit.values()] bucket = {} for row in rows: bucket.setdefault(row[a["group"]], []).append(row[a["measure"]]) summary = {g: (sum(v) / len(v) if a["summary"] == "mean" else median(v)) for g, v in bucket.items()} ranked = sorted(summary.items(), key=lambda p: p[1], reverse=a["direction"] == "descending") return ranked[:a.get("top", len(ranked))] BASE = {"source": READINGS, "group": "region", "measure": "consumption_m3", "top": 3} CHANGES = [("unread rows", "missing", "count", "drop"), ("measure weight", "weight", "row", "subscriber"), ("summary function", "summary", "mean", "median"), ("period scope", "period", "all", "2026-01"), ("sort direction", "direction", "descending", "ascending")] baseline = dict(solve_full(BASE))["east"] y2 = align([20, 12, 12, 14, 12, 24]) print("\n" + y2(["hidden default", "default", "east m3", "changed", "east m3", "reported top three"])) values = [baseline] for name, key, before, after in CHANGES: request = {**BASE, key: after} top = solve_full(request) d = dict(solve_full({k: v for k, v in request.items() if k != "top"}))["east"] values.append(d) print(y2([name, before, f"{baseline:.3f}", after, f"{d:.3f}", " ".join(b for b, _ in top)])) print(f"\nsingle-statement expression writes {len(BASE)} decisions, leaves {len(DEFAULTS)} " f"decisions to defaults") print(f"if defaults are written the request grows {len(BASE)} -> {len(BASE) + len(DEFAULTS)} " f"keys and the expression is still 1 statement") print(f"east mean moves {max(values) - min(values):.3f} m3 between {min(values):.3f} and " f"{max(values):.3f}") ``` ``` hidden default default east m3 changed east m3 reported top three unread rows count 24.674 drop 25.593 east central north measure weight row 24.674 subscriber 24.814 east central north summary function mean 24.674 median 24.120 east central north period scope all 24.674 2026-01 25.540 east central north sort direction descending 24.674 ascending 24.674 west south north single-statement expression writes 4 decisions, leaves 5 decisions to defaults if defaults are written the request grows 4 -> 9 keys and the expression is still 1 statement east mean moves 1.473 m3 between 24.120 and 25.593 ``` A single statement writes four decisions and leaves five decisions to defaults. When the five defaults are changed one by one, the east mean becomes 25.593 — 24.814 — 24.120 — 25.540 m³ respectively instead of 24.674 m³; between the lowest and the highest there is a difference of **1.473 m³**, about six percent of the base value. The fifth does not move the number at all but changes the three reported regions completely: in descending order east, central and north are written, in ascending order west, south and north. The real observation here is this: the same five decisions **exist in the procedural expression as well**. There too the unread rows are counted, the weight is the row, and the summary is the arithmetic mean. The difference is **the number of places where the decisions can be written**. In the procedural expression the same five defaults are scattered across three statements — the loop, the division and the sorting lines — and each one sits in a place that can be changed by hand. In the single-statement expression there is no such line; a decision becomes visible only if it is added to the request as a key. When all the defaults are written explicitly, the request goes from four keys to nine keys and the expression is **still a single statement** — the cost of the record is not the statement but the key. The rule of the course finds its exact counterpart here. An abstraction shortens the work as much as it hides decisions; a function whose defaults you do not know is a function that decides on your behalf. A short expression is not bad for this reason — not knowing where a decision is written is bad. #### Summary - When the same analysis is written in three expression styles, it gives the same result: east 24.674 — central 22.396 — north 20.210 m³. - The measure of an expression is the number of statements, intermediate variables and loops: procedural 6/6/2, whole-object 4/3/0, declarative 1/0/0. - Written loops fall from 2 to 0 while executed loops rise from 2 to 3; abstraction does not remove the loop, it moves it, and in this example it adds one more. - The single-statement expression writes 4 decisions and leaves 5 decisions to defaults; when the five defaults are changed, the east mean moves within a range 1.473 m³ wide and the sort direction changes the three reported regions completely. - The same decisions exist in the procedural expression as well; what changes is the number of places where a decision can be written. #### Next Step The declarative expression came down to a single statement, but this descent has a price: a dictionary of four keys does not tell, in order, what an analysis does. The next lesson measures the form in between — the approach that writes the transformation as a chain made of individual **verbs**. The measure is from the same family: when the same seven-step transformation is written with intermediate variables, how many names are bound; when it is written as nested calls, in how many steps the reading order overlaps with the execution order; when it is written as a chain, which intermediate result's row count becomes invisible. ### Grammar of Dataframes Source: https://academia.sh/en/courses/data-analysis-with-python/grammar-of-dataframes The same seven-step transformation is expressed in three writings and all three give the same result: the writing with intermediate variables holds 7 statements and 6 intermediate variables, the nested call and the verb chain a single statement and 0 intermediate variables. The distinction shows up in reading order — reading order overlaps with execution order in 7/7 steps in the writing with intermediate variables, 7/7 in the verb chain and 1/7 in the nested call. The price of the chain is that the intermediate results stay unnamed: 0 of 6 intermediate results have a visible row count, and when a tracing verb is added the call count rises from 7 to 15 and the sequence 3,199 → 3,080 → 3,080 → 750 → 750 → 5 → 5 → 3 becomes readable. When the position of one of the same six verbs changes, the number of reported regions falls from 5 to 1 and the east mean from 7.301 m³ to 0.593 m³. The previous lesson brought the declarative expression down to a single statement and counted its price: a dictionary of four keys wrote four decisions and left five decisions to defaults. That dictionary had a second shortcoming. An analysis is not only a definition, it is **an order**: first these rows are discarded, then this column is derived, then it is grouped. A key-value dictionary never writes this order. This lesson measures the form in between. The **grammar of dataframes** writes the transformation as a chain made of individual **verbs**: filter, add, group, summarize, sort, first. The measure is two numbers: with how many statements and how many intermediate variables the same transformation is written, and which intermediate result becomes invisible when the chain gets shorter. - **AE5.** The dataset is inherited from Data Collection and Preparation and is **fiction**: 3,199 rows, 1,329 subscribers, 10/25/40 m³ tariff tiers. The seed is written in the code. - **AE6.** The verb chain is **modeled with the standard library**. `Frame` is a small model: every verb returns a new frame, and no verb changes its own input. No tool is imitated. - **AE7.** The measured transformation consists of seven verbs: filter the read rows, add the excess over 25 m³, filter the rows whose excess is positive, group by region, summarize the mean excess, sort in descending order, take the first three. - **AE8.** Two numbers stand side by side. The **gain** is the intermediate variable and statement count. The **hidden default** has two headings: in the chain the row counts of the intermediate results are invisible, and even though the verb order is written in the code, the reasoning of the decision that chose the order is not written anywhere. #### Seven Verbs, Three Writings ```python # grammar.py — MODEL: the 3,199-row fiction dataset of Data Collection and # Preparation is regenerated from the same seed, then the verb-based chain is # modeled with the standard library. import ast import inspect import math SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF PERIODS, TARIFF = ["2026-01", "2026-02", "2026-03"], [10, 25, 40] REGION = [("north", 0.28, 21), ("south", 0.22, 17), ("east", 0.18, 26), ("west", 0.14, 14), ("central", 0.18, 23)] def rng(t: int): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value() -> float: s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u: float, w: list[float]) -> int: t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 _subscribers, _dropped = [], [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] record = {"subscriber_id": 10001 + i, "region": b[0], "base": b[2], "household_size": pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]), "satisfaction": pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) + 1} (_dropped if r() < 0.046 else _subscribers).append(record) _valid = {k["subscriber_id"] for k in _subscribers} READINGS = [] for k in _subscribers + _dropped: r = rng(SEED + 7000 + k["subscriber_id"]) for d in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): t = 0.0 if r() < 0.038 else math.floor( k["base"] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - d * 0.05) * 100 + 0.5) / 100 if k["subscriber_id"] not in _valid: r() continue READINGS.append({"subscriber_id": k["subscriber_id"], "region": k["region"], "period": PERIODS[d], "consumption_m3": t, "satisfaction": k["satisfaction"], "tier": 1 if t < TARIFF[0] else 2 if t < TARIFF[1] else 3 if t < TARIFF[2] else 4}) r() class Frame: """MODEL of the dataframe grammar: every verb returns a new frame.""" def __init__(self, rows, key=None, trace=None): self.rows, self.key = rows, key self.trace = [] if trace is None else trace def filter(self, condition): return Frame([r for r in self.rows if condition(r)], self.key, self.trace) def add(self, name, f): return Frame([{**r, name: f(r)} for r in self.rows], self.key, self.trace) def group(self, name): return Frame(self.rows, name, self.trace) def summarize(self, name, f): bucket = {} for r in self.rows: bucket.setdefault(r[self.key], []).append(r) return Frame([{self.key: g, name: f(v)} for g, v in bucket.items()], None, self.trace) def sort(self, name, descending=True): return Frame(sorted(self.rows, key=lambda r: r[name], reverse=descending), self.key, self.trace) def first(self, n): return Frame(self.rows[:n], self.key, self.trace) def count(self, label): self.trace.append((label, len(self.rows))) return Frame(self.rows, self.key, self.trace) VERBS = ("filter", "add", "group", "summarize", "sort", "first") # The function form of the same verbs, for the nested writing. def filter(c, k): return c.filter(k) def add(c, a, f): return c.add(a, f) def group(c, a): return c.group(a) def summarize(c, a, f): return c.summarize(a, f) def sort(c, a): return c.sort(a) def first(c, n): return c.first(n) # The conditions and the measure of the transformation. def was_read(r): return r["consumption_m3"] > 0 def excess(r): return r["consumption_m3"] - 25 def positive(r): return r["excess_m3"] > 0 def mean_excess(v): return sum(r["excess_m3"] for r in v) / len(v) SOURCE = Frame(READINGS) def with_intermediates(c): read_rows = c.filter(was_read) with_excess = read_rows.add("excess_m3", excess) over = with_excess.filter(positive) grouped = over.group("region") summary = grouped.summarize("mean_excess", mean_excess) ordered = summary.sort("mean_excess") return ordered.first(3) def nested(c): return first(sort(summarize(group(filter(add( filter(c, was_read), "excess_m3", excess), positive), "region"), "mean_excess", mean_excess), "mean_excess"), 3) def chained(c): return (c .filter(was_read) .add("excess_m3", excess) .filter(positive) .group("region") .summarize("mean_excess", mean_excess) .sort("mean_excess") .first(3)) def tree(f): return ast.parse(inspect.getsource(f)).body[0] def verb_name(n): if not isinstance(n, ast.Call): return None name = n.func.attr if isinstance(n.func, ast.Attribute) else getattr(n.func, "id", None) return name if name in VERBS else None # For every verb call: in which statement, how many verb calls it contains, where in the code. def calls(f): g, out = tree(f), [] for i, statement in enumerate(g.body): for n in ast.walk(statement): if verb_name(n): inner = sum(1 for m in ast.walk(n) if m is not n and verb_name(m)) out.append({"statement": i, "inner": inner, "position": (n.func.end_lineno, n.func.end_col_offset)}) return out def name_count(f): return len({n.id for n in ast.walk(tree(f)) if isinstance(n, ast.Name) and isinstance(n.ctx, ast.Store)}) # Reading order is read from the position in the code, execution order from the nesting. def overlap(f): c = calls(f) reading = sorted(range(len(c)), key=lambda j: c[j]["position"]) execution = sorted(range(len(c)), key=lambda j: (c[j]["statement"], c[j]["inner"])) return sum(1 for a, b in zip(reading, execution) if a == b), len(c) def align(w): return lambda h: "".join(str(v).ljust(w[0]) if j == 0 else str(v).rjust(w[j]) for j, v in enumerate(h)) FORMS = [("with intermediates", with_intermediates), ("nested call", nested), ("verb chain", chained)] y = align([20, 11, 8, 23, 19, 27]) print(y(["form", "statement", "verb", "intermediate variable", "reading=execution", "named intermediate result"])) for name, f in FORMS: n, t = overlap(f) print(y([name, len(tree(f).body), t, name_count(f), f"{n}/{t}", f"{name_count(f)}/{t - 1}"])) results = [[(r["region"], round(r["mean_excess"], 3)) for r in f(SOURCE).rows] for _, f in FORMS] print(f"\nfictional dataset {len(READINGS)} rows; the three writings give the same result: " f"{'yes' if all(s == results[0] for s in results) else 'no'}") print(" " + ", ".join(f"{b} {v}" for b, v in results[0])) ``` ``` form statement verb intermediate variable reading=execution named intermediate result with intermediates 7 7 6 7/7 6/6 nested call 1 7 0 1/7 0/6 verb chain 1 7 0 7/7 0/6 fictional dataset 3199 rows; the three writings give the same result: yes east 7.301, central 6.532, north 4.986 ``` The three writings call the same seven verbs and return the same three rows. Where they part is visible in the first three columns. The writing with intermediate variables binds every step to a name: seven statements, six intermediate variables. The nested call and the verb chain are both a single statement and bind no names. That all three writings give the same result comes from a property of the model: no verb changes its own input, each returns a new frame. This is the precondition for the chain to be built at all. When a verb that changes its input in place is attached to a chain, a second expression looking at the same frame silently reads a different result; in the writing with intermediate variables this at least shows up as the overwriting of a name, in the chain it does not show up at all. The fourth column separates these two. **Reading order** is read from the position in the code — which verb name is written earlier on the page. **Execution order** comes from the nesting — the innermost call runs first. In the writing with intermediate variables the two overlap in all seven steps, because each statement takes the result of the one before. In the verb chain they also overlap in seven steps: since the chain walks from left to right and from top to bottom, the order read is the order run. In the nested call the overlap falls to **one in seven**. The `first` that is read first on the page runs last, the `filter` that is read last runs first; only the `group` in the middle is fourth in both orders. This is the gain of the chain: the same single-statement writing removes reading in reverse. #### The Chain That Leaves a Trace The last column gives the price. In the writing with intermediate variables, six of the six intermediate results are bound to a name; the row count of any of them can be read the moment it is wanted. In the chain this number is **zero**. The chain runs as a whole and nothing stops in between; whether the filter left 3,000 rows or 3 rows cannot be known without looking at the result. This is the decision the grammar hides: **how many rows were dropped in which step**. In the model, the `count` verb gives this back. ```python traced = (Frame(READINGS, None, []).count("input") .filter(was_read).count("filter read") .add("excess_m3", excess).count("add excess_m3") .filter(positive).count("filter excess>0") .group("region").count("group region") .summarize("mean_excess", mean_excess).count("summarize mean") .sort("mean_excess").count("sort descending") .first(3).count("first 3")) y2 = align([20, 10, 10]) print(y2(["stage", "rows", "change"])) previous = None for label, n in traced.trace: print(y2([label, n, "-" if previous is None else n - previous])) previous = n print(f"\ntrace taken with {len(traced.trace)} count calls: verb 7 -> calls {7 + len(traced.trace)}, " f"visible intermediate result 0/6 -> 6/6") ``` ``` stage rows change input 3199 - filter read 3080 -119 add excess_m3 3080 0 filter excess>0 750 -2330 group region 750 0 summarize mean 5 -745 sort descending 5 0 first 3 3 -2 trace taken with 8 count calls: verb 7 -> calls 15, visible intermediate result 0/6 -> 6/6 ``` The sequence tells what the chain does, row by row. The first filter drops 119 rows — the number of unread records, the same as in the previous lesson. `add` touches no row, it only adds a column. The second filter drops **2,330 rows**, and the real narrowing is here: of 3,080 readings only 750 are above 25 m³. Summarizing melts 745 rows down to five regions, and the last verb cuts two of them. The price is visible too: 7 verbs become 15 calls. The visibility of the intermediate result goes from 0/6 to 6/6, and its counterpart is eight extra calls. The readability of the chain and the visibility of the intermediate result are not obtained at the same time; one of the two is bought with eight calls. Said as a number: **the price of visibility per verb is slightly more than one call**, and in a chain that is not traced a drop of 2,330 rows passes without producing any warning. The trace has one more side gain: the zero changes at the `group` and `sort` stages show with numbers that those two verbs never touch the row count. Knowing which verbs of a chain produce or drop rows brings the places to look at when searching for an error from seven down to two. #### The Order of the Verbs Is a Decision The verbs of the chain are written in the code and their order is visible. What is not visible is **why** that order is that order. Below, the same six verbs are written in two different orders: once with the filter before summarizing, once with the filter after summarizing. ```python def filter_before(c): return (c.filter(was_read).add("excess_m3", excess).filter(positive) .group("region").summarize("mean_excess", mean_excess) .sort("mean_excess")) def filter_after(c): return (c.filter(was_read).add("excess_m3", excess) .group("region").summarize("mean_excess", mean_excess) .filter(lambda r: r["mean_excess"] > 0).sort("mean_excess")) ORDERS = [("filter before summarizing", filter_before), ("filter after summarizing", filter_after)] y3 = align([26, 8, 14, 12, 12]) print(y3(["verb order", "verb", "output rows", "east m3", "total m3"])) outcomes = [] for name, f in ORDERS: result_frame = f(Frame(READINGS, None, [])) d = {r["region"]: r["mean_excess"] for r in result_frame.rows} outcomes.append((len(result_frame.rows), d.get("east", 0.0))) print(y3([name, len(calls(f)), len(result_frame.rows), f"{d.get('east', 0):.3f}", f"{sum(d.values()):.3f}"])) print(f"\nthe same {len(calls(filter_before))} verbs, the position of a single verb changed: " f"reported region count {outcomes[0][0]} -> {outcomes[1][0]}, " f"east mean {outcomes[0][1]:.3f} -> {outcomes[1][1]:.3f} m3") ``` ``` verb order verb output rows east m3 total m3 filter before summarizing 6 5 7.301 24.539 filter after summarizing 6 1 0.593 0.593 the same 6 verbs, the position of a single verb changed: reported region count 5 -> 1, east mean 7.301 -> 0.593 m3 ``` The six verbs are the same, one changed place, and the report says something entirely different. In the first order the filter is applied to the rows: the region means of the 750 readings with an excess are computed, and 7.301 m³ comes out for east. In the second order the filter is applied to the **summary**: the region mean of all the readings is taken — negative excesses are counted too — and then the regions whose mean stays positive are kept. Of the five regions only east is left standing, and its number falls to 0.593 m³. Neither number is wrong; they are answers to two different questions. The first is the answer to "how large is the mean excess of the readings that exceed", the second to "which region's average subscriber exceeds the tier". What the grammar gives is the ability to write both with two lines of code; what it does not give is a record of which one was asked. The chain writes the order, **it does not write the reasoning of the order**; the rule of the course applies here as well — not a verb whose position is unwritten, but an order whose reasoning is unwritten counts as unmeasured. #### Summary - The same seven verbs give the same result in three writings: east 7.301 — central 6.532 — north 4.986 m³ mean excess. - The writing with intermediate variables holds 7 statements and 6 intermediate variables, the nested call and the verb chain 1 statement and 0 intermediate variables. - Reading order overlaps with execution order in 7/7 steps in the writing with intermediate variables, 7/7 in the chain and 1/7 in the nested call; the gain of the chain is that it removes reading in reverse. - In the chain the row counts of 0 of 6 intermediate results are visible; when the tracing verb is added the calls rise from 7 to 15 and the sequence 3,199 → 3,080 → 750 → 5 → 3 becomes readable. The second filter alone drops 2,330 rows. - When the position of one of the same six verbs changes, the number of reported regions falls from 5 to 1 and the east mean from 7.301 m³ to 0.593 m³. #### Next Step The transformation was built from verbs and each verb took the output of the one before. The same thought shows up in visualization as well: a chart is not a single call but is made of **layers** placed on top of one another — data, aesthetic, geometry, statistic, scale. The next lesson counts this: when the same chart is built with a convenience wrapper, how many independent arguments it takes; when it is built with layers, how many layers it holds; and when one more series is added on top, how many layers are reused. The hidden default stands beside it: the statistic a layer applies silently. ### Grammar of Graphics Source: https://academia.sh/en/courses/data-analysis-with-python/grammar-of-graphics The same bar chart is built in two ways: a convenience wrapper takes 9 arguments, a layered setup holds 5 layers, and both give the same values. When a reference line is added, the signature of the convenience wrapper rises from 9 to 11 and no item from the previous setup is reused; the layered setup rises from 5 to 6 layers and 5 layers stay as they are. The hidden default sits in the statistic layer: when four of the same five layers are held fixed and only the statistic is changed, the longest bar is north, east and north again — count 921, mean 24.674 m³ and total 18,613.81 m³ produce three different orderings. The position setting behaves the same way: the 6 layers stay unchanged while the number read for east/2 drops from 281 to 0.504. The previous lesson built the transformation from verbs and showed that each verb takes the output of the one before it. Visualization has a similar construction, but the parts do not follow one another — they **stack on top of one another**. The **grammar of graphics** treats a chart not as a single call but as the sum of mutually independent layers: data, mapping, statistic, geometry, scale, position. This lesson's number is the layer count: how many items are written when the same chart is built with a convenience wrapper, how many layers it holds when built with layers, and how many items are reused when the chart grows. Next to it stands the course's second number: which default steps in when a layer is not written, and what the number read from the chart becomes. - **AE9.** The dataset is inherited from Data Collection and Preparation and is **fiction**: 3,199 rows, 1,329 subscribers, 10/25/40 m³ tariff tiers. The seed is written in the code. - **AE10.** The layered chart is **modeled with the standard library**. `Layer` is a kind and a few settings, `Chart` is a list of layers; the drawing is done with text bars, because the measured thing is not the form but **the layer count**. No tool or library is imitated. - **AE11.** Whether the chart type fits the scale of the axis field is **not measured** in this lesson; that question was measured in the Basic Charts lesson of the Analysis with Spreadsheets course. Here the type is only a **geometry layer**, and what is counted is the number of layers and whether they are independent of one another. - **AE12.** Two numbers stand side by side. The **gain** is the number of items reused while the chart is extended. The **hidden default** is the unwritten layer: when the statistic and position layers are not written, the chart is still drawn, but the length of the bars shows something else. #### How Many Layers in a Chart ```python # chart.py — MODEL. First the 3,199-row fiction dataset of Data Collection and # Preparation is regenerated from the same seed, then layered visualization is # modeled with the standard library. import inspect import math SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF PERIODS, TARIFF = ["2026-01", "2026-02", "2026-03"], [10, 25, 40] REGION = [("north", 0.28, 21), ("south", 0.22, 17), ("east", 0.18, 26), ("west", 0.14, 14), ("central", 0.18, 23)] def rng(t: int): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value() -> float: s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u: float, w: list[float]) -> int: t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 _subscribers, _dropped = [], [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] record = {"subscriber_id": 10001 + i, "region": b[0], "base": b[2], "household_size": pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]), "satisfaction": pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) + 1} (_dropped if r() < 0.046 else _subscribers).append(record) _valid = {k["subscriber_id"] for k in _subscribers} READINGS = [] for k in _subscribers + _dropped: r = rng(SEED + 7000 + k["subscriber_id"]) for d in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): t = 0.0 if r() < 0.038 else math.floor( k["base"] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - d * 0.05) * 100 + 0.5) / 100 if k["subscriber_id"] not in _valid: r() continue READINGS.append({"subscriber_id": k["subscriber_id"], "region": k["region"], "period": PERIODS[d], "consumption_m3": t, "satisfaction": k["satisfaction"], "tier": 1 if t < TARIFF[0] else 2 if t < TARIFF[1] else 3 if t < TARIFF[2] else 4}) r() class Layer: """MODEL of layered visualization: a layer is a kind and a few settings.""" def __init__(self, kind, **setting): self.kind, self.setting = kind, setting def signature(self): return (self.kind, tuple(sorted( (k, f"{len(v)} rows" if isinstance(v, list) else str(v)) for k, v in self.setting.items()))) SUMMARY = {"count": lambda v, a: float(len(v)), "mean": lambda v, a: sum(s[a] for s in v) / len(v), "total": lambda v, a: sum(s[a] for s in v)} class Chart: def __init__(self, layers=()): self.layers = list(layers) def add(self, *k): return Chart(self.layers + list(k)) def replace(self, k): return Chart([k if e.kind == k.kind else e for e in self.layers]) def setting(self, kind, **missing): for k in self.layers: if k.kind == kind: return {**missing, **k.setting} return missing # Layers are processed in order: data is bucketed, the statistic is applied, position is adjusted. def value(self): data = self.setting("data", rows=[])["rows"] e = self.setting("mapping") summary = self.setting("statistic", summary="count")["summary"] bucket = {} for s in data: bucket.setdefault((s[e["x"]], s[e["color"]] if "color" in e else None), []).append(s) d = {a: SUMMARY[summary](v, e["y"]) for a, v in bucket.items()} if self.setting("position", mode="stacked")["mode"] == "proportional": whole = {} for (x, _), v in d.items(): whole[x] = whole.get(x, 0.0) + v d = {a: v / whole[a[0]] for a, v in d.items()} return d def draw(self, title): d = self.value() low = self.setting("scale", low=0.0)["low"] high = max(d.values()) print(f"{title} — {len(self.layers)} layers") for (x, r), v in sorted(d.items(), key=lambda p: -p[1]): n = round(24 * (v - low) / (high - low)) if high > low else 0 print(f" {x if r is None else f'{x}/{r}':<14}{v:9.3f} " + "#" * max(0, n)) for k in self.layers: if k.kind == "geometry" and k.setting.get("form") == "line": v = k.setting["value"] print(f" {k.setting['label']:<14}{v:9.3f} " + "-" * max(0, round(24 * (v - low) / (high - low)))) BASE = Chart().add( Layer("data", rows=READINGS), Layer("mapping", x="region", y="consumption_m3"), Layer("statistic", summary="mean"), Layer("geometry", form="bar"), Layer("scale", axis="y", low=0.0)) BASE.draw("mean consumption per region (m3)") ``` ``` mean consumption per region (m3) — 5 layers east 24.674 ######################## central 22.396 ###################### north 20.210 #################### south 16.602 ################ west 13.778 ############# ``` The chart was built from five layers, and each layer answers a single question. The **data** layer says which rows will be drawn. The **mapping** says which field goes to which axis. The **statistic** layer says how the number to be drawn is derived from the raw rows — here, the mean of each region. The **geometry** layer says in what form that number is drawn, and the **scale** layer says the axis. What matters about this separation is that the layers are **independent of one another**. Changing the geometry from bar to point does not touch the other four layers; changing the statistic from mean to count does not change the geometry. A chart is not a fixed list of options but a sentence built from combinable parts. Independence does not mean the layers are unordered. The `value` method does the work in a fixed order: first the data is separated into buckets according to the mapping, then the statistic is applied to each bucket, and the position setting is applied last. In whatever order the layers are written on the page, this order of processing does not change; the only thing that changes is which layers are in the list. In the verb chain the order was the writing itself; here the order is a rule of the grammar and is not read from the writing. #### Adding a Layer, Widening a Call The same chart can also be built with a convenience wrapper: all the decisions are gathered in a single signature, and the caller only fills in the arguments. The difference between the two shows up when the chart grows. ```python def ready_bar(rows, x, y, summary, color=None, position="stacked", low=0.0, high=None, title=""): """MODEL of the convenience wrapper: all decisions are gathered in a single signature.""" return Chart().add(Layer("data", rows=rows), Layer("mapping", x=x, y=y), Layer("statistic", summary=summary), Layer("geometry", form="bar"), Layer("scale", axis="y", low=low)) # To add a reference line, two more arguments have to enter the signature. def ready_bar_with_reference(rows, x, y, summary, color=None, position="stacked", low=0.0, high=None, title="", reference=None, reference_label=""): g = ready_bar(rows, x, y, summary, color, position, low, high, title) return g if reference is None else g.add( Layer("geometry", form="line", label=reference_label, value=reference)) OVERALL = sum(s["consumption_m3"] for s in READINGS) / len(READINGS) WITH_REFERENCE = BASE.add(Layer("geometry", form="line", label="overall mean", value=round(OVERALL, 3))) old = [k.signature() for k in BASE.layers] kept = sum(1 for k in WITH_REFERENCE.layers if k.signature() in old) def arg_count(f): return len(inspect.signature(f).parameters) def align(w): return lambda h: "".join(str(v).ljust(w[0]) if j == 0 else str(v).rjust(w[j]) for j, v in enumerate(h)) same = ready_bar(READINGS, "region", "consumption_m3", "mean").value() == BASE.value() print(f"convenience wrapper and layered setup give the same values: " f"{'yes' if same else 'no'}") y = align([28, 16, 22, 22]) print("\n" + y(["setup", "base chart", "reference added", "reused"])) print(y(["convenience wrapper (args)", arg_count(ready_bar), arg_count(ready_bar_with_reference), 0])) print(y(["layer list", len(BASE.layers), len(WITH_REFERENCE.layers), kept])) print() WITH_REFERENCE.draw("same chart, reference line added") ``` ``` convenience wrapper and layered setup give the same values: yes setup base chart reference added reused convenience wrapper (args) 9 11 0 layer list 5 6 5 same chart, reference line added — 6 layers east 24.674 ######################## central 22.396 ###################### north 20.210 #################### south 16.602 ################ west 13.778 ############# overall mean 19.713 ------------------- ``` The first line is the anchor: the two setups give the same values, so what is measured is not the result but the **structure**. The convenience wrapper builds the base chart with 9 arguments, the layer list with 5 layers. As a count, the convenience wrapper does not look like it holds fewer items, but the real distinction is in the second column. When a line showing the overall mean is added to the chart, the **signature** of the convenience wrapper rises from 9 to 11. This is a new signature seen by every place that uses that call, and the number of items reused from the previous setup is **zero**: the call is a whole, and a part of it cannot be carried over. In the layered setup the list rises from 5 to 6 and **5 layers stay as they are**; the only thing added is one geometry layer. The extension is written into the list here, into the signature there. There is also a reverse side to this, and it has to be written down. The convenience wrapper gives a frequently used chart in a single line and does not require stacking five layers by hand; in the layered setup, everyone who builds the same chart rewrites the same five layers. The measure is not a ranking of superiority: the convenience wrapper makes **repetition** cheaper, the layered setup makes **extension** cheaper. #### If the Statistic Layer Is Not Written The price of the independence of layers is that when a layer is not written, the chart is drawn without an error. The model puts a default in place of the unwritten layer, and the bars are still visible. ```python y2 = align([14, 12, 12, 12, 12, 12, 16]) REGIONS = ("north", "east", "central", "south", "west") print(y2(["statistic"] + list(REGIONS) + ["longest bar"])) longest = [] for summary in ("count", "mean", "total"): g = BASE.replace(Layer("statistic", summary=summary)) d = {a[0]: v for a, v in g.value().items()} longest.append(max(d.items(), key=lambda p: p[1])[0]) print(y2([summary] + [f"{d[b]:.3f}" for b in REGIONS] + [longest[-1]])) print(f"\nof the same {len(BASE.layers)} layers four are identical; only the statistic layer " f"changed and the longest bar became {' -> '.join(longest)}") COLORED = Chart([Layer("data", rows=READINGS), Layer("mapping", x="region", y="consumption_m3", color="tier"), Layer("statistic", summary="count"), Layer("geometry", form="bar"), Layer("scale", axis="y", low=0.0)]) y3 = align([18, 12, 12, 12, 12]) print("\n" + y3(["position setting", "east/1", "east/2", "east/3", "east/4"])) second = [] for mode in ("stacked", "proportional"): d = COLORED.add(Layer("position", mode=mode)).value() second.append(d[("east", 2)]) print(y3([mode] + [f"{d[('east', b)]:.3f}" for b in (1, 2, 3, 4)])) print(f"\nlayer count {len(COLORED.layers) + 1} is the same in both settings; a single setting " f"of a single layer changed and the number read for east/2 became {second[1]:.3f} " f"instead of {second[0]:.0f}") ``` ``` statistic north east central south west longest bar count 921.000 557.000 600.000 645.000 476.000 north mean 20.210 24.674 22.396 16.602 13.778 east total 18613.810 13743.340 13437.310 10708.020 6558.150 north of the same 5 layers four are identical; only the statistic layer changed and the longest bar became north -> east -> north position setting east/1 east/2 east/3 east/4 stacked 21.000 281.000 225.000 30.000 proportional 0.038 0.504 0.404 0.054 layer count 6 is the same in both settings; a single setting of a single layer changed and the number read for east/2 became 0.504 instead of 281 ``` The three rows use the same data, the same mapping, the same geometry and the same scale. The only thing that changes is the statistic layer, and what the chart says changes three times. When count is chosen, the longest bar is in the north, because the north has 921 readings. When mean is chosen, the longest bar moves to the east, because the east, with 557 readings, is the less crowded region but has the highest mean consumption. When total is chosen, the bar goes back to the north. All three are correct and **all three are the same bar chart**; the only thing that says which one is drawn is the statistic layer, written or unwritten. The position setting shows the same behavior once more. The six layers are exactly the same in both runs; the only difference is the setting of the position layer. In the stacked setting, the second tariff tier of the east region has **281 readings**, and the length of the bar is a number. In the proportional setting the same place becomes **0.504**, and the length of the bar is a share. A reader looks at the second chart and says "in the east the second tier is close to half" and is right; looks at the first and says "in the east the second tier has 281 readings" and is right too. How many readings there are cannot be read from the second chart, and what the share is cannot be read from the first. The course's rule applies to the chart as well: an abstraction shortens the work as much as it hides decisions. What the layered setup provides is that the decisions are **named** — the statistic is a layer, the position is a layer, and someone looking at the chart's list can see which one is written. An unwritten layer, on the other hand, falls to a default and the chart is still drawn; the person looking at the page is left with no record saying what the bar measures. #### Summary - The same bar chart is built with 9 arguments by the convenience wrapper and with 5 layers by the layered setup, and both give the same values. - When a reference line is added, the signature of the convenience wrapper rises from 9 to 11 and the number of reused items is 0; the layer list rises from 5 to 6 and 5 layers stay as they are. - The convenience wrapper makes repetition cheaper, the layered setup makes extension cheaper; the measure is not a ranking of superiority. - When the statistic layer changes, four layers of the chart stay fixed but the longest bar becomes north, east and north again: count 921, mean 24.674 m³, total 18,613.81 m³. - The position setting changes the number read without changing the layer count: the second tier of the east region reads 281 in the stacked setting and 0.504 in the proportional setting. #### Next Step Three lessons measured three approaches: in how many statements the same analysis is expressed, with how many verbs and how many intermediate variables the transformation is written, with how many layers the chart is built. In all of them the measure was the form of the expression, and in none of them was one environment said to be better than another. What remains is the choice itself: what does a team look at when it decides which environment to work in. The next lesson ties this to a criterion — the skill the team already has, the coverage of the ready-made parts around it and the portability of the work to production — and compares the three environments by number on three criteria, writing down in which criterion the choice changes. The course closes there and counts the gain and the hidden default of the twenty-two lessons in a single table. ### Environment Selection Source: https://academia.sh/en/courses/data-analysis-with-python/environment-selection Three environments are compared by number on three criteria, and each criterion puts a different environment in front: the number of people who can write in a team of nine comes out 7 / 2 / 4, the share of the nine steps covered by ready-made parts 7 / 9 / 5, the share of the nine production items covered by the environment 3 / 1 / 9. At equal weights the raw total is 17 / 12 / 18 and the choice is decided by a score. A weight sweep shows that the choice changes on all three of the three criteria: when the team weight is 2, the ecosystem weight is 2 and the production weight is 0, the choice moves to another environment; the third environment is chosen only when the ecosystem weight reaches 4, with 39 points. When the same nine numbers are normalized by stretching them over the range, the scores become 175 / 100 / 140 and the choice turns without a single raw number changing. Three lessons measured three approaches: in how many statements the same analysis is expressed, with how many verbs the transformation is written, with how many layers the chart is built. In all three, the measured thing was the form of the expression, and in none of them was one environment said to be better than another. What remains is the choice itself. This lesson's number is the **number per criterion**. Three environments are counted on three criteria, and what is searched for is the criterion on which the choice changes. Next to it stands the course's second number: the operation that reduces three criteria to a single score is also an abstraction, and it carries its own defaults. - **AE13.** The dataset is inherited from Data Collection and Preparation and is **fiction**: 3,199 rows, 1,329 subscribers, 10/25/40 m³ tariff tiers. The seed is written in the code. - **AE14.** The three environments are named by their **approaches** and are **modeled** in Python: the array and dataframe interface of a general-purpose language, the whole-object environment of an analysis-oriented language, a declarative query environment where the computation is done in the data store. No library or product name appears. - **AE15.** The criteria and their units are as follows. **Team:** the number of people in a team of nine who can write the analysis in that environment. **Ecosystem:** the number of the analysis's nine steps that are covered by ready-made parts. **Portability to production:** the number of nine production items that the environment covers. - **AE16.** The inventory is **fiction** and is for this team's own nine steps; it is not a general ranking of the environments. **No claim of comparative superiority is written.** #### Three Environments, Three Criteria The analysis asked is the same in all three environments and is the anchor of the lesson: the mean consumption per region, the first three regions in descending order. What changes is not the answer but the inventory of the road to the answer. ```python # environment_selection.py — MODEL. First the 3,199-row fiction dataset of Data # Collection and Preparation is regenerated from the same seed, then the inventory # of the three environments is modeled with the standard library. import math SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF REGION = [("north", 0.28, 21), ("south", 0.22, 17), ("east", 0.18, 26), ("west", 0.14, 14), ("central", 0.18, 23)] def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 # Household and satisfaction are drawn but not used in this lesson; the generator order is kept. _subscribers, _dropped = [], [] for i in range(RAW): r = rng(SEED + i) b = REGION[pick(r(), [x[1] for x in REGION])] pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]) pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) (_dropped if r() < 0.046 else _subscribers).append( {"subscriber_id": 10001 + i, "region": b[0], "base": b[2]}) _valid = {k["subscriber_id"] for k in _subscribers} READINGS = [] for k in _subscribers + _dropped: r = rng(SEED + 7000 + k["subscriber_id"]) for d in range(pick(r(), [0.05, 0.12, 0.21, 0.62])): t = 0.0 if r() < 0.038 else math.floor( k["base"] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - d * 0.05) * 100 + 0.5) / 100 if k["subscriber_id"] not in _valid: r() continue READINGS.append({"region": k["region"], "consumption_m3": t}) r() ENVIRONMENTS = ["general-purpose", "analysis-oriented", "declarative"] G, A, B = ENVIRONMENTS STEPS = ["read", "type", "filter", "join", "group", "sample", "text", "chart", "publish"] ITEMS = ["version", "lock", "scheduling", "monitoring", "error", "access", "log", "rollback", "scale"] # The inventory is FICTION and is for this team's own nine steps; it is not a general # ranking of the environments. The second column is the environments that can write, the third those that can read. TEAM = [("analyst-1", {G, A, B}, {G, A, B}), ("analyst-2", {G, A}, {G, A, B}), ("analyst-3", {G, B}, {G, A, B}), ("analyst-4", {G, B}, {G, B}), ("analyst-5", {G}, {G, B}), ("analyst-6", {G}, {G, B}), ("analyst-7", {G}, {G}), ("analyst-8", {B}, {G, B}), ("analyst-9", set(), {G})] READY = {G: set(STEPS) - {"chart", "publish"}, A: set(STEPS), B: set(STEPS[:5])} COVERED = {G: {"version", "lock", "log"}, A: {"version"}, B: set(ITEMS)} CRITERIA = {"team": {o: sum(1 for _, y, _ in TEAM if o in y) for o in ENVIRONMENTS}, "ecosystem": {o: len(READY[o]) for o in ENVIRONMENTS}, "production": {o: len(COVERED[o]) for o in ENVIRONMENTS}} def align(w): return lambda h: "".join(str(v).ljust(w[0]) if j == 0 else str(v).rjust(w[j]) for j, v in enumerate(h)) def top_three(rows): t, n = {}, {} for s in rows: t[s["region"]] = t.get(s["region"], 0.0) + s["consumption_m3"] n[s["region"]] = n.get(s["region"], 0) + 1 return sorted(((b, t[b] / n[b]) for b in t), key=lambda p: -p[1])[:3] print(f"fiction dataset (seed {SEED}): {len(READINGS)} rows, {len(_valid)} subscribers") print("the analysis asked is the same in all three environments: " + ", ".join(f"{b} {v:.3f}" for b, v in top_three(READINGS))) y = align([27, 18, 20, 14]) print("\n" + y(["criterion (out of 9)"] + ENVIRONMENTS)) for k, d in CRITERIA.items(): print(y([k] + [d[o] for o in ENVIRONMENTS])) print(y(["raw total (out of 27)"] + [sum(CRITERIA[k][o] for k in CRITERIA) for o in ENVIRONMENTS])) ``` ``` fiction dataset (seed 20260218): 3199 rows, 1329 subscribers the analysis asked is the same in all three environments: east 24.674, central 22.396, north 20.210 criterion (out of 9) general-purpose analysis-oriented declarative team 7 2 4 ecosystem 7 9 5 production 3 1 9 raw total (out of 27) 17 12 18 ``` The table starts with a single observation: **all three criteria put a different environment in front**. In the team, the general-purpose environment is ahead with 7 people; in the ecosystem, the analysis-oriented environment with 9 steps; in portability to production, the declarative environment with 9 items. In the declarative environment there are four steps to be built by hand: resampling, text parsing, layered charts and publishing. In such a table there is no answer to the question "which is better", because the criteria count different things. The only question that has an answer is this: how will these three numbers be combined. The raw total gives 17, 12 and 18, and the choice is decided by **a single score**. #### On Which Criterion the Choice Changes The raw total gives equal weight to the three criteria; this equality is not measured, it is assumed. Changing the weight criterion by criterion and looking at where the choice turns is the countable way of testing the assumption. ```python def score(weight): return {o: sum(weight[k] * CRITERIA[k][o] for k in CRITERIA) for o in ENVIRONMENTS} def choice(p): return max(ENVIRONMENTS, key=lambda o: p[o]) def fmt(p): return " / ".join(str(p[o]) for o in ENVIRONMENTS) ONE = {k: 1 for k in CRITERIA} BASE = score(ONE) y2 = align([18, 20, 20, 20, 20, 20]) print("\n" + y2(["criterion weight"] + list(range(5)))) for k in CRITERIA: print(y2([k] + [choice(score({**ONE, k: w})) for w in range(5)])) print(f"\nchoice at equal weights: {choice(BASE)} ({fmt(BASE)})") for k in CRITERIA: candidates = [(abs(w - 1), w) for w in range(13) if w != 1 and choice(score({**ONE, k: w})) != choice(BASE)] w = min(candidates)[1] p = score({**ONE, k: w}) print(f" {k} weight 1 -> {w}: choice {choice(p)} ({fmt(p)})") which = [w for w in range(13) if choice(score({**ONE, "ecosystem": w})) == A] p = score({**ONE, "ecosystem": which[0]}) print(f" the analysis-oriented environment is chosen only when the ecosystem weight is " f"{which[0]} ({fmt(p)})") ``` ``` criterion weight 0 1 2 3 4 team declarative declarative general-purpose general-purpose general-purpose ecosystem declarative declarative general-purpose general-purpose analysis-oriented production general-purpose declarative declarative declarative declarative choice at equal weights: declarative (17 / 12 / 18) team weight 1 -> 2: choice general-purpose (24 / 14 / 22) ecosystem weight 1 -> 2: choice general-purpose (24 / 21 / 23) production weight 1 -> 0: choice general-purpose (14 / 11 / 9) the analysis-oriented environment is chosen only when the ecosystem weight is 4 (38 / 39 / 33) ``` The answer is in the table: **the choice changes on all three of the three criteria**, but the thresholds are different. When the team weight rises from 1 to 2, the choice moves to the general-purpose environment and the scores become 24 and 22; when the ecosystem weight is 2, it moves to the same place; if portability to production is not counted at all, again to the same place. The reason all three can turn the choice is a starting gap of one point. The behavior of the third environment has to be written separately. The analysis-oriented environment gets the lowest total, 12 points, at equal weights; when the ecosystem weight is 4 it becomes the chosen environment with 39 points. No number has changed; the only thing that has changed is how many times the ready-made coverage of the nine steps is counted. A weight is not a measurement but a decision. #### The Criterion's Own Default Weight is not the only default. Before the numbers are added, they are made comparable, and that too is a rule. Three rules are put side by side: adding the raw numbers, counting the highest in each criterion as 100 and taking the others in proportion, counting the highest as 100 and the lowest as 0 and stretching what lies between. ```python def normalize(rule, criteria): total = {o: 0.0 for o in ENVIRONMENTS} for d in criteria.values(): v = [d[o] for o in ENVIRONMENTS] for o in ENVIRONMENTS: if rule == "raw count": total[o] += d[o] elif rule == "ratio to best": total[o] += 100.0 * d[o] / max(v) else: total[o] += 100.0 * (d[o] - min(v)) / (max(v) - min(v)) return total y3 = align([22, 18, 20, 14, 18]) print("\n" + y3(["normalization"] + ENVIRONMENTS + ["choice"])) for rule in ("raw count", "ratio to best", "range stretching"): t = normalize(rule, CRITERIA) print(y3([rule] + [f"{t[o]:.1f}" for o in ENVIRONMENTS] + [choice(t)])) READER = dict(CRITERIA, team={o: sum(1 for _, _, k in TEAM if o in k) for o in ENVIRONMENTS}) print("\n" + y3(["team criterion"] + ENVIRONMENTS + ["clear margin"])) for name, criteria in (("can write", CRITERIA), ("can read", READER)): t = normalize("raw count", criteria) s = sorted((t[o] for o in ENVIRONMENTS), reverse=True) print(y3([name] + [f"{t[o]:.0f}" for o in ENVIRONMENTS] + [f"{s[0] - s[1]:.0f}"])) ``` ``` normalization general-purpose analysis-oriented declarative choice raw count 17.0 12.0 18.0 declarative ratio to best 211.1 139.7 212.7 declarative range stretching 175.0 100.0 140.0 general-purpose team criterion general-purpose analysis-oriented declarative clear margin can write 17 12 18 1 can read 19 13 21 2 ``` The three rules produce three different scores from the same nine numbers, and the third one turns the choice. A one-point difference in the raw total corresponds to 1.6 points between 211.1 and 212.7 in the ratio rule, and the choice stays the same. The range stretching rule zeroes the lowest value in every criterion and compresses the middle value according to the width of the range: in the production criterion the range is 8 units, so the middle value of 3 points falls to 25 points and the totals become 175 and 140. The ranking has changed without a single raw number changing. The second table shows one level further in. The team criterion was defined as the number of people who can write; the number of people who can read is also a defensible definition. When the definition changes, the choice does not change in this set, but the totals become 19 / 13 / 21 and the clear margin rises from 1 point to 2 points; the same choice appears to have been given with twice the confidence. The function that reduces three criteria to a single score is an abstraction; the weight, the normalization rule and the criterion definition are its defaults. A score whose defaults are not written is not a measurement but a choice. #### Summary - When three environments are counted on three criteria, each criterion puts a different environment in front: team 7 / 2 / 4, ecosystem 7 / 9 / 5, portability to production 3 / 1 / 9. - At equal weights the raw total is 17 / 12 / 18 and the choice is decided by a one-point difference. - The choice changes on all three of the three criteria, and the thresholds are different: team 2, ecosystem 2, production 0. The third environment is chosen only when the ecosystem weight reaches 4, with 39 points. - The three normalization rules give 17 / 12 / 18, 211.1 / 139.7 / 212.7 and 175 / 100 / 140 points; the third turns the choice without a single raw number changing. - When the team criterion is defined as the number who can read, the choice stays the same and the clear margin rises from 1 point to 2 points. #### Course Wrap-Up The account of twenty-two lessons stands in a single table, and each number was read from its own lesson. | Lesson | By hand | With abstraction | Hidden default | |---|---|---|---| | Numerical 01 Working Environment | 2/17 steps identical | 11/17 with lock file | 5 decisions, 1,116 → 1,264 | | Numerical 02 Arrays | 22 lines, 188,864 bytes | 2 lines, 30,320 bytes | 2 decisions, 31.896 ↔ 26.839 | | Numerical 03 Vectorized Operations | 10 lines, 1 pass | 2 lines, 3 passes | 3 decisions, 102,034 → 1,240 | | Numerical 04 Broadcasting | 30,320 bytes extra | 0 bytes extra | 2 decisions, 1,260 → 1,587,600 | | Numerical 05 Basic Linear Algebra Operations | 823,059,744 operations | 704 operations | 2 decisions, 50.09% deviation | | Dataframes 01 Series and Dataframe | 781,710 comparisons | 2,361 passes | 2 decisions, 3,316.42 m³ difference | | Dataframes 02 Reading and Writing Data | 7 conversion rules | 1 read call | 3 decisions, 1,123 cells to missing | | Dataframes 03 Indexing and Slicing | 2,658 comparisons | 21 passes | 3 decisions, 95 / 94 / 617 rows | | Dataframes 04 Filtering and Querying | 5,034 passes, 16,184 bytes | 6,398 passes, 3,399 bytes | 2 decisions, 1,835 / 2,026 / 1,915 | | Dataframes 05 Grouping and Aggregation | 15,995 key lookups | 3,199 passes | 2 decisions, 160 rows drop | | Dataframes 06 Join Operations | 2,616,782 comparisons | 4,017 passes | 2 decisions, 12,250.65 m³ excess | | Dataframes 07 Time Series Operations | 41,587 passes | 3,212 passes | 3 decisions, 4,850.82 ↔ 7,882.58 | | Dataframes 08 Text Operations | 9 lines, 0 lines of output | 3 calls, 3,199 rows | 3 decisions, 256 rows silently | | Performance 01 Memory Usage | 173,837 bytes | 3,628 bytes, 47.9 times | 3 decisions, 108 rows change tier | | Performance 02 Chunked Processing | 1,152,229 bytes peak | 36,132 bytes, 32 times | 3 decisions, 1,035 → 0 subscribers | | Performance 03 Columnar Engines | 19,194 values | 2,835 values, 6.8 times | 3 decisions, 2,835 → 4,883 values | | Performance 04 Parallel and Distributed Dataframes | 3,199 units | 8,921 units, threshold 10,833 | 3 decisions, 10,833 → 20,119 threshold | | Performance 05 Taking the Query to the Data | 3,199 rows moved | 5 rows, 1,919 times | 2 decisions, 1,919 → 11 times | | Alternative 01 Statistical Programming Languages | 6 statements, 6 intermediate variables | 1 statement, 0 intermediate variables | 5 decisions, 1.473 m³ gap | | Alternative 02 Grammar of Dataframes | 7 statements, 6 intermediate variables | 1 statement, 0 intermediate variables | 2 decisions, 7.301 → 0.593 m³ | | Alternative 03 Grammar of Graphics | 9 arguments | 5 layers, 5 reused | 2 decisions, 921 ↔ 24.674 | | Alternative 04 Environment Selection | 3 criteria, 3 leading | 1 score, 1 choice | 3 decisions, 1-point difference | The direction of the second and third columns is not fixed: sometimes the abstraction shortens the work (2,616,782 comparisons fall to 4,017 passes), sometimes it does not (11,340 multiplications are the same in both writings), sometimes it raises the price (30,320 bytes rise to 90,960 bytes). The direction of the gain comes not from the abstraction but from the measured work. The fourth column is the rule of the course: **an abstraction shortens the work as much as it hides decisions; a function whose defaults you do not know is a function that decides on your behalf. A call whose defaults are not written down is counted as unmeasured.** The **hidden default count** written by the twenty-two lessons by name **is 60**. This number is not a defect but the output of the course: a default that is not counted looks like zero, and a default that looks like zero is applied silently at every call. In twenty-two lessons the toolset was built and every default of it was written down. But the tool still says only **what is**: the mean of the east region is 24.674 m³, that of the north 20.210 m³. Whether these numbers came from a sample or from the population was not asked; whether the 4.464 m³ difference between them is real or noise was not asked either. The next course, Statistical Analysis and Exploratory Data Analysis, takes up these two questions.