Unit 03 of 04
Scale and Performance
Counting what is left of an abstraction's convenience once the dataset no longer fits in memory: the bytes the same column holds in three types and the gain from the categorical dtype, how chunk size relates to the memory peak and the number of passes, and the operations that cannot be chunked, the number of values a row-based and a columnar read touch in the same query, the line item that partitioning and shuffle add and the job size at which moving to distribution starts to pay off, and the number of rows that cross the network once the filter and the aggregation are pushed to the source.
Published Updated
Lessons
- 01 Memory Usage In the 3,199-row dataset of the measurement network, the number of bytes a single column holds in memory depends on the type that column is stored in: the region column takes 173,837 bytes as an object list, 19,251 bytes as fixed-width text and 3,628 bytes as a categorical code, a gain of 47.9 times. Two defensible measurements of the same column differ by 6.6 times, because the measuring function counts the pointer and does not follow the object it points to. When the reading column is cut from an object list to 1 byte, memory shrinks 31.4 times but the total drifts by 6.9 m³ and 108 rows change tariff tier; the categorical dtype gains 5.3 times on a column with 5 distinct values and loses 4.1 times on a column with 1,329 distinct values. 3 min read
- 02 Chunked Processing When the 3,199-row file of the measurement network is read chunk by chunk, the memory peak drops with chunk size but the state the operation carries does not: at a chunk of 100 rows the peak is 36,132 bytes, 32 times smaller than holding the whole dataset, and the total comes out as 66,667.9 m³ at all four chunk sizes. The total, the count and the region breakdown finish inside the chunk with 3–5 items of state; the exact median runs on a 321-bucket histogram and the exact distinct subscriber count runs on 1,329 items of state, and those 87,166 bytes do not shrink as the chunk shrinks. Chunking silently makes an order assumption: the streaming computation returns a result for 1,035 subscribers on the subscriber-ordered file and, without any error, a result for 0 subscribers on the period-ordered file; when the chunk boundary moves to the region, the mean of the chunk means departs from the overall mean by 0.1039 m³. 14 min read
- 03 Columnar Engines In the 3,199-row, six-column dataset of the measurement network a single question is answered by four read paths, and all four give the same number: the row-based scan reads 19,194 values, columnar early materialization 9,597, late materialization with the filter moved forward 3 times fewer, 4,869, and the block-skipping path 2,835 values, and the gain reaches 6.8 times. Block skipping depends on the dataset being clustered on that column; when the same dataset is written in subscriber order two of the seven blocks are skipped instead of four, in shuffled order none is skipped, and the values read rise from 2,835 to 4,883. The gain of late materialization also depends on the selectivity of the filter: when the matching rows rise from 835 to 2,691 the gain falls from 1.97 times to 1.12 times and scattered access rises from 1,671 to 5,383. 13 min read
- 04 Parallel and Distributed Dataframes When the 3,199-row dataset of the measurement network is split into partitions, the cost of distribution itself appears and stays larger than the work of a single machine: a five-partition distribution by the region key costs 11,651 units and a four-partition distribution by the subscriber key costs 8,921 units, while a single machine spends 3,199 units. The cost has three parts — the work units of the largest partition, the rows shuffled and the coordination per partition — and the point at which it turns into gain moves with the partitioning key: 10,833 rows with a balanced key, 20,119 rows with a skewed key. A query that is not aligned never sees this threshold; in the period partitioning 66.7% of the rows do not stay in place and the shuffle item alone rises to 6.7 times the work of a single machine. 14 min read
- 05 Taking the Query to the Data The source of the measurement network holds 3,199 rows and a single question is answered with four strategies; all four give the same number, but the number of records transferred to the analysis environment falls from 3,199 to 5. When the filter is pushed to the source 835 rows are transferred, when column selection is pushed too 1,670 values, when aggregation is pushed as well 10 values, and the gain reaches 1,919 times. The price of the gain is visibility: the pushed total does not bring the 835 rows that passed the filter, the 3,199 rows in the source or the number of rows per group, and writing the decision next to the answer raises the values transferred from 10 to 27 and lowers the ratio to 710 times. The gain also depends on the number of groups: 3,199 times with 3 groups, 426 times with 15 groups, 11 times with 835 groups, and it is zeroed when the number of groups equals the number of filtered rows. 14 min read