Unit 03 of 03
Analysis at Scale
Counting which part of the same report can be divided once it is built on a cluster that does not fit on a single machine: reducing volume, velocity, and variety to measurable units, the number of passes and the records moved in the map–reduce model, the coordination overhead item that data frame processing on a cluster adds, the values a columnar file layout makes you read and the gain from skipping row groups, the ceiling the serial fraction puts on speedup, and the job size at which distribution is unnecessary.
Published Updated
Lessons
- 01 Big Data Concept Reducing volume, velocity, and variety to countable units instead of qualitative claims: the 3,199 rows of the measurement network take 914,929 bytes in memory, 286.0 bytes per row, and the threshold at which the dataset no longer fits on a single machine ranges from 7,508,560 to 240,273,951 rows depending on the working space and intermediate share decisions. The same 1,260 meters produce 1,260 rows/period under periodic reading and 3,628,800 rows/period under fifteen-minute reading; the periods left to the threshold fall from 17,875 to 7, and from 168 to 6 in a network that grows 4% per period. Of the 18 fields in five source schemas, 9 cannot be joined and two mapping tables bring that number down to 6; key absence closes, unit and time format conflicts do not. 13 min read
- 02 Map–Reduce Model The simplest abstraction of distributed computation is built and run with three functions, and the measure becomes the number of records moved in the shuffle: on the fictional dataset spread across eight source parts, a regional average moves 3,199 records in the order that shuffles first and then reduces and 40 records in the order that reduces locally first and then shuffles, a gain of 79.97 times. The gain depends on the number of keys and falls to 2.53 times on the subscriber key. The same question gives two values with two keys: the east average is 24.674 m³ weighted by rows and 24.814 m³ as the average of subscriber averages, and the records moved by the two passes rise from 40 to 1,305. If the reduce function is not associative the computation does not enter the model: the sum and the maximum do not change across seven different part counts, the median of part medians gives seven different values, and its spread is 0.2175 m³. 3 min read
- 03 Distributed Analysis Engines Counting the coordination overhead items that data frame processing on a cluster adds per job: the three wide transformations of an eight-operation chain draw four stage boundaries and amount to 100 coordination overhead items over eight partitions; none of the items processes a row and their number is independent of job size. Retry is not the dominant item: at a 2% failure rate it raises the item count by 0-2%, at a 15% failure rate by 8-23%. The measure is the number of rows per coordination overhead item: 61.5 rows at 3,199 rows, 31,990 rows at 3,199,000 rows, and the ratio saturates at 32,878. A target of 10,000 rows per item is met at 520,000 rows in the four-stage chain and at 260,000 rows in the two-stage chain. 14 min read
- 04 Columnar Storage Formats Measuring the number of values read by a file layout suited to analytical queries: the 3,199 rows and 7 columns of the fictional dataset amount to 22,393 values; the same query reads 22,393 values in the row-ordered flat layout, 6,398 in the column-bounded layout, and 2,069 in the row-grouped layout with headers, and five of the seven row groups are never opened. The row group size is a write decision and costs in both directions: in a 32-row group the headers make you read 300 values, in a single 3,199-row group nothing is skipped; the size that reads the fewest values is 64 rows and gives 19.07 times with 1,174 values. Which headers are read is the second decision and moves the best size from 64 to 128. The gain of the column boundary melts with the number of columns the query asks for: it falls to 1.00 times for a query asking for seven columns, and row group skipping gives 4.92 times. 3 min read
- 05 Limits of Parallel Processing The cost of distribution and when it is unnecessary are counted through the serial fraction and the speedup ceiling: of the 6,753 touches of the fictional report chain 310 are indivisible, the serial fraction is 4.59% and the ceiling is 21.78 times. At a serial fraction of 2% the ceiling is 50, at 30% it is 3.33 times, and even 4,096 units cannot pass the ceiling. The gain per added unit falls from 0.91 to 0.00011; after 256 units each new unit gains less than two thousandths. How the ceiling behaves with job size is an assumption decision: if the indivisible items are held fixed the ceiling rises to 206,388 times, if the coordination overhead item and the row group header grow with job size it saturates at 43.64 times. The course wrap-up gathers the source and read-back numbers of the eighteen lessons, and the curriculum wrap-up the six courses and 90 lessons of the Data Analytics curriculum in a single table. 13 min read