Skip to content
academia.sh
tr
Log in

Course Intermediate

Visualization, Reporting and Analysis at Scale

Published Updated

By the end of this course

Start course

01

Chart Design

Measuring how much of a number reaches the reader as it passes through an encoding channel: which channel the comparison, distribution, composition, and relationship questions require, the shape that a histogram bin and a box summary cannot give back from a distribution, the read-back error that stacked and grouped displays produce on slices with no baseline, what a scatter and a heatmap carry from a relationship, separating trend from seasonality on the time axis, the area channel magnifying the ratio by its square, distortions beyond axis truncation, and the number of series that remain indistinguishable when color stands alone.

  1. 01 Chart Type Selection The same five regional means are passed through five encoding channels and read back, and the mean read-back error comes out as follows: position 0.070 m³, length 0.183 m³, angle 0.543 m³, color step 0.931 m³, area 2.468 m³. The 4.463 m³ difference between east and north is read back as 2.500 m³ from the area channel and 5.426 m³ from the angle channel, and not a single reading changes. The color step separates four of the five regions, with east and central falling on the same step. Each of the four question types asks for something different to be read back, and the channel pairing follows from that demand.
  2. 02 Distribution Visualization The twin set that gives the same five numbers as the 921 readings of north is passed through three displays, and what is read back is not a number but a distributional feature: bin counts and the density display read back the two peaks of the twin, while the box summary carries no peak information at all. In the counts that pass through the length channel three bins drop to zero and 7 readings are erased from the display, and the 45.47 m³ extreme value of the twin disappears; the same extreme value is read back in north, because the tallest bar is 234, not 329. When the bin width is 12 m³ the number of peaks read back falls from two to one, and when the kernel width is 0.5 m³ four peaks that are not in north appear.
  3. 03 Categorical Visualization The crosstab of five regions and four tariff tiers is encoded with two displays, and the difference shows up at the baseline: in the stacked display the bottom slice is read from the position channel and its relative error is 2.7%, while the upper slices are read from the length channel and the error is 22.0% in the third slice and 47.2% in the fourth. North's fourth slice of 8 readings and west's third slice of 11 readings are read back as zero, that is, they leave the display entirely. Of the 39 region pairs that must be separated, the stacked display separates 35 and the grouped display 39; no pair is read reversed, so what is lost is not the order but the distinction. The same stacked display gives the total with a single reading: 921 observations come back as 920.
  4. 04 Relationship Visualization When 1,101 subscriber pairs are encoded as a scatter plot, the mark width leaves 1,600 distinguishable cells, and 792 of the points, that is 71.9%, overlap and can no longer be read back; the densest cell holds 16 points. When the same pairs are put on a heat map, not a single point is lost but the count is: the 28 distinct count values in the source fall to six of the seven color steps, 37 of the 51 filled cells land on a single step, and the mean relative read-back error comes out at 280.9%. When the steps are divided into an equal number of cells instead of equal intervals, the same error falls to 15.7%. A line that gives a single y for each x reduces the cloud's mean span of 44.1 steps to a single step.
  5. 05 Time Series Visualization The same 36-period series is encoded at the same height with two axis scales: on the axis that starts from zero the reading resolution is 2.273 m³ and the mean read-back error per point is 0.540 m³, on the truncated axis 0.909 m³ and 0.246 m³. At a precision of 0.05 m³/period the slope is read back only from a window of 24 periods on the axis that starts from zero and from a window of 16 periods on the truncated axis. The seasonal component has a slope of its own and changes sign with the start of the window: the same 24-period length gives -0.2239, -0.0487 and -0.2260 m³/period in three separate places, while the trend put into the fictional series is -0.140.
  6. 06 Geographic Visualization Five regions are encoded on a 12x12 coverage grid with five color steps, and what the reader reads back is not the step but the area the step covers: the correct mean weighted by the number of readings is 19.713 m³, while the read-back weighted by area gives 18.484 m³, a difference of -1.228 m³. West carries 14.9% of the readings and 25% of the area, so it gets a weight factor of 1.680. Projection stretching suppresses two regions of equal area by a factor of 1.237 relative to each other. When the area share is made equal to the read share, the error falls to 0.396 m³, at the price of 93 cells changing region and the pairs of neighboring regions dropping from 8 to 4.
  7. 07 Misleading Visuals The true ratio of 1.2208 between east and north gives a bar ratio of 3.0769 when the axis is started from 18 m³, and a reader who takes length as proportional to value reads north as 8.019 m³ instead of 20.210. When the same ratio is encoded into a square, both sides are multiplied, so the area ratio comes out as 1.4905 and east is read as 30.123 m³. For two series with different units, nine defensible axis pairs give seven separate crossing periods. When 12 irregularly spaced periods are laid out at equal intervals, a slope of -0.2079 m³/period is read as -0.6635. All four arise from a defensible default behavior.
  8. 08 Accessible Charts When eight series are encoded with a default palette made of two hue families and four lightness steps, all eight are distinguished by the full decoder, while a decoder that does not read the hue distinction is left with only four. When the shape channel is distributed in sequence, the number stays at four, because the shapes start over after four series and the color collision also occurs four series apart; when the shape is distributed by collision, it rises to eight. A legend box asks for 16 transitions, an ordered legend 6, direct labeling 0. Of nine lightness steps, five are distinguished at an interval of two, and with four shapes the ceiling of a single display comes out as 20 series.

02

Reporting

Counting how much of a finding survives when it is carried to the person who will decide: the number of metrics that can be read back from a dashboard and the number of decisions the layout supports, how many ways a metric can be gamed without moving what it measures, the number the refresh lag between the data model and the data source writes into the report, how many steps it takes to build the narrative from finding to recommendation and the gap a skipped step leaves in the reader, and the number of items that must be pinned to produce the same report a second time.

  1. 01 Dashboard Design The same twelve metrics are encoded in three layouts, and the number of metrics read back at a glance is counted for each: 1 in the layout arranged by insertion order, 2 when every metric is given its own display, and 9 in the layout split by decision. The number of supported decisions is 0, 0 and 3 respectively; when the glance budget is raised to twelve, the first two layouts have 2 and 6 metrics read back but still support no decision. In the length channel, the 1.861 difference between two ratios read back separately with errors of 1.50% and 3.08% is read as 2.500, an error of 34.3%.
  2. 02 Key Performance Indicators The gameability of a metric is counted: the reading coverage rate, which stands at 60.948%, is tried in six ways without a single reading changing in the measurement network, and five of them exceed the gaming threshold — denominator narrowing 64.286, counting zero as a reading 63.055, window widening 81.791, rounding 61.000, target normalization 93.766. A six-item redefinition lowers the number of ways exceeding the threshold from 5 to 0; a way that does not exceed the threshold is not counted as closed, because its movement of 0.037 points comes not from the definition but from the region sizes. When 106 readings are really lost in the measurement network, the tight definition drops from 60.948 to 53.273, while two of the six ways of the loose definition keep the number above its pre-drop value in the same drop.
  3. 03 Business Intelligence Tools The same metric gives a different value under three refresh schemes and the difference is counted: in a report produced twelve days after the close of period 3, the reading rate comes out 56.659 with a live query, 48.081 with a periodic extract and 36.042 with a pre-aggregated model; the final value at the source is 63.055. The same lag moves mean consumption by only 0.038 m³. The grain and the field set that the report data model fixes permanently close three of six questions: the model that stores only the mean gives a wrong answer to one question by 0.192 m³, while the model that stores the sum and the count answers three questions correctly. Three schemes and five report days produce fifteen reports, five distinct values and a spread of 27.013 points.
  4. 04 Building Narrative with Data When one step is removed from the five-step chain that runs from finding to recommendation, the premises the reader is forced to complete on their own are counted: removing the finding leaves 4 inferences open, removing the comparison 3, removing the candidate cause 2, removing the uncertainty 3, removing the recommendation 3. Of the eleven wrong defaults the chain carries, nine strengthen the recommendation and two weaken it — the gaps are not filled at random. The place of the margin of variation of the Statistical Analysis and Exploratory Data Analysis course is settled: if the margin is never written, 4 inferences and 3 wrong defaults remain; if it is written only at the end in a limitations paragraph, 1 inference and 1 wrong default remain; when it is written next to the finding and in the uncertainty step, both go to zero.
  5. 05 Report Reproducibility The items that must be fixed to produce the same report a second time are counted: the report's own 11 items are added on top of the 39 decisions in the preparation ledger of the Data Collection and Preparation course and the 9 environment items of the Data Analysis with Python course, for a total of 59. Of the eleven items, 10 move the report when released alone — the reading rate goes anywhere between 36.042 and 100.000; the one item that does not move it, released together with another item, carries 87.168 to 94.432. The data cut-off and the refresh scheme change all seven lines of the seven-line report, while the presentation items change only one; the eleven items move 26 lines in total.

03

Analysis at Scale

Counting which part of the same report can be divided once it is built on a cluster that does not fit on a single machine: reducing volume, velocity, and variety to measurable units, the number of passes and the records moved in the map–reduce model, the coordination overhead item that data frame processing on a cluster adds, the values a columnar file layout makes you read and the gain from skipping row groups, the ceiling the serial fraction puts on speedup, and the job size at which distribution is unnecessary.

  1. 01 Big Data Concept Reducing volume, velocity, and variety to countable units instead of qualitative claims: the 3,199 rows of the measurement network take 914,929 bytes in memory, 286.0 bytes per row, and the threshold at which the dataset no longer fits on a single machine ranges from 7,508,560 to 240,273,951 rows depending on the working space and intermediate share decisions. The same 1,260 meters produce 1,260 rows/period under periodic reading and 3,628,800 rows/period under fifteen-minute reading; the periods left to the threshold fall from 17,875 to 7, and from 168 to 6 in a network that grows 4% per period. Of the 18 fields in five source schemas, 9 cannot be joined and two mapping tables bring that number down to 6; key absence closes, unit and time format conflicts do not.
  2. 02 Map–Reduce Model The simplest abstraction of distributed computation is built and run with three functions, and the measure becomes the number of records moved in the shuffle: on the fictional dataset spread across eight source parts, a regional average moves 3,199 records in the order that shuffles first and then reduces and 40 records in the order that reduces locally first and then shuffles, a gain of 79.97 times. The gain depends on the number of keys and falls to 2.53 times on the subscriber key. The same question gives two values with two keys: the east average is 24.674 m³ weighted by rows and 24.814 m³ as the average of subscriber averages, and the records moved by the two passes rise from 40 to 1,305. If the reduce function is not associative the computation does not enter the model: the sum and the maximum do not change across seven different part counts, the median of part medians gives seven different values, and its spread is 0.2175 m³.
  3. 03 Distributed Analysis Engines Counting the coordination overhead items that data frame processing on a cluster adds per job: the three wide transformations of an eight-operation chain draw four stage boundaries and amount to 100 coordination overhead items over eight partitions; none of the items processes a row and their number is independent of job size. Retry is not the dominant item: at a 2% failure rate it raises the item count by 0-2%, at a 15% failure rate by 8-23%. The measure is the number of rows per coordination overhead item: 61.5 rows at 3,199 rows, 31,990 rows at 3,199,000 rows, and the ratio saturates at 32,878. A target of 10,000 rows per item is met at 520,000 rows in the four-stage chain and at 260,000 rows in the two-stage chain.
  4. 04 Columnar Storage Formats Measuring the number of values read by a file layout suited to analytical queries: the 3,199 rows and 7 columns of the fictional dataset amount to 22,393 values; the same query reads 22,393 values in the row-ordered flat layout, 6,398 in the column-bounded layout, and 2,069 in the row-grouped layout with headers, and five of the seven row groups are never opened. The row group size is a write decision and costs in both directions: in a 32-row group the headers make you read 300 values, in a single 3,199-row group nothing is skipped; the size that reads the fewest values is 64 rows and gives 19.07 times with 1,174 values. Which headers are read is the second decision and moves the best size from 64 to 128. The gain of the column boundary melts with the number of columns the query asks for: it falls to 1.00 times for a query asking for seven columns, and row group skipping gives 4.92 times.
  5. 05 Limits of Parallel Processing The cost of distribution and when it is unnecessary are counted through the serial fraction and the speedup ceiling: of the 6,753 touches of the fictional report chain 310 are indivisible, the serial fraction is 4.59% and the ceiling is 21.78 times. At a serial fraction of 2% the ceiling is 50, at 30% it is 3.33 times, and even 4,096 units cannot pass the ceiling. The gain per added unit falls from 0.91 to 0.00011; after 256 units each new unit gains less than two thousandths. How the ceiling behaves with job size is an assumption decision: if the indivisible items are held fixed the ceiling rises to 206,388 times, if the coordination overhead item and the row group header grow with job size it saturates at 43.64 times. The course wrap-up gathers the source and read-back numbers of the eighteen lessons, and the curriculum wrap-up the six courses and 90 lessons of the Data Analytics curriculum in a single table.

Final exam

Visualization, Reporting and Analysis at Scale — Final Exam

In preparation

This course's question bank is not complete yet. The exam opens here automatically once enough questions are added.

Start typing to search.

↑↓ ↵ Esc navigate · open · close