Skip to content
academia.sh
tr
Log in

Course Intermediate

Data Analysis with Python

Published Updated

By the end of this course

Start course

01

Numerical Computing

Separating, by number, what the array abstraction gains from what it hides: the items that must be pinned in an isolated environment for the same notebook to give the same number on another machine, the lines, passes and bytes a list and an array interface take for the same summation job, the type promotion and overflow that vectorization hides, broadcasting on incompatible shapes returning an unexpected array instead of an error, and the difference in pass count between hand-written loops and the abstraction in matrix multiplication and solving.

  1. 01 Working Environment Counts the items that must be pinned for a notebook to give the same number on another machine: when the nine environment items and the seventeen-step preparation pipeline of the Data Collection and Preparation course are written side by side, nothing pinned reproduces 2 of 17 steps, the dependency list 5, the lock file 11; the remaining 6 steps depend on five items the lock file never writes. Running the same notebook on two machines, the locale changes all 1,260 meter indexes and moves the total from 1,236,664.22 to 41,433,984.33, the chunk count moves the mean by 0.006053 m³ (12 times the threshold), and the notebook cell order raises the count of cells above 40 m³ from 1,116 to 1,264.
  2. 02 Arrays The unit of measure of the course is set up: the same job (total per subscriber, overall mean, count of cells above 40 m³) is done with nested lists and with an array interface, and the line count falls from 22 to 2 and the allocated bytes from 188,864 to 30,320 (from 49.96 to 8.02 bytes per element), while the pass count falls from 3 to 3, that is, does not fall at all. The hidden default is that an array has no empty value: when the 581 empty cells are filled with zero the mean comes out 26.993180, with a sentinel value 26.839476, with filled cells only 31.895661 m³; the count above 40 m³ is 1,116 under all three decisions. The dtype choice keeps 3,780 elements between 7,560 and 30,240 bytes.
  3. 03 Vectorized Operations The same derivation job is done with a single loop and with a chain of vectorized calls: the line count falls from 10 to 2 while the pass count rises from 1 to 3 and the allocated bytes from 30,320 to 90,960; every link in the chain adds one pass and one intermediate array. Three decisions the call makes silently are counted: a single float constant carries the chain to 64 bits and raises the allocated bytes from 30,400 to 60,720, a 16-bit dtype shifts 1,538 of the 3,780 elements so that the total of 102,034.22 m³ shows as 1,239.85 m³, and of the 581 empty cells marked with a sentinel value 0 stay distinguishable after the first operation, while the count of cells whose excess looks zero rises to 1,783.
  4. 04 Broadcasting The broadcasting rule is modeled with the standard library and run on eight shape pairs: 4 pairs give the correct result, 2 pairs give an error, 2 pairs silently produce an unexpected array. The cost of the silent case is counted: multiplying (1260, 1) by (1, 1260) gives 1,587,600 elements instead of 1,260, the allocated bytes rise from 10,160 to 12,700,880, only 1,260 of the elements are correct, and the reported per-person mean becomes 36.497564 instead of 36.938180 m³. In the second silent case the period base produces 9 values instead of 3, and the adjusted base total comes out 251.846368 m³ instead of 83.901904 m³.
  5. 05 Basic Linear Algebra Operations In matrix multiplication, abstraction is measured not to change the operation count: 11,340 multiplications stay the same in both ways, the lines at the call site fall from 13 to 2, and 26,460 hand-written index computations disappear. In solving, the gain is a choice of algorithm: the cofactor expansion written from the definition asks for 823,059,744 operations at n = 12, elimination finishes in 704 operations, a ratio of 1,169,119 times. The hidden default is pivoting — while the operation count stays fixed at 14, elimination without pivoting carries the residual from 8.538e-11 to 7.237e+01 as the first coefficient shrinks, and when the coefficient is zeroed it produces no solution at all. In an ill-conditioned tariff system, a single 0.01 m³ change in the input moves the unit price of the first tier by 50.09%.

02

Dataframes

Making the decisions that labeled data structures take on visible one by one: the number of rows an index alignment produces in an addition, the column that type inference silently misreads, label-based and position-based slicing returning different rows, a missing value giving three different filter results in three comparisons, a missing value in a group key silently dropping, the number of rows and the fan-out produced by inner, outer and cross joins, whether the empty period is counted as zero or skipped in a resample, the behavior of vectorized string operations on a missing value.

  1. 01 Series and Dataframe January and February readings are added per subscriber, and the same two series give three different results under three join modes: positional addition produces 1,101 rows and only 4 of them point to the same subscriber under both labels, yet its total of 44,271.89 m³ is within 119.13 m³ of the correct answer. Inner alignment by label also produces 1,101 rows and all of them are correct; outer alignment by label produces 1,260 rows and 159 of them are missing. Manual alignment without a label index makes 781,710 label comparisons, while adding the series reaches the same result in 2,361 passes. Counting the missing side as zero moves the total from 44,391.02 m³ to 47,707.44 m³.
  2. 02 Reading and Writing Data A file of 3,199 rows and 7 columns, 139,238 bytes in size, is read with a single call, and type inference returns 3 of the 7 columns with the wrong type: the subscriber number is an integer, so its average can be computed as 10,690.5; the average of the tariff tier comes out as 27.4 m³; the 3,008 numeric values in the meter reading column turn into text because of 80 'n/a' entries, and the largest value of the column becomes 'n/a'. When inference is made from the first 100 rows instead of all rows, the pass count drops from 22,393 to 700, but the meter number slides to integer: 1,123 cells of the lettered codes that start at row 2,077 become missing without producing any warning, and 3,199 cells change in the round trip. The format loss is 422 cells under both inferences.
  3. 03 Indexing and Slicing The same two ends, 10,100 and 10,200, return four different row counts in four separate spellings: sliced by label it is 95 rows because a label slice includes the end, sliced by the same two positions it is 94 rows because a position slice leaves the end out, written with the ends mistaken for positions it is 100 rows, and in a long table with a non-unique index it is 230 rows. When the frame is sorted by total consumption the label index stops being sorted and the same slice returns 617 rows; an end that is not in the index gives a slice of 22 rows in a sorted index and raises an error in an unsorted one. A linear scan without an index takes 2,658 passes, while binary search in the label index reaches the same 95 rows in 21 passes.
  4. 04 Filtering and Querying Readings with a meter reading above 120,000 are filtered with three separate spellings and three different row counts come out: the numeric comparison gives 1,835 rows and 36,435.83 m³, the negation of the same condition gives 2,026 rows and 40,356.71 m³, and the same comparison with the column read as text gives 1,915 rows and 38,035.93 m³. The difference is in the missing values: the negation takes in all 191 missing rows, the text comparison takes in the 80 'n/a' cells because a letter counts as greater than a digit. When two conditions are combined the difference grows, and two expressions written as equivalent in logic return 446 and 502 rows. A manual filter chain builds an intermediate table of 16,184 bytes in 5,034 passes, while the mask makes 6,398 passes and holds 3,399 bytes.
  5. 05 Grouping and Aggregation The split, apply and combine steps fit into a single call, and that call carries two decisions at once: the 160 rows whose group key is empty appear in no group under the default grouping, and the total of the five-row summary comes out as 60,025.56 m³ instead of 63,060.63 m³, that is, 4.81% of the dataset drops out without producing any warning. When the missing key is kept, the summary has 6 rows and the dropped part becomes visible as 3,035.07 m³. The second decision is the denominator of the aggregation: the north group holds 878 rows but the meter reading column has 831 filled values, and the average comes out as 126,247.9 or 119,489.8 depending on the denominator. A separate loop per measure costs 15,995 key lookups, while a single split feeds five measures with 3,199 lookups.
  6. 06 Join Operations The reading table of 3,199 rows and the visit table of 818 rows are joined on the same key with four modes and four different row counts come out: inner join 1,965, left outer 3,820, full outer 3,856, cross join 2,616,782 rows. The inner join drops 1,855 readings and opens the remaining 1,344 rows into 1,965 rows; this fan-out of 621 rows moves the total from 26,450.81 m³ to 38,701.46 m³, that is, it produces 12,250.65 m³ of excess counting. Although the relationship of the key is many-to-many, it is verified nowhere: the key of 468 rows on the left side corresponds to more than one row on the right, and these 468 rows silently open into 1,089 rows. A hand-written nested loop makes 2,616,782 comparisons, while an indexed join finishes in 4,017 passes.
  7. 07 Time Series Operations 3,199 readings are reduced to seven-day periods and two answers come out for the 5 periods that carry no record: if they are counted as zero the series has 13 rows, if they are skipped it has 8 rows. The total is 63,060.63 m³ in both and does not change, but every calculation after the total does. The weekly average moves between 4,850.82 m³ and 7,882.58 m³; of the 12 period-to-period percent change ratios, 5 are left undefined because they fall on a zero denominator, while none of the 7 ratios of the skipped series is undefined; the three-period moving average splits between 708.03 m³ and 9,387.54 m³ in the period of 2026-02-12. The same dataset gives 3 periods with a thirty-day step and none of them is empty: what creates the empty period is not the data but the step decision.
  8. 08 Text Operations Of the 3,199 rows of the address code column, 2,943 are complete, 160 are empty and 96 do not carry the third part. The hand-written loop fails at row 20 and stops, producing 0 rows; the vectorized chain that does the same job produces all 3,199 rows, raises no error at all and silently turns 256 rows into missing. The filter that comes after the chain leaves these 256 rows outside on equality and takes them in on inequality: 1,004 and 2,195 rows, whose sum is exactly the dataset. Because the first step of the chain converts the code to lowercase, the uppercase filter returns 0 rows and that does not raise a warning either. A text column with three intermediate series holds 32.6 bytes of pointers per row, and the same structure reaches 0.61 gibibytes at 20 million rows.

03

Scale and Performance

Counting what is left of an abstraction's convenience once the dataset no longer fits in memory: the bytes the same column holds in three types and the gain from the categorical dtype, how chunk size relates to the memory peak and the number of passes, and the operations that cannot be chunked, the number of values a row-based and a columnar read touch in the same query, the line item that partitioning and shuffle add and the job size at which moving to distribution starts to pay off, and the number of rows that cross the network once the filter and the aggregation are pushed to the source.

  1. 01 Memory Usage In the 3,199-row dataset of the measurement network, the number of bytes a single column holds in memory depends on the type that column is stored in: the region column takes 173,837 bytes as an object list, 19,251 bytes as fixed-width text and 3,628 bytes as a categorical code, a gain of 47.9 times. Two defensible measurements of the same column differ by 6.6 times, because the measuring function counts the pointer and does not follow the object it points to. When the reading column is cut from an object list to 1 byte, memory shrinks 31.4 times but the total drifts by 6.9 m³ and 108 rows change tariff tier; the categorical dtype gains 5.3 times on a column with 5 distinct values and loses 4.1 times on a column with 1,329 distinct values.
  2. 02 Chunked Processing When the 3,199-row file of the measurement network is read chunk by chunk, the memory peak drops with chunk size but the state the operation carries does not: at a chunk of 100 rows the peak is 36,132 bytes, 32 times smaller than holding the whole dataset, and the total comes out as 66,667.9 m³ at all four chunk sizes. The total, the count and the region breakdown finish inside the chunk with 3–5 items of state; the exact median runs on a 321-bucket histogram and the exact distinct subscriber count runs on 1,329 items of state, and those 87,166 bytes do not shrink as the chunk shrinks. Chunking silently makes an order assumption: the streaming computation returns a result for 1,035 subscribers on the subscriber-ordered file and, without any error, a result for 0 subscribers on the period-ordered file; when the chunk boundary moves to the region, the mean of the chunk means departs from the overall mean by 0.1039 m³.
  3. 03 Columnar Engines In the 3,199-row, six-column dataset of the measurement network a single question is answered by four read paths, and all four give the same number: the row-based scan reads 19,194 values, columnar early materialization 9,597, late materialization with the filter moved forward 3 times fewer, 4,869, and the block-skipping path 2,835 values, and the gain reaches 6.8 times. Block skipping depends on the dataset being clustered on that column; when the same dataset is written in subscriber order two of the seven blocks are skipped instead of four, in shuffled order none is skipped, and the values read rise from 2,835 to 4,883. The gain of late materialization also depends on the selectivity of the filter: when the matching rows rise from 835 to 2,691 the gain falls from 1.97 times to 1.12 times and scattered access rises from 1,671 to 5,383.
  4. 04 Parallel and Distributed Dataframes When the 3,199-row dataset of the measurement network is split into partitions, the cost of distribution itself appears and stays larger than the work of a single machine: a five-partition distribution by the region key costs 11,651 units and a four-partition distribution by the subscriber key costs 8,921 units, while a single machine spends 3,199 units. The cost has three parts — the work units of the largest partition, the rows shuffled and the coordination per partition — and the point at which it turns into gain moves with the partitioning key: 10,833 rows with a balanced key, 20,119 rows with a skewed key. A query that is not aligned never sees this threshold; in the period partitioning 66.7% of the rows do not stay in place and the shuffle item alone rises to 6.7 times the work of a single machine.
  5. 05 Taking the Query to the Data The source of the measurement network holds 3,199 rows and a single question is answered with four strategies; all four give the same number, but the number of records transferred to the analysis environment falls from 3,199 to 5. When the filter is pushed to the source 835 rows are transferred, when column selection is pushed too 1,670 values, when aggregation is pushed as well 10 values, and the gain reaches 1,919 times. The price of the gain is visibility: the pushed total does not bring the 835 rows that passed the filter, the 3,199 rows in the source or the number of rows per group, and writing the decision next to the answer raises the values transferred from 10 to 27 and lowers the ratio to 710 times. The gain also depends on the number of groups: 3,199 times with 3 groups, 426 times with 15 groups, 11 times with 835 groups, and it is zeroed when the number of groups equals the number of filtered rows.

04

Alternative Analysis Environments

Counting how the same analysis is expressed in other environments and in which criterion the choice changes: how many steps analysis-oriented languages need for the same job, the intermediate variable and step counts of verb-based transformation chains, how many layers the same chart takes in a layered visualization approach, and a numeric comparison of three environments on team, ecosystem and production-readiness criteria.

  1. 01 Statistical Programming Languages The same analysis — mean consumption per region and the top three regions in descending order, on the 3,199-row fiction dataset — is written in three expression styles and all three give the same number: the procedural expression holds 6 statements, 6 intermediate variables and 2 loops; the whole-object expression holds 4 statements, 3 intermediate variables and 0 loops; the declarative expression holds 1 statement, 0 intermediate variables and 0 loops. Written loops drop from 2 to 0 while executed loops rise from 2 to 3: the loop does not disappear, it moves into a helper, and one more is added. The single-statement expression writes 4 decisions and leaves 5 decisions to defaults; when the five defaults are changed one by one, the east mean becomes 25.593, 24.814, 24.120 and 25.540 m³ instead of 24.674 m³, and when the sort direction changes, the reported three regions change completely.
  2. 02 Grammar of Dataframes The same seven-step transformation is expressed in three writings and all three give the same result: the writing with intermediate variables holds 7 statements and 6 intermediate variables, the nested call and the verb chain a single statement and 0 intermediate variables. The distinction shows up in reading order — reading order overlaps with execution order in 7/7 steps in the writing with intermediate variables, 7/7 in the verb chain and 1/7 in the nested call. The price of the chain is that the intermediate results stay unnamed: 0 of 6 intermediate results have a visible row count, and when a tracing verb is added the call count rises from 7 to 15 and the sequence 3,199 → 3,080 → 3,080 → 750 → 750 → 5 → 5 → 3 becomes readable. When the position of one of the same six verbs changes, the number of reported regions falls from 5 to 1 and the east mean from 7.301 m³ to 0.593 m³.
  3. 03 Grammar of Graphics The same bar chart is built in two ways: a convenience wrapper takes 9 arguments, a layered setup holds 5 layers, and both give the same values. When a reference line is added, the signature of the convenience wrapper rises from 9 to 11 and no item from the previous setup is reused; the layered setup rises from 5 to 6 layers and 5 layers stay as they are. The hidden default sits in the statistic layer: when four of the same five layers are held fixed and only the statistic is changed, the longest bar is north, east and north again — count 921, mean 24.674 m³ and total 18,613.81 m³ produce three different orderings. The position setting behaves the same way: the 6 layers stay unchanged while the number read for east/2 drops from 281 to 0.504.
  4. 04 Environment Selection Three environments are compared by number on three criteria, and each criterion puts a different environment in front: the number of people who can write in a team of nine comes out 7 / 2 / 4, the share of the nine steps covered by ready-made parts 7 / 9 / 5, the share of the nine production items covered by the environment 3 / 1 / 9. At equal weights the raw total is 17 / 12 / 18 and the choice is decided by a score. A weight sweep shows that the choice changes on all three of the three criteria: when the team weight is 2, the ecosystem weight is 2 and the production weight is 0, the choice moves to another environment; the third environment is chosen only when the ecosystem weight reaches 4, with 39 points. When the same nine numbers are normalized by stretching them over the range, the scores become 175 / 100 / 140 and the choice turns without a single raw number changing.

Final exam

Data Analysis with Python — Final Exam

In preparation

This course's question bank is not complete yet. The exam opens here automatically once enough questions are added.

Start typing to search.

↑↓ ↵ Esc navigate · open · close