Unit 01 of 02
Domain and Lifecycle
Building a fictional data pipeline with five stages and counting how much each stage carries: 2400 source rows passing through screening, deduplication, consumption, summary and serving stages down to 5 served values and how many rows stand behind each served value, the stage at which each of six corruption classes shows its effect and which of them never change the served number, the same corruption reflecting differently on the data engineer's measure and the model builder's, and a unit change being visible in neither, the corruption profile carried by the application database, log and external service source classes and the detection lag of each, a corruption showing up one period later in batch processing and in the same period in stream processing, along with the number of stages streaming enlarges and the state it carries.
Published Updated
Lessons
- 01 What Is Data Engineering Passing 2400 meter readings of a fictional measurement network through a five-stage pipeline and counting the rows each stage carries: screening 2400, deduplication 2400, consumption 2200, summary 55 and serving 5. The consumption stage drops zero rows, yet its output is two hundred rows smaller, because the difference for the first period does not exist by definition. Behind the five served values stand 2400 source rows, an average of 480 per value; the largest value rests on 588 source rows, the smallest on 372. Because the pipeline is deterministic, all of these numbers can be derived without a run. In the uninstrumented pipeline the corruption caught is zero and the detection lag is undefined, because there is nothing that measures. 14 min read
- 02 Lifecycle Passing six corruption classes through the same five-stage pipeline and counting each stage's own corruption exposure: screening sees only the field name change, deduplication sees only the double record, the consumption stage raises the cost of a missing row to exactly double (40 rows to 80, 46 rows to 92, 200 rows to 400), the summary cell count falls from 55 to 45 and 53, and the serving stage produces nothing other than five values in any of the six classes. Three classes leave the served number exactly the same: duplicated is swallowed in deduplication, type_drift is silently repaired by the pipeline's own type conversion, and unit_changed, although it destroys ten cells of the summary, does not change the served value because two periods cancel each other. late_arrival and unit_changed corrupt exactly the same number of summary cells, yet one changes the serving and the other does not. 4 min read
- 03 Data Engineering and Data Science Distinction Printing the same six corruption classes side by side on the measure of two roles: the data engineer sees the row trace of five stages, while the model builder sees the mean absolute deviation (baseline 10.832 m3) of a forecast rule that takes the per-region mean served by the pipeline as its feature. The two measures rank the same six classes differently: schema_changed is the noisiest class in the engineer's trace (2200 rows fall to 1800) yet moves the model's deviation by only 0.003; late_arrival removes only 40 rows from the source yet moves the deviation by 0.105, shifting it the most; missing_partition lowers the deviation by 0.003, that is, the corruption improves the measure. unit_changed and type_drift leave no trace in either. The invisibility of the unit change comes from a telescoping identity: the sum of the eleven cells of a region equals the difference of the end indexes, and the corruption does not touch the ends; when the same table is read on the period axis, the July mean comes out as 40592074.02 instead of 26.7. 4 min read
- 04 Source Systems Extracting the corruption profile of three source classes and counting after how many stages each corruption class is caught: with an intact delivery all three give 2400 rows and produce the same five served values, and no check fires. They part ways once corruption arrives: the application database carries three of the six classes with a total lag of 4 stages, the log carries five with a total lag of 8, and the external service carries all six with a total lag of again 8. Permissive parsing makes type drift invisible in both sources: 200 rows are corrupted, no check catches it, and the five served values stay exactly the same. The row count check at tolerance 0.05 catches none of the three classes because the largest row diff is 60, that is, 0.0250; lowering the tolerance to 0.0100 catches all three, but when its reference is the producer's own declaration, it catches none at any tolerance. 8 min read
- 05 Batch and Stream Processing Running the same pipeline in two modes and counting the trade-off between freshness and detection lag: in batch mode the corruption becomes visible one period later, in the 2026-08 run, in stream mode in the same period, in the 2026-07 window. In return, stream mode raises the number of stages from 5 to 8 and the carried state from 0 to 2600 entries; on the other hand, the rows read fall from 13200 to 2400. The period window fixes the flawed partition check: the missing partition that batch mode never catches is caught at stage 0 in stream mode. For the late-arriving row the trade-off reverses: batch mode takes the row back in the next run and leaves all 55 summary cells correct, while stream mode drops the 40 rows at the watermark and permanently dirties 10 cells, yet the five served values stay exactly the same in both modes. 4 min read