Skip to content
academia.sh
tr
Log in

Course Intermediate

Introduction to Data Engineering

Published Updated

By the end of this course

Start course

01

Domain and Lifecycle

Building a fictional data pipeline with five stages and counting how much each stage carries: 2400 source rows passing through screening, deduplication, consumption, summary and serving stages down to 5 served values and how many rows stand behind each served value, the stage at which each of six corruption classes shows its effect and which of them never change the served number, the same corruption reflecting differently on the data engineer's measure and the model builder's, and a unit change being visible in neither, the corruption profile carried by the application database, log and external service source classes and the detection lag of each, a corruption showing up one period later in batch processing and in the same period in stream processing, along with the number of stages streaming enlarges and the state it carries.

  1. 01 What Is Data Engineering Passing 2400 meter readings of a fictional measurement network through a five-stage pipeline and counting the rows each stage carries: screening 2400, deduplication 2400, consumption 2200, summary 55 and serving 5. The consumption stage drops zero rows, yet its output is two hundred rows smaller, because the difference for the first period does not exist by definition. Behind the five served values stand 2400 source rows, an average of 480 per value; the largest value rests on 588 source rows, the smallest on 372. Because the pipeline is deterministic, all of these numbers can be derived without a run. In the uninstrumented pipeline the corruption caught is zero and the detection lag is undefined, because there is nothing that measures.
  2. 02 Lifecycle Passing six corruption classes through the same five-stage pipeline and counting each stage's own corruption exposure: screening sees only the field name change, deduplication sees only the double record, the consumption stage raises the cost of a missing row to exactly double (40 rows to 80, 46 rows to 92, 200 rows to 400), the summary cell count falls from 55 to 45 and 53, and the serving stage produces nothing other than five values in any of the six classes. Three classes leave the served number exactly the same: duplicated is swallowed in deduplication, type_drift is silently repaired by the pipeline's own type conversion, and unit_changed, although it destroys ten cells of the summary, does not change the served value because two periods cancel each other. late_arrival and unit_changed corrupt exactly the same number of summary cells, yet one changes the serving and the other does not.
  3. 03 Data Engineering and Data Science Distinction Printing the same six corruption classes side by side on the measure of two roles: the data engineer sees the row trace of five stages, while the model builder sees the mean absolute deviation (baseline 10.832 m3) of a forecast rule that takes the per-region mean served by the pipeline as its feature. The two measures rank the same six classes differently: schema_changed is the noisiest class in the engineer's trace (2200 rows fall to 1800) yet moves the model's deviation by only 0.003; late_arrival removes only 40 rows from the source yet moves the deviation by 0.105, shifting it the most; missing_partition lowers the deviation by 0.003, that is, the corruption improves the measure. unit_changed and type_drift leave no trace in either. The invisibility of the unit change comes from a telescoping identity: the sum of the eleven cells of a region equals the difference of the end indexes, and the corruption does not touch the ends; when the same table is read on the period axis, the July mean comes out as 40592074.02 instead of 26.7.
  4. 04 Source Systems Extracting the corruption profile of three source classes and counting after how many stages each corruption class is caught: with an intact delivery all three give 2400 rows and produce the same five served values, and no check fires. They part ways once corruption arrives: the application database carries three of the six classes with a total lag of 4 stages, the log carries five with a total lag of 8, and the external service carries all six with a total lag of again 8. Permissive parsing makes type drift invisible in both sources: 200 rows are corrupted, no check catches it, and the five served values stay exactly the same. The row count check at tolerance 0.05 catches none of the three classes because the largest row diff is 60, that is, 0.0250; lowering the tolerance to 0.0100 catches all three, but when its reference is the producer's own declaration, it catches none at any tolerance.
  5. 05 Batch and Stream Processing Running the same pipeline in two modes and counting the trade-off between freshness and detection lag: in batch mode the corruption becomes visible one period later, in the 2026-08 run, in stream mode in the same period, in the 2026-07 window. In return, stream mode raises the number of stages from 5 to 8 and the carried state from 0 to 2600 entries; on the other hand, the rows read fall from 13200 to 2400. The period window fixes the flawed partition check: the missing partition that batch mode never catches is caught at stage 0 in stream mode. For the late-arriving row the trade-off reverses: batch mode takes the row back in the next run and leaves all 55 summary cells correct, while stream mode drops the 40 rows at the watermark and permanently dirties 10 cells, yet the five served values stay exactly the same in both modes.

02

Quality and Principles

Turning quality dimensions into running checks and counting both what each check catches and its own flaw: the detection lag of six corruption classes and the three classes a flawed check set never catches; a partition check that stays green even though one period of one region was never written, because it looks at the region set and not at the period; the rows dropped by a schema change, the stage at which a data contract catches it, and the cost of the contract itself; a pipeline that is not idempotent duplicating rows on reprocessing, and the served number staying unchanged because the deduplication stage absorbs the duplicates; the stage and row count one must visit to go from a corrupted served value back to the source, compared with and without a lineage record; how many of the five stages carry an owner, and who receives a corruption caught at an ownerless stage; and the number of periods a retention policy makes impossible to audit retroactively.

  1. 01 Data Quality Dimensions Turning quality dimensions into running checks and counting both what each check catches and its own flaw: the detection lag of the six corruption classes is 4, 0, 0, 4, 0 and 0 stages in order; with a flawed check set three classes are never caught; because the partition check looks at the region set, it stays green when a single period of one region was never written; a unit change moves ten cells of the summary and leaves the served values exactly the same.
  2. 02 Schema Contracts Counting whom the schema contract between producer and consumer breaks and what the contract itself costs: when a field is renamed, the producer side has 0 faulty rows while on the consumer side 200 rows are dropped at screening, 200 more at consumption, and the summary goes from 55 cells to 45; once the contract is written the same corruption is caught at stage 0 but the output is not repaired; a strict contract that pins the field set exactly blocks 1200 good rows with no corruption present and drops the summary to 25 cells; batch-level enforcement blocks 2200 good rows and yields no output at all, while an adapter stage returns the serving to the baseline and grows the pipeline to six stages.
  3. 03 Resilience to Repetition and Reprocessing Measuring that processing the same load a second time duplicates rows and that the served number does not show it: in an appending store, backfilling one period takes the raw rows from 2400 to 2600 and a full rerun takes them to 4800, reaching 7200 in three runs, yet the served value stays at 1212.55 in all three runs because the deduplication stage absorbs the surplus; when the resent load carries a correction, the same stage throws the correction away and leaves the value at 794.36; and when deduplication looks at the whole row instead of the key, the same reruns carry the value to 2425.09 and 3637.64.
  4. 04 Data Lineage Counting the rows one has to look at to go from a corrupted served value back to the source, with and without a lineage record: a five-stage pipeline carries 9,460 rows across six layers while the lineage record holds 11,455 links, because every consumption row has two ancestors; without the record, explaining a served value means reading 9,260 rows and that number does not depend on which value one starts from, while the same search with the record ends between 94 and 155 rows. Two runs give two different loss stages: for late-arriving records the loss is at the source, for a schema change it is at the screening stage. The size of the serving deviation does not give the size of the corruption: when five times as many rows are corrupted, the serving moves less and three values go up.
  5. 05 Data Governance Counting the ownership gap and the retention policy: in a pipeline with six positions counting the source, three of the five stages carry an owner, yet all six corruptions are caught at an ownerless position, because the checks run only at the source and at the summary layer. A finding walks downstream until it reaches an owner, and three of the six findings go to an owner who can repair them; in the other three the recipient is one to five stages away from the repair point and has no access to the evidence. The three classes whose repair point is ownerless are exactly the three classes the flawed check set never catches. A six-period retention window deletes six periods but raises the periods that can no longer be audited retroactively to seven, the summary falls from 55 cells to 25, 2200 of the lineage's 4400 consumption links point to a deleted row, and the correct partition check raises a false alarm on intact data. The lesson is the course wrap-up.

Final exam

Introduction to Data Engineering — Final Exam

In preparation

This course's question bank is not complete yet. The exam opens here automatically once enough questions are added.

Start typing to search.

↑↓ ↵ Esc navigate · open · close