--- title: 'What Is Data Engineering' source: 'https://academia.sh/en/courses/introduction-to-data-engineering/what-is-data-engineering' course: 'Introduction to Data Engineering' language: en updated: '2026-10-21T05:03:01+00:00' license: 'CC BY-SA 4.0' objectives: - 'Defines the responsibility of data engineering through the stages of a pipeline and separates a pipeline that crashes from a pipeline that carries wrong data.' - 'Runs a fictional source through a five-stage pipeline and counts the remaining and the dropped rows of each stage separately.' - 'Computes how many source rows stand behind each served value and shows how many rows the serving layer reduces to a single number.' - 'Builds the uninstrumented pipeline as the baseline and explains why the corruption caught and the detection lag are undefined on this baseline.' --- # What Is Data Engineering Passing 2400 meter readings of a fictional measurement network through a five-stage pipeline and counting the rows each stage carries: screening 2400, deduplication 2400, consumption 2200, summary 55 and serving 5. The consumption stage drops zero rows, yet its output is two hundred rows smaller, because the difference for the first period does not exist by definition. Behind the five served values stand 2400 source rows, an average of 480 per value; the largest value rests on 588 source rows, the smallest on 372. Because the pipeline is deterministic, all of these numbers can be derived without a run. In the uninstrumented pipeline the corruption caught is zero and the detection lag is undefined, because there is nothing that measures. The AI Engineering curriculum, opened by the Introduction to AI Engineering course, built the machinery around the model in one hundred thirty-two lessons and wrote down the cost of every part. Throughout those lessons the data entering the model was assumed to be **ready**: there was a table, it was read; there was a feature, it was computed. How that table got filled, what it takes to keep it running in production, and **who will know when it breaks** were never asked. The subject of this curriculum is that pipeline, and one peculiar flaw of the pipeline defines the whole measure: **a pipeline gives notice when it crashes, and says nothing when it carries wrong data.** A crashing pipeline leaves an error record, triggers an alert, piles up a queue; someone looks. A pipeline that carries wrong rows runs, finishes, writes the table, and the dashboard stays green. The job of this lesson is to build that pipeline and to **count**: how many rows go in, how many each stage carries, how many reach the served number. - **DL1.** The code is a **fiction**. No real data store, file format, orchestrator or runtime is called; the pipeline is written with the plain standard library. The source is the **fictional** meter readings of a measurement network, and the seed is **20260218**. - **DL2.** The source is **200 subscribers x 12 periods = 2400 readings**. Each reading carries an index value, the index only increases, and the unit is `m3`. Consumption is computed from the index difference. - **DL3.** The pipeline has **five stages**: `screen`, `dedupe`, `consumption`, `summary`, `serve`. Each stage **counts** the rows it drops; there is no uncounted drop. - **DL4. The procedure for collection, cleaning, missing values and outliers is not repeated.** It was measured in the Data Collection and Preparation course. The only thing added here is the **stage counter**: at which step and why a row dropped is written down. - **DL5. The procedure for schema, constraints and normalization is not repeated.** It was measured in the Data Modeling and Relational Theory course. In this lesson the schema is reduced to the single question "is the required field there". - **DL6.** The pipeline is **deterministic**. `random` is not used, the generator is written inside the lesson, and the same seed gives the same 2400 rows. For this reason the measures are independent of the run, and all of the numbers below can be derived without a run. - **DL7.** No product, vendor or format name is written; stores and processors are named by their **kind**. Time is not written as raw duration; lag is written as **stages** or **periods**. - **DL8.** The resolution of the set is the lower bound of the measurement band: over 2400 rows the smallest step of a row ratio is **0.0004**, over 55 summary cells **0.0182**, over 5 served values **0.2000**. **No ratio is printed on the serving layer**; the value itself is written. ## The Five Stages of the Pipeline The responsibility of data engineering fits in one sentence: to make data produced in one place usable in another. The word "usable" in that sentence hides a series of steps, and unless the steps are counted the responsibility cannot be measured. The pipeline below splits those steps into five parts: a row without a required field is **screened out**, the same subscriber-period pair is **deduplicated**, **consumption** is computed from the index difference, a **summary** is produced per region and period, and the period mean per region is **served**. The order of these five is not arbitrary, and the code itself enforces two of the ties. The `dedupe` stage reads the `subscriber` and `period` fields of the row directly; if one of these fields is missing, the stage stops with an error. So **screening has to come before deduplication**, or the pipeline crashes. The second tie is quieter: the `consumption` stage produces one output row for every input row, and the `summary` stage adds these rows up. If a double record entered consumption without being deduplicated, the same subscriber-period difference would produce two output rows and would be added to the summary cell **twice**. When the first tie is violated, the pipeline stops and gives notice; when the second tie is violated, the pipeline runs, finishes and **writes a wrong number**. The stage order of a pipeline is the first decision that determines which errors will be loud and which will be silent. ```python # FICTIONAL pipeline -- no real store, format or orchestrator is called. # The source is FICTIONAL meter readings of a measurement network, seed 20260218. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value REGION = ["east", "west", "north", "south", "central"] TYPE = ["residential", "commercial"] PERIODS = [f"2026-{a:02d}" for a in range(1, 13)] SUBSCRIBERS = {} for i in range(200): r = rng(SEED + 37 * i) SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)], "start": 10000 + int(r() * 60000)} # One reading per subscriber per period. Unit m3 , the index only increases. def generate_source(): records = [] for i, (sub, a) in enumerate(SUBSCRIBERS.items()): e = a["start"] for p in PERIODS: r = rng(SEED + 101 * i + 7 * PERIODS.index(p)) e += 4 + int(r() * 46) records.append({"subscriber": sub, "period": p, "region": a["region"], "type": a["type"], "index": e, "unit": "m3"}) return records def s1_screen(records): """A row that lacks a required field is dropped.""" REQUIRED = ("subscriber", "period", "region", "index", "unit") keep = [k for k in records if all(a in k for a in REQUIRED)] return keep, len(records) - len(keep) def s2_dedupe(records): seen, keep = set(), [] for k in records: key = (k["subscriber"], k["period"]) if key in seen: continue seen.add(key) keep.append(k) return keep, len(records) - len(keep) def s3_consumption(records): """Consumption from the period difference. Without a previous period the row cannot be computed.""" table = {(k["subscriber"], k["period"]): k for k in records} out, dropped = [], 0 for k in records: i = PERIODS.index(k["period"]) if i == 0: continue prev = table.get((k["subscriber"], PERIODS[i - 1])) if prev is None: dropped += 1 continue try: f = int(k["index"]) - int(prev["index"]) except (TypeError, ValueError): dropped += 1 continue out.append(dict(k, m3=f)) return out, dropped def s4_summary(records): totals = {} for k in records: key = (k["region"], k["period"]) totals[key] = totals.get(key, 0) + k["m3"] return totals, 0 def s5_serve(summary): """Serving: per-region mean over the periods.""" region = {} for (b, d), v in summary.items(): region.setdefault(b, []).append(v) return {b: round(sum(v) / len(v), 2) for b, v in region.items()}, 0 STAGES = [("screen", s1_screen), ("dedupe", s2_dedupe), ("consumption", s3_consumption), ("summary", s4_summary), ("serve", s5_serve)] def pipeline(records): trace, data = [], records for name, f in STAGES: data, dropped = f(data) n = len(data) if hasattr(data, "__len__") else 0 trace.append({"stage": name, "remaining": n, "dropped": dropped}) return data, trace source = generate_source() served, trace = pipeline(source) print("source rows", len(source), " subscribers", len(SUBSCRIBERS), " periods", len(PERIODS)) print("stage remaining dropped") for a in trace: print(f"{a['stage']:<13}{a['remaining']:>10}{a['dropped']:>9}") print("served") for b, v in sorted(served.items(), key=lambda x: -x[1]): print(f" {b:<8}{v:>9.2f}") ``` ``` source rows 2400 subscribers 200 periods 12 stage remaining dropped screen 2400 0 dedupe 2400 0 consumption 2200 0 summary 55 0 serve 5 0 served central 1321.45 north 1212.55 south 1065.55 east 881.36 west 833.09 ``` The first two rows of the trace table pass with zero drops: because the source is intact, there is no row with a missing field to screen out and no double record to deduplicate. The third row is the key point of the reading. The `consumption` stage **drops zero rows**, yet its output is **two hundred rows smaller**. This difference is not a loss; it is the definition itself: consumption is the index difference of two periods, and the **difference for the first period of each subscriber does not exist**. The first period of two hundred subscribers cannot be computed, but these are not faulty rows. That is why the "remaining" and "dropped" columns of a pipeline trace are kept separate: **a row that does not come out by definition and a row removed because of an error cannot be written in the same column.** None of these numbers depends on the run. Because the pipeline is deterministic, the table can be derived without a run: 200 subscribers x 12 periods = 2400; each subscriber loses one period, so 200 x 11 = 2200; 5 regions x 11 periods = 55 cells; 5 regions = 5 values. A second run with the same seed gives the same table, so every difference seen in the comparisons below comes from the corruption itself, not from variability. ## The Rows Behind a Served Value The most looked-at place of a pipeline is its last stage, because the report is read from there. The last stage of this pipeline produces **five numbers**. How many rows stand behind those five numbers says how much of the pipeline they hide. ```python # How many rows stand behind each served value v1, _ = s1_screen(source) v2, _ = s2_dedupe(v1) v3, _ = s3_consumption(v2) summary, _ = s4_summary(v3) subs, rows, cells = {}, {}, {} for a in SUBSCRIBERS.values(): subs[a["region"]] = subs.get(a["region"], 0) + 1 for k in v3: rows[k["region"]] = rows.get(k["region"], 0) + 1 for b, d in summary: cells[b] = cells.get(b, 0) + 1 print("region served summary consumption source subscribers") for b, v in sorted(served.items(), key=lambda x: -x[1]): print(f"{b:<9}{v:>9.2f}{cells[b]:>9}{rows[b]:>13}{subs[b] * 12:>8}{subs[b]:>13}") print(f"{'total':<9}{len(served):>9}{len(summary):>9}{len(v3):>13}{len(source):>8}{len(SUBSCRIBERS):>13}") ``` ``` region served summary consumption source subscribers central 1321.45 11 539 588 49 north 1212.55 11 506 552 46 south 1065.55 11 440 480 40 east 881.36 11 374 408 34 west 833.09 11 341 372 31 total 5 55 2200 2400 200 ``` The table plainly writes the pipeline's **compression ratio**. The only number visible in the report for `central`, **1321.45**, stands on the **588 source rows** of forty-nine subscribers over twelve periods, on the **539 consumption rows** computed from them, and on eleven **summary cells**. The smallest value, `west`, stands on **372 source rows**. Together the five values represent **2400 rows**, that is, an average of **480 source rows** per value. This compression has two faces. The useful face is clear: nobody can decide by looking at two thousand four hundred rows, and that is exactly the job of a summary. Its cost is the subject of the measure. When a source row is corrupted, that corruption enters the served number **with a weight of one in 480**; for `central`, one in 588. The ratio of the consumption layer to the source is **0.9167**, and since the smallest step of this ratio is 0.0004, it can be read to three digits; but writing a ratio on the serving layer is meaningless, because only five values exist there and the proportional weight of one value is 0.2000. What is to be written on the serving layer is **the value itself**. From here comes the first structural result of the pipeline: **the corruption of a single row dilutes so far that it is not visible in the served number, but the corruption of a hundred rows in the same direction does not dilute that far.** At which threshold which corruption becomes visible is the real quantity this pipeline is to measure, and this lesson does not measure it yet; it only lays the ground. Compression has one more side that does not show in the row count: the pipeline drops not only rows but also a **distinguishing dimension**. The output of the first three stages is keyed by the `subscriber` and `period` pair, and two hundred separate subscribers stand in the table. The output of the `summary` stage, however, is keyed by the `region` and `period` pair; the subscriber column does **not remain** there. From none of the fifty-five cells can it be read which subscribers were added up. The `serve` stage does the same thing to the period axis: from none of the five values can it be read how much each period added. This is not a loss but the definition of a summary, yet its consequence in production is heavy. When a served number looks suspicious, walking backward cannot be done by **reading** the intermediate outputs; the first three stages of the pipeline have to be **rerun**. Writing at which stage the pipeline leaves which dimension is the same thing as writing in advance the cost of investigating a corruption. ## The Count of the Uninstrumented Pipeline The pipeline above runs, finishes and writes five numbers. The only thing known about this pipeline is **how many rows it carries**. Nothing is known about whether the rows it carries are **correct**, because the pipeline has no step that asks this. From here on, in every lesson of the curriculum, **three numbers will stand side by side**: 1. **Uninstrumented pipeline.** No check is enabled. Corruption caught **0**, detection lag **undefined**. The pipeline in this lesson is exactly this. 2. **Instrumented pipeline.** The same workload, with checks added. The corruption caught and the **detection lag** are written as a stage number. 3. **Cost.** The detection lag, the good rows wrongly blocked, the rows reprocessed and the stages added. The items in the third point are the most often skipped side of this curriculum. Adding a check is not a free improvement: a check can also block a good row, can stop the pipeline needlessly, can duplicate rows during reprocessing and **can itself be flawed**. That is why, when the instrumented pipeline is written, not only the corruption caught but also the **cost of the check itself** goes into the same table. When the number of corruptions caught is written alone, the improvement is overstated. The word "undefined" in the first point should be read carefully. The detection lag of the uninstrumented pipeline is not **large**, it **does not exist**; since nothing measures, a quantity called lag does not arise. This is the difference between zero and undefined, and it is often confused in reporting: the sentence "no corruption was caught" is, when there is no check, not an announcement of success but an announcement **that no measurement was made**. From here comes the rule of the curriculum, and it holds through six courses: **the number of a data pipeline is not the rows it carries, but after how many steps a wrong row is caught. A pipeline whose check that catches the corruption is not written counts as unmeasured.** The trace of the pipeline above looks flawless, because the only thing the trace shows is how many rows the stages hand over to one another. About the content of the rows handed over, the trace is silent, and the silence is written in green. ## Summary - The responsibility of data engineering is to make data produced in one place usable in another; this responsibility becomes measurable only when the stages of the pipeline and the rows each stage carries are counted. - A pipeline gives notice when it crashes and says nothing when it carries wrong data; the measure of this curriculum is the second case. - The five-stage pipeline carries 2400 source rows as 2400, 2400, 2200, 55 and 5; the `consumption` stage drops zero rows, yet its output is two hundred rows smaller, because the difference for the first period does not exist by definition. - Behind the five served values stand 2400 source rows: an average of 480 per value, 588 for the largest, 372 for the smallest. The corruption of a row enters the served number diluted by this weight. - In the uninstrumented pipeline the corruption caught is 0 and the detection lag is **undefined**; the sentence "no corruption was caught" is an announcement that no measurement was made. - The rule of the curriculum: the number of a data pipeline is after how many steps a wrong row is caught; a pipeline whose check that catches the corruption is not written counts as unmeasured. ## Next Step The pipeline of this lesson ran with an intact source, and the trace table gave the expected number at every stage. The source being intact was an assumption; in production the source does not arrive intact. The records of a period arrive late, the same record is written twice, the name of a field changes, a reading is sent in another unit, the partition of a region is never written, a number field turns into text. None of these accidents is an attack, and all of them happen on the producer side. The next lesson defines these six classes by name and passes them through the same pipeline; what is to be measured is **at which stage** each corruption shows its effect. The most uncomfortable column of the table will be the last one: three of the six classes leave the five values the pipeline serves **exactly the same**.