What Is Data Engineering

The AI Engineering curriculum, opened by the Introduction to AI Engineering course, built the machinery around the model in one hundred thirty-two lessons and wrote down the cost of every part. Throughout those lessons the data entering the model was assumed to be ready: there was a table, it was read; there was a feature, it was computed. How that table got filled, what it takes to keep it running in production, and who will know when it breaks were never asked.

The subject of this curriculum is that pipeline, and one peculiar flaw of the pipeline defines the whole measure: a pipeline gives notice when it crashes, and says nothing when it carries wrong data. A crashing pipeline leaves an error record, triggers an alert, piles up a queue; someone looks. A pipeline that carries wrong rows runs, finishes, writes the table, and the dashboard stays green. The job of this lesson is to build that pipeline and to count: how many rows go in, how many each stage carries, how many reach the served number.

• DL1. The code is a fiction. No real data store, file format, orchestrator or runtime is called; the pipeline is written with the plain standard library. The source is the fictional meter readings of a measurement network, and the seed is 20260218.
• DL2. The source is 200 subscribers x 12 periods = 2400 readings. Each reading carries an index value, the index only increases, and the unit is m3. Consumption is computed from the index difference.
• DL3. The pipeline has five stages: screen, dedupe, consumption, summary, serve. Each stage counts the rows it drops; there is no uncounted drop.
• DL4. The procedure for collection, cleaning, missing values and outliers is not repeated. It was measured in the Data Collection and Preparation course. The only thing added here is the stage counter: at which step and why a row dropped is written down.
• DL5. The procedure for schema, constraints and normalization is not repeated. It was measured in the Data Modeling and Relational Theory course. In this lesson the schema is reduced to the single question “is the required field there”.
• DL6. The pipeline is deterministic. random is not used, the generator is written inside the lesson, and the same seed gives the same 2400 rows. For this reason the measures are independent of the run, and all of the numbers below can be derived without a run.
• DL7. No product, vendor or format name is written; stores and processors are named by their kind. Time is not written as raw duration; lag is written as stages or periods.
• DL8. The resolution of the set is the lower bound of the measurement band: over 2400 rows the smallest step of a row ratio is 0.0004, over 55 summary cells 0.0182, over 5 served values 0.2000. No ratio is printed on the serving layer; the value itself is written.

The Five Stages of the Pipeline

The responsibility of data engineering fits in one sentence: to make data produced in one place usable in another. The word “usable” in that sentence hides a series of steps, and unless the steps are counted the responsibility cannot be measured. The pipeline below splits those steps into five parts: a row without a required field is screened out, the same subscriber-period pair is deduplicated, consumption is computed from the index difference, a summary is produced per region and period, and the period mean per region is served.

The order of these five is not arbitrary, and the code itself enforces two of the ties. The dedupe stage reads the subscriber and period fields of the row directly; if one of these fields is missing, the stage stops with an error. So screening has to come before deduplication, or the pipeline crashes. The second tie is quieter: the consumption stage produces one output row for every input row, and the summary stage adds these rows up. If a double record entered consumption without being deduplicated, the same subscriber-period difference would produce two output rows and would be added to the summary cell twice. When the first tie is violated, the pipeline stops and gives notice; when the second tie is violated, the pipeline runs, finishes and writes a wrong number. The stage order of a pipeline is the first decision that determines which errors will be loud and which will be silent.

# FICTIONAL pipeline -- no real store, format or orchestrator is called.
# The source is FICTIONAL meter readings of a measurement network, seed 20260218.
SEED, M32 = 20260218, 0xFFFFFFFF


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


REGION = ["east", "west", "north", "south", "central"]
TYPE = ["residential", "commercial"]
PERIODS = [f"2026-{a:02d}" for a in range(1, 13)]
SUBSCRIBERS = {}
for i in range(200):
    r = rng(SEED + 37 * i)
    SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)],
                                    "start": 10000 + int(r() * 60000)}


# One reading per subscriber per period. Unit m3 , the index only increases.
def generate_source():
    records = []
    for i, (sub, a) in enumerate(SUBSCRIBERS.items()):
        e = a["start"]
        for p in PERIODS:
            r = rng(SEED + 101 * i + 7 * PERIODS.index(p))
            e += 4 + int(r() * 46)
            records.append({"subscriber": sub, "period": p, "region": a["region"], "type": a["type"],
                            "index": e, "unit": "m3"})
    return records


def s1_screen(records):
    """A row that lacks a required field is dropped."""
    REQUIRED = ("subscriber", "period", "region", "index", "unit")
    keep = [k for k in records if all(a in k for a in REQUIRED)]
    return keep, len(records) - len(keep)


def s2_dedupe(records):
    seen, keep = set(), []
    for k in records:
        key = (k["subscriber"], k["period"])
        if key in seen:
            continue
        seen.add(key)
        keep.append(k)
    return keep, len(records) - len(keep)


def s3_consumption(records):
    """Consumption from the period difference. Without a previous period the row cannot be computed."""
    table = {(k["subscriber"], k["period"]): k for k in records}
    out, dropped = [], 0
    for k in records:
        i = PERIODS.index(k["period"])
        if i == 0:
            continue
        prev = table.get((k["subscriber"], PERIODS[i - 1]))
        if prev is None:
            dropped += 1
            continue
        try:
            f = int(k["index"]) - int(prev["index"])
        except (TypeError, ValueError):
            dropped += 1
            continue
        out.append(dict(k, m3=f))
    return out, dropped


def s4_summary(records):
    totals = {}
    for k in records:
        key = (k["region"], k["period"])
        totals[key] = totals.get(key, 0) + k["m3"]
    return totals, 0


def s5_serve(summary):
    """Serving: per-region mean over the periods."""
    region = {}
    for (b, d), v in summary.items():
        region.setdefault(b, []).append(v)
    return {b: round(sum(v) / len(v), 2) for b, v in region.items()}, 0


STAGES = [("screen", s1_screen), ("dedupe", s2_dedupe), ("consumption", s3_consumption),
          ("summary", s4_summary), ("serve", s5_serve)]


def pipeline(records):
    trace, data = [], records
    for name, f in STAGES:
        data, dropped = f(data)
        n = len(data) if hasattr(data, "__len__") else 0
        trace.append({"stage": name, "remaining": n, "dropped": dropped})
    return data, trace


source = generate_source()
served, trace = pipeline(source)
print("source rows", len(source), " subscribers", len(SUBSCRIBERS), " periods", len(PERIODS))
print("stage         remaining  dropped")
for a in trace:
    print(f"{a['stage']:<13}{a['remaining']:>10}{a['dropped']:>9}")
print("served")
for b, v in sorted(served.items(), key=lambda x: -x[1]):
    print(f"  {b:<8}{v:>9.2f}")

source rows 2400  subscribers 200  periods 12
stage         remaining  dropped
screen             2400        0
dedupe             2400        0
consumption        2200        0
summary              55        0
serve                 5        0
served
  central   1321.45
  north     1212.55
  south     1065.55
  east       881.36
  west       833.09

The first two rows of the trace table pass with zero drops: because the source is intact, there is no row with a missing field to screen out and no double record to deduplicate. The third row is the key point of the reading. The consumption stage drops zero rows, yet its output is two hundred rows smaller. This difference is not a loss; it is the definition itself: consumption is the index difference of two periods, and the difference for the first period of each subscriber does not exist. The first period of two hundred subscribers cannot be computed, but these are not faulty rows. That is why the “remaining” and “dropped” columns of a pipeline trace are kept separate: a row that does not come out by definition and a row removed because of an error cannot be written in the same column.

None of these numbers depends on the run. Because the pipeline is deterministic, the table can be derived without a run: 200 subscribers x 12 periods = 2400; each subscriber loses one period, so 200 x 11 = 2200; 5 regions x 11 periods = 55 cells; 5 regions = 5 values. A second run with the same seed gives the same table, so every difference seen in the comparisons below comes from the corruption itself, not from variability.

The Rows Behind a Served Value

The most looked-at place of a pipeline is its last stage, because the report is read from there. The last stage of this pipeline produces five numbers. How many rows stand behind those five numbers says how much of the pipeline they hide.

# How many rows stand behind each served value
v1, _ = s1_screen(source)
v2, _ = s2_dedupe(v1)
v3, _ = s3_consumption(v2)
summary, _ = s4_summary(v3)
subs, rows, cells = {}, {}, {}
for a in SUBSCRIBERS.values():
    subs[a["region"]] = subs.get(a["region"], 0) + 1
for k in v3:
    rows[k["region"]] = rows.get(k["region"], 0) + 1
for b, d in summary:
    cells[b] = cells.get(b, 0) + 1
print("region      served  summary  consumption  source  subscribers")
for b, v in sorted(served.items(), key=lambda x: -x[1]):
    print(f"{b:<9}{v:>9.2f}{cells[b]:>9}{rows[b]:>13}{subs[b] * 12:>8}{subs[b]:>13}")
print(f"{'total':<9}{len(served):>9}{len(summary):>9}{len(v3):>13}{len(source):>8}{len(SUBSCRIBERS):>13}")

region      served  summary  consumption  source  subscribers
central    1321.45       11          539     588           49
north      1212.55       11          506     552           46
south      1065.55       11          440     480           40
east        881.36       11          374     408           34
west        833.09       11          341     372           31
total            5       55         2200    2400          200

The table plainly writes the pipeline’s compression ratio. The only number visible in the report for central, 1321.45, stands on the 588 source rows of forty-nine subscribers over twelve periods, on the 539 consumption rows computed from them, and on eleven summary cells. The smallest value, west, stands on 372 source rows. Together the five values represent 2400 rows, that is, an average of 480 source rows per value.

This compression has two faces. The useful face is clear: nobody can decide by looking at two thousand four hundred rows, and that is exactly the job of a summary. Its cost is the subject of the measure. When a source row is corrupted, that corruption enters the served number with a weight of one in 480; for central, one in 588. The ratio of the consumption layer to the source is 0.9167, and since the smallest step of this ratio is 0.0004, it can be read to three digits; but writing a ratio on the serving layer is meaningless, because only five values exist there and the proportional weight of one value is 0.2000. What is to be written on the serving layer is the value itself.

From here comes the first structural result of the pipeline: the corruption of a single row dilutes so far that it is not visible in the served number, but the corruption of a hundred rows in the same direction does not dilute that far. At which threshold which corruption becomes visible is the real quantity this pipeline is to measure, and this lesson does not measure it yet; it only lays the ground.

Compression has one more side that does not show in the row count: the pipeline drops not only rows but also a distinguishing dimension. The output of the first three stages is keyed by the subscriber and period pair, and two hundred separate subscribers stand in the table. The output of the summary stage, however, is keyed by the region and period pair; the subscriber column does not remain there. From none of the fifty-five cells can it be read which subscribers were added up. The serve stage does the same thing to the period axis: from none of the five values can it be read how much each period added. This is not a loss but the definition of a summary, yet its consequence in production is heavy. When a served number looks suspicious, walking backward cannot be done by reading the intermediate outputs; the first three stages of the pipeline have to be rerun. Writing at which stage the pipeline leaves which dimension is the same thing as writing in advance the cost of investigating a corruption.

The Count of the Uninstrumented Pipeline

The pipeline above runs, finishes and writes five numbers. The only thing known about this pipeline is how many rows it carries. Nothing is known about whether the rows it carries are correct, because the pipeline has no step that asks this.

From here on, in every lesson of the curriculum, three numbers will stand side by side:

1. Uninstrumented pipeline. No check is enabled. Corruption caught 0, detection lag undefined. The pipeline in this lesson is exactly this.
2. Instrumented pipeline. The same workload, with checks added. The corruption caught and the detection lag are written as a stage number.
3. Cost. The detection lag, the good rows wrongly blocked, the rows reprocessed and the stages added.

The items in the third point are the most often skipped side of this curriculum. Adding a check is not a free improvement: a check can also block a good row, can stop the pipeline needlessly, can duplicate rows during reprocessing and can itself be flawed. That is why, when the instrumented pipeline is written, not only the corruption caught but also the cost of the check itself goes into the same table. When the number of corruptions caught is written alone, the improvement is overstated.

The word “undefined” in the first point should be read carefully. The detection lag of the uninstrumented pipeline is not large, it does not exist; since nothing measures, a quantity called lag does not arise. This is the difference between zero and undefined, and it is often confused in reporting: the sentence “no corruption was caught” is, when there is no check, not an announcement of success but an announcement that no measurement was made.

From here comes the rule of the curriculum, and it holds through six courses: the number of a data pipeline is not the rows it carries, but after how many steps a wrong row is caught. A pipeline whose check that catches the corruption is not written counts as unmeasured. The trace of the pipeline above looks flawless, because the only thing the trace shows is how many rows the stages hand over to one another. About the content of the rows handed over, the trace is silent, and the silence is written in green.

Summary

• The responsibility of data engineering is to make data produced in one place usable in another; this responsibility becomes measurable only when the stages of the pipeline and the rows each stage carries are counted.
• A pipeline gives notice when it crashes and says nothing when it carries wrong data; the measure of this curriculum is the second case.
• The five-stage pipeline carries 2400 source rows as 2400, 2400, 2200, 55 and 5; the consumption stage drops zero rows, yet its output is two hundred rows smaller, because the difference for the first period does not exist by definition.
• Behind the five served values stand 2400 source rows: an average of 480 per value, 588 for the largest, 372 for the smallest. The corruption of a row enters the served number diluted by this weight.
• In the uninstrumented pipeline the corruption caught is 0 and the detection lag is undefined; the sentence “no corruption was caught” is an announcement that no measurement was made.
• The rule of the curriculum: the number of a data pipeline is after how many steps a wrong row is caught; a pipeline whose check that catches the corruption is not written counts as unmeasured.

Next Step

The pipeline of this lesson ran with an intact source, and the trace table gave the expected number at every stage. The source being intact was an assumption; in production the source does not arrive intact. The records of a period arrive late, the same record is written twice, the name of a field changes, a reading is sent in another unit, the partition of a region is never written, a number field turns into text. None of these accidents is an attack, and all of them happen on the producer side. The next lesson defines these six classes by name and passes them through the same pipeline; what is to be measured is at which stage each corruption shows its effect. The most uncomfortable column of the table will be the last one: three of the six classes leave the five values the pipeline serves exactly the same.