Skip to content
academia.sh
tr
Log in

Course Intermediate

Data Pipelines and Orchestration

Published Updated

By the end of this course

Start course

01

Data Ingestion

Counting how data is ingested from the source into the pipeline and what each method misses: the rows transported by full load, the late-arriving change that incremental load misses, and the case where change data capture sees both; the calls a connector completes under pagination and a rate limit and the requests a retry adds; the work that topic and partition counts distribute to a consumer group and the order preserved within each partition; the result a windowing decision changes for a late-arriving record and the cost of carrying state; the 19 rows that at-most-once delivery loses, shifting every region, against the 19 duplicates of at-least-once delivery that never change the serving thanks to a keyed write; the loop that writing processed data back to an operational system opens and the number of stages after which that loop returns to its own input.

  1. 01 Ingestion Types Counting side by side what three ingestion methods transport and what each misses: a graph built from four tasks per period carries 48 nodes and 47 edges, full load transports 15,560 rows over twelve runs and misses no change, incremental load transports 2,360 rows and misses 148 changes, and change data capture transports 2,508 rows and misses none. The 40 late-arriving rows and 108 uncorrected rows that incremental load misses pull the serving down in every region, yet the row count check stays silent at a ratio of 0.0167 and only the distribution check catches the corruption, at 0.3469 in stage 4.
  2. 02 Connectors and API Consumption Counting the calls a connector completes and the rows it receives under pagination and a rate limit: with a quota of eight calls per window, on a 25-row page 83 of 96 calls complete and 2075 rows are received, on a 100-row page 20 of 24 calls complete and 2000 rows are received, so the rows missed per dropped call grow from 25 to 100 with the page size. Retry eats the quota on the small page and drops the rows received from 2075 to 2025, delays 112 pages by one window, and the 2026-12 period is never received; in return, the serving deviation falls from −374 to −125, because the loss stops being scattered and collects at the tail of the queue. The connector's own call ledger reports the shortfall without delay: expected 96, completed 83.
  3. 03 Messaging Systems Counting side by side the work that the partition count distributes to a consumer group and the order preserved per partition: in a group of four consumers a single partition leaves three consumers idle, four partitions put all of them to work. In a topic partitioned by the subscriber key no subscriber is split and consumption stays at 2200 rows at every partition count, but the load skews between 708 and 492. The most balanced key, the record key, squeezes the load into 583 to 621 and splits all 200 subscribers: consumption falls to 527 rows at four partitions and the serving shifts between 616 and 977 in every region. The period key, on the other hand, loses 1,200 of the 2200 rows at two partitions while moving the serving only between −15.29 and +16.15.
  4. 04 Stream Processing Turning the window decision and the carried state into numbers: when the period window is closed the moment the period ends, 40 records that arrive one period late are dropped and 10 of the 55 summary cells are polluted; when the window is kept open one period longer the dropped rows fall to 0, but the 2026-07 cell is written in 2026-08 and the carried state grows from 400 entries to 600. Tolerance is a bet on delay: when the same records are two periods late, tolerance 1 gives exactly the same result as tolerance 0. In none of the nine setups do the five served values change, only the number of polluted cells does. If the window never closes the state grows to 2400 and the mode turns back into batch processing; with the state lost without a checkpoint, 200 rows cannot be computed, the summary falls to 50 cells, and the serving shifts in every region.
  5. 05 Delivery Guarantees Turning the three delivery modes into numbers on the same pipeline: with at-most-once delivery 19 rows of period 2026-07 are lost, consumption falls from 2200 to 2162, 10 of the 55 summary cells are polluted, and the serving shifts in every region (−9.18 … −28.36); with at-least-once delivery 19 rows arrive twice but consumption stays at 2200 and the serving difference comes out empty, because the dedupe stage swallows the duplication; with exactly-once delivery the keyed write zeroes out both. The real finding is in the check column: the uniqueness check catches the duplication at stage 0, while the loss slips past every check — the row ratio of 0.0079 stays below the 0.05 threshold and the largest cell deviation of 0.2488 stays below the 0.25 threshold. Loss is silent, duplication is repairable.
  6. 06 Reverse ETL Counting the circle that opens when processed data is written back to an operational system: when the regional threshold computed in the serving layer is written back, the chain returns to its own input after seven stages through ingest, screen, dedupe, consumption, summary, serve and write_back. What the written-back value does in the next run is measured: the threshold written by the first run clips 1129 readings in the second run, the serving shifts in every run (the largest difference between runs is 47.00, then 10.90 and 14.36), and the loop stops at the fourth run. Where it stops is not the baseline: while the consumption rows stay at 2200, 55 of the 55 summary cells are polluted, the serving deviates by −19.27 … −59.46, and because the largest cell deviation of 0.1788 stays below the 0.25 threshold, no check fires. When the written-back value is kept apart from the pipeline's input, every run gives the same result.

02

Transformation

Counting where, in which layer and with which guarantee the transformation is done: the effect of running the transformation at the source or at the target on the transported rows and the cells read, the tasks that dependency resolution orders in query-based modeling and what happens when a cycle slips through, the rows each of the raw, refined and serving layers keeps and which layer can stop a corruption, the corruption classes caught by data tests and expectation definitions against the residue that none of them catches, and the way applying a rule change to only two periods shifts the report by 16.98 to 29.57 from full application while both runs look internally consistent and trip no check.

  1. 01 ETL and ELT Counting whether the transformation runs at the source or at the target: when the boundary is moved to the source, the rows transported to the target fall from 2400 to 55 and the transported cells from 14400 to 165, but the total number of cells read by all three setups stays fixed at 33110 and only the identity of the reader changes; the hybrid boundary lowers the transported rows to 2200 while raising the transported cells to 15400; when a reporting rule changes, the number of rules that can be recomputed at the target is 3 for the setup that keeps the raw data, 1 for the hybrid setup and 0 for the setup that transforms at the source, and the last setup has to go back to the source and can recompute only three periods.
  2. 02 Transformation Frameworks Counting how transformations are written as models that refer to one another and where the order comes from: over eleven models and thirteen declared dependencies the resolver order gives 0 violations, while the hand-kept write order violates 1 dependency and runs 7 models with missing input, and the fixed list violates 5 dependencies and runs 10 models with missing input; the violating run produces no error and lowers the serving output in every region by 17.46 to 28.37; when one more reference is added to a query text, the resolver orders only 3 of the 11 models and does not start the run, while the manual order runs all eleven and silently breaks one edge of the cycle; because a single undeclared read never appears in the graph, the resolver reports 0 violations while the actual violation is 1.
  3. 03 Staged Layers Counting the rows each of the raw, refined and serving layers keeps and the layer in which a corruption can be stopped: the sound pipeline leaves 2400 rows in the raw layer, 2200 rows in the refined layer and 55 summary cells in the serving layer; of the six corruption classes four stop in the raw layer, one in the refined layer and one only in the serving layer, and three of them do not change the served value at all; the three-layer setup stores 4655 rows and stops all six classes, while the two-layer setup stops the same six with 2455 rows but catches the unit change one layer late, and the single-layer setup stores 55 cells, stops only two classes and never sees four.
  4. 04 Testable Transformations Counting what expectations placed on the transformation's own output catch and what they do not: six expectations never fire on the sound pipeline and catch four of the six corruption classes, while duplication and type drift are swallowed by the transformation itself and never appear in the output. The unit change is caught by an expectation on the output at the third stage, and no baseline is needed for it, although the serving stays exactly the same. A rule error that shifts the period label back by one period leaves 2200 rows, 55 cells and five serving values exactly the same, so it trips none of the six expectations; only a unit test on a fixed input catches it.
  5. 05 Backfill Counting the application of a changed rule to past periods: when the rule is applied to all twelve periods, 12 periods and 2200 rows are recomputed, while applying it to only two periods takes 2 periods and 400 rows, but the report deviates from the full application by 16.98 to 29.57. The mixed report breaks none of the six expectations, comes out internally consistent, and trips no check, because the 2.75% jump the rule creates in the central cells stays below the median of the natural period volatility. The source of the deviation is not the data but which periods were recomputed; when the row is stamped with the rule version, the mixing becomes visible with a single comparison.

03

Orchestration and Observability

Counting how a workflow with dependencies is built as a graph and what the rest does when one task fails: in a graph of 48 nodes and 47 edges, the consumption task also depending on the screening of the previous period and five tasks being unable to run at all when the ingestion task of the first period fails; how many times and when the time, event and data readiness triggers run the same job; the rows a retry multiplies in a pipeline that is not idempotent and the failure an alert threshold misses; a half-written partition making serving wrong in every region whereas an atomic write never writes the partition and so is caught by the period-aware completeness check; how many periods later the freshness, volume and schema evolution indicators show a corruption; and, in the environment and version management of the pipeline code, the different result the same code produces in two environments.

  1. 01 Workflow Schedulers Writing the dependencies between the steps of a pipeline as a graph and measuring how many of the remaining tasks cannot run at all when one task fails: twelve periods with four task types give 48 nodes and 47 edges; because the consumption task also depends on the screening of the previous period, 5 tasks cannot run when the ingest task of the first period fails, 4 when the screening of the seventh period fails and 1 when its consumption fails; when the eleven edges between periods are not written, the total blast radius falls from 116 to 72 and 44 tasks are left behind without being rerun, whereas in a manual run all 48 tasks are rerun for a single failed task.
  2. 02 Dependency and Triggering Measuring what starts the task at the head of the chain and how many times and when the same job runs: while the load of the source starts one step late in three periods and arrives in two parts in four periods, the fixed calendar finishes 3 of 12 runs empty and 3 partial, and by leaving six periods missing it drops consumption from 2200 to 900 rows; when a safety window of two periods is added to the calendar, the runs rise to 24 and all twelve periods are processed a second time, yet one period is still missing; the readiness trigger gives the full result with 12 runs and without empty or partial runs, and its cost is a total lag of seven periods; when the readiness criterion is tied to a fixed row count, however, four periods are never triggered.
  3. 03 Retry and Alerting Measuring what a task failing in the middle does to serving and whether retrying repairs the pipeline: when the partition of the seventh period is written halfway at 120 rows, the raw rows stay at 2320, both the row count and the period-aware completeness check stay green, yet serving is wrong in every region (between −49.09 and −121.82); with an atomic write the same task leaves no rows, consumption falls from 2200 to 1800, the deviation stays between −3.25 and +12.80 and both checks fire; the write with a three-try retry that appends duplicates 240 rows and lifts the raw count of the store to 2640, with the keyed write the duplication is zero, and when the retry succeeds the task alert goes silent.
  4. 04 Pipeline Observability Counting what three indicators placed on top of a running pipeline see: the freshness, volume and schema indicators look at a total of 14 observation points and none of them reads the value of a cell, so of the six corruption classes only the unit change, which alters the value, stays invisible to all three. Volume shows three classes in the same period, schema two classes in the same period, freshness the field rename one period later; while an indicator fires for at most one period the corruption lasts two periods, and in two classes serving does not change at all although the indicator fires. As the band widens from 0.05 to 0.30, the number of classes volume sees falls from 3 to 0 and the largest loss the band hides rises from 10 rows to 60 rows; when the freshness tolerance is 0 the lag falls from 1 period to 0, but a false alarm arises on a healthy pipeline.
  5. 05 Environment and Version Management Counting the different result the same pipeline code produces in three environments and the identity that is lost when no version stamp is written: the development, staging and production environments produce 400, 1000 and 2200 consumption rows respectively from the same 2400 rows and their reports differ from one another; the number of loaded partitions shifts serving between 20.41 and 62.55, the rule version between 20.77 and 35.87, and because the two effects are of the same order of magnitude they cannot be told apart by looking at serving, and in one region they cancel each other and bring the difference down to −5.72. In an archive of eleven runs, when the rule version changes after the seventh period, the share of the code in the observed jump is between 20.40 and 34.71 and stays within the range of 0.42 to 38.30 that the usual period movement would show if the code never changed; in four regions the data goes down while the report goes up. In the unstamped archive none of the 11 reports carries an identity and identification requires 22 runs. The lesson is the close of the course.

Final exam

Data Pipelines and Orchestration — Final Exam

In preparation

This course's question bank is not complete yet. The exam opens here automatically once enough questions are added.

Start typing to search.

↑↓ ↵ Esc navigate · open · close