Unit 03 of 03
Orchestration and Observability
Counting how a workflow with dependencies is built as a graph and what the rest does when one task fails: in a graph of 48 nodes and 47 edges, the consumption task also depending on the screening of the previous period and five tasks being unable to run at all when the ingestion task of the first period fails; how many times and when the time, event and data readiness triggers run the same job; the rows a retry multiplies in a pipeline that is not idempotent and the failure an alert threshold misses; a half-written partition making serving wrong in every region whereas an atomic write never writes the partition and so is caught by the period-aware completeness check; how many periods later the freshness, volume and schema evolution indicators show a corruption; and, in the environment and version management of the pipeline code, the different result the same code produces in two environments.
Published Updated
Lessons
- 01 Workflow Schedulers Writing the dependencies between the steps of a pipeline as a graph and measuring how many of the remaining tasks cannot run at all when one task fails: twelve periods with four task types give 48 nodes and 47 edges; because the consumption task also depends on the screening of the previous period, 5 tasks cannot run when the ingest task of the first period fails, 4 when the screening of the seventh period fails and 1 when its consumption fails; when the eleven edges between periods are not written, the total blast radius falls from 116 to 72 and 44 tasks are left behind without being rerun, whereas in a manual run all 48 tasks are rerun for a single failed task. 14 min read
- 02 Dependency and Triggering Measuring what starts the task at the head of the chain and how many times and when the same job runs: while the load of the source starts one step late in three periods and arrives in two parts in four periods, the fixed calendar finishes 3 of 12 runs empty and 3 partial, and by leaving six periods missing it drops consumption from 2200 to 900 rows; when a safety window of two periods is added to the calendar, the runs rise to 24 and all twelve periods are processed a second time, yet one period is still missing; the readiness trigger gives the full result with 12 runs and without empty or partial runs, and its cost is a total lag of seven periods; when the readiness criterion is tied to a fixed row count, however, four periods are never triggered. 14 min read
- 03 Retry and Alerting Measuring what a task failing in the middle does to serving and whether retrying repairs the pipeline: when the partition of the seventh period is written halfway at 120 rows, the raw rows stay at 2320, both the row count and the period-aware completeness check stay green, yet serving is wrong in every region (between −49.09 and −121.82); with an atomic write the same task leaves no rows, consumption falls from 2200 to 1800, the deviation stays between −3.25 and +12.80 and both checks fire; the write with a three-try retry that appends duplicates 240 rows and lifts the raw count of the store to 2640, with the keyed write the duplication is zero, and when the retry succeeds the task alert goes silent. 14 min read
- 04 Pipeline Observability Counting what three indicators placed on top of a running pipeline see: the freshness, volume and schema indicators look at a total of 14 observation points and none of them reads the value of a cell, so of the six corruption classes only the unit change, which alters the value, stays invisible to all three. Volume shows three classes in the same period, schema two classes in the same period, freshness the field rename one period later; while an indicator fires for at most one period the corruption lasts two periods, and in two classes serving does not change at all although the indicator fires. As the band widens from 0.05 to 0.30, the number of classes volume sees falls from 3 to 0 and the largest loss the band hides rises from 10 rows to 60 rows; when the freshness tolerance is 0 the lag falls from 1 period to 0, but a false alarm arises on a healthy pipeline. 5 min read
- 05 Environment and Version Management Counting the different result the same pipeline code produces in three environments and the identity that is lost when no version stamp is written: the development, staging and production environments produce 400, 1000 and 2200 consumption rows respectively from the same 2400 rows and their reports differ from one another; the number of loaded partitions shifts serving between 20.41 and 62.55, the rule version between 20.77 and 35.87, and because the two effects are of the same order of magnitude they cannot be told apart by looking at serving, and in one region they cancel each other and bring the difference down to −5.72. In an archive of eleven runs, when the rule version changes after the seventh period, the share of the code in the observed jump is between 20.40 and 34.71 and stays within the range of 0.42 to 38.30 that the usual period movement would show if the code never changed; in four regions the data goes down while the report goes up. In the unstamped archive none of the 11 reports carries an identity and identification requires 22 runs. The lesson is the close of the course. 13 min read