--- title: 'Network Operations and Automation' source: 'https://academia.sh/en/courses/network-operations' language: en updated: '2026-08-17T04:39:12+00:00' license: 'CC BY-SA 4.0' lessons: 16 outcomes: - 'Designs traffic distribution and caching layers end to end' - 'Collects network telemetry and turns it into meaningful dashboards' - 'Manages network configuration as code' --- # Network Operations and Automation In operations the operator touches a single lever and that lever applies to forty subjects at once; from traffic distribution to telemetry and configuration automation, the course counts the minority each decision serves wrongly. ## Traffic Management The difference between layer 4 and layer 7 balancing, the tail distribution rules leave on subjects with unequal capacity, the purpose of the two proxy types, edge caching's pull and push models, and what classification and shaping make whom pay. ### Load Balancers Source: https://academia.sh/en/courses/network-operations/load-balancers A balancer working at the transport layer distributes connections, one working at the application layer distributes requests; on the same forty subjects, one overruns 17 subjects by 438 units, the other 16 subjects by 310 units, and the transport layer's own gauge shows the same number for all forty of the forty subjects. The previous course measured how a limit gets **written**: which power, which threshold, which rule sequence, which metric. No lesson asked what happens after the limit is written. A written limit sits in a living system; its load changes, its drift goes unseen by anyone. **Writing a limit and operating what was written are separate jobs.** Every decision up to this point was for **a single object**: a packet, a flow, a client, a rule line. Operations does not work this way. The operator does not touch objects one by one; they touch **a lever** — a distribution rule, a sampling rate, a template — and that lever gets applied to **forty subjects at once.** This is this course's question, and the first lever is the load balancer. #### Lever and Subject The same population is measured throughout the course: **forty subjects.** The subjects are not identical to each other. Their capacities spread between 22 and 99 units, their delays differ, each belongs to one of three service classes, and nine of them are special devices that a future lever has to leave out. We know what is in the population **because we wrote it**; the measurement's oracle is the fiction itself. What is measured is not the lever's nature. **An operations decision's number is how many subjects it serves wrongly.** The lever is single, the subjects are forty, and the minority carrying the difference between them is called, in this course, the **lever's tail.** Tail has only this one sense in this course: the set of subjects the lever serves wrongly. The tail has two separate numbers, and both are written at every measurement. The first is **how many subjects** stay in the tail. The second is **how many** of those in the tail **do not show up in the lever's own gauge.** This lesson opens with the cleanest example of the second. This is where it becomes visible why an average is not enough. A load split across forty subjects can stay below capacity on average while several subjects at the same time sit far above their capacity. An average folds **both ends** of the distribution into the same number: the overrunning subject's excess and the idle subject's shortfall cancel out in the sum. The tail measure refuses exactly this cancellation. This is why, in every table below, the overrunning subject count and the idle unit count are **separate columns** and are never reduced to a single number in any measurement. #### What the Balancer Reads A **load balancer** is an intermediary that splits incoming work across a set of subjects. Its role and the distinction by layer were established in the Traffic Layer course of the System Design and Distributed Systems curriculum; that account is not repeated here. The difference is one sentence: **there the balancer's design was measured, here the tail a single lever leaves on forty subjects with unequal capacity is measured.** The side of this distinction that matters for this lesson is what the balancer **can read.** A balancer working at the **transport layer** sees the connection four-tuple: source and destination address, source and destination port. It does not open what is inside the connection. The unit of distribution is therefore the **connection**: it binds every incoming new connection to a subject and stays bound to it until that connection closes. A balancer working at the **application layer** terminates the connection on itself, reads the request, and forwards it to subjects over connections it opens itself. The unit of distribution is the **request.** Because it can read the request's path and header fields, it can sort subjects into classes and send each class to a separate pool. The choice between the two is a configuration line: ```text # taught configuration transcript, not run balancer transport { listen *:443 unit connection rule round-robin pool subject-01 ... subject-40 } balancer application { listen *:443 unit request rule round-robin match /interactive/* -> pool-interactive match /batch/* -> pool-batch match /standby/* -> pool-standby } ``` This transcript is documentation, not a measurement. The numbers come from the measurement below. #### A Health Check Is a Lever Too The balancer has another lever besides distribution: the **health check.** An interval, a timeout, and how many consecutive failures are enough to pull a subject out of the pool are written down. These three numbers, too, get applied not to a single subject but to **all forty at once.** Whether the check's presence affects the service was measured in the Traffic Layer course of the System Design and Distributed Systems curriculum and is not repeated here. The difference is one sentence: **there the check's presence was measured, here the tail a single threshold leaves on forty subjects.** The threshold's tail has two ends, and the ends do not close together. If the timeout is written loose, a subject that has slowed down but still responds is counted healthy and keeps having work routed to it; the tail is the requests going to that subject. If the timeout is written tight, subjects whose delay is already high fail the check and drop out of the pool; the tail this time is the remaining subjects that take on the dropped subjects' load. Because the forty subjects' delays are all different, **a single timeout cannot close both ends at once.** It should also not be forgotten that what the check reads is the subject's **own announcement**: the subject says it is healthy, the check believes it. The observation left by the Application Layer Protocols course holds here too — nothing verifies the announcement, only its absence not arriving is information. The same duality shows up when a subject is pulled out of the pool. Connections that exist at the moment of removal are not cut; a **draining** period is granted. This period, too, is a single number applied to all forty subjects at once: written long, a failed subject's work sits there for a long time; written short, half-finished work gets cut off. It is the same lever; only who makes up the tail changes. #### The Lever's Own Gauge The only thing the transport-layer balancer can count is the connection. When it distributes with a round-robin rule, its own counter, by definition, shows **the same value** for all forty of the forty subjects. The gauge is flat, and being flat reads like proof the rule is correct. Yet what the connections carry is not equal. A persistent connection can carry one request, or thirty. The balancer cannot see this, because what it cannot see is the connection's inside. **What gets distributed equally and what the subject is actually loaded with are not the same thing**, and the difference between the two never shows up in the lever's gauge. The application-layer balancer closes this gap: it counts requests, distributes requests, and its gauge shows the real unit of work. The measurement asks how much of this it actually solves. The measurement's assumptions: - **TM1** — The forty subjects are the course's fixed set; the capacity, delay, and class fields are not changed during the measurement. The oracle is this set itself. - **TM2** — Four hundred connections are generated, and each carries a variable number of requests. One-eighth of the connections are long-lived and carry between twenty and forty requests; the rest carry between one and four. - **TM3** — The balancer distributes with a round-robin rule. The rule is the same across all three measurements; **the only thing that changes is the unit of distribution** — connection, request, or request within a class pool. - **TM4** — The transport-layer balancer cannot read the number of requests a connection carries; the application-layer one can. The fiction codes this distinction directly. - **TM5** — Capacity is the number of requests a subject can serve at once. Every unit above this number is overrun; every unit below it is idle. - **TM6** — The measurement is not a network measurement; no connection is opened. What is counted is the distribution in the fiction itself. #### The Measurement ```python """Layer 4 vs layer 7 balancing: two levers on the same forty subjects.""" SEED = 20260812 CLASSES = ("interactive", "batch", "standby") def generator(seed): d = seed % 2147483646 + 1 def r(n): nonlocal d d = (d * 48271) % 2147483647 return d % n return r def subjects(count=40, seed=SEED): r, result = generator(seed), [] for i in range(count): result.append({"no": i + 1, "capacity": 20 + r(81), "delay": 5 + r(45), "class": CLASSES[r(3)], "special": r(9) == 0}) return result def connections(count=400, seed=SEED + 11): r, result = generator(seed), [] for i in range(count): long_lived = r(8) == 0 result.append({"requests": 20 + r(21) if long_lived else 1 + r(4), "class": CLASSES[r(3)]}) return result def layer4(conns, subs): """Distributes the connection; does not know how many requests it carries.""" share = {s["no"]: 0 for s in subs} count = {s["no"]: 0 for s in subs} for i, c in enumerate(conns): s = subs[i % len(subs)] count[s["no"]] += 1 share[s["no"]] += c["requests"] return share, count def layer7(conns, subs, pooled): """Distributes the request; to the class pool if asked.""" pool = {c: [s for s in subs if s["class"] == c] for c in CLASSES} count = {c: 0 for c in CLASSES} share, i = {s["no"]: 0 for s in subs}, 0 for c in conns: p = pool[c["class"]] if pooled else subs for _ in range(c["requests"]): if pooled: share[p[count[c["class"]] % len(p)]["no"]] += 1 count[c["class"]] += 1 else: share[p[i % len(p)]["no"]] += 1 i += 1 return share def overrun(share, subs): n = units = 0 for s in subs: over = share[s["no"]] - s["capacity"] if over > 0: n, units = n + 1, units + over return n, units subs, conns = subjects(), connections() print(f"subjects {len(subs)} | total capacity {sum(s['capacity'] for s in subs)} | " f"smallest {min(s['capacity'] for s in subs)} largest {max(s['capacity'] for s in subs)}") print(f"connections {len(conns)} | requests {sum(c['requests'] for c in conns)}") for c in CLASSES: h = [s for s in subs if s["class"] == c] print(f" {c:12s} subjects {len(h):2d} capacity {sum(s['capacity'] for s in h):5d} " f"requests {sum(x['requests'] for x in conns if x['class'] == c):5d}") share4, count = layer4(conns, subs) measurements = (("layer 4 (connection)", share4), ("layer 7 (request)", layer7(conns, subs, False)), ("layer 7 (class pool)", layer7(conns, subs, True))) print() print(f"{'lever':<24s} {'over':>5s} {'over units':>11s} {'idle':>7s} " f"{'min':>6s} {'max':>7s} {'no load at all':>16s}") for name, share in measurements: n, units = overrun(share, subs) print(f"{name:<24s} {n:5d} {units:11d} " f"{sum(max(0, s['capacity'] - share[s['no']]) for s in subs):7d} " f"{min(share.values()):6d} {max(share.values()):7d} " f"{sum(1 for s in subs if share[s['no']] == 0):16d}") print() print(f"layer 4's own gauge — connection count: min {min(count.values())}, " f"max {max(count.values())}; {overrun(share4, subs)[0]} subjects over capacity " f"do not show up in this gauge") ``` ``` subjects 40 | total capacity 2336 | smallest 22 largest 99 connections 400 | requests 2193 interactive subjects 11 capacity 580 requests 655 batch subjects 20 capacity 1346 requests 700 standby subjects 9 capacity 410 requests 838 lever over over units idle min max no load at all layer 4 (connection) 17 438 581 21 164 0 layer 7 (request) 16 310 453 54 55 0 layer 7 (class pool) 18 558 701 35 94 0 layer 4's own gauge — connection count: min 10, max 10; 17 subjects over capacity do not show up in this gauge ``` #### Where the Tail Is The last line is this lesson's core. The transport-layer balancer's counter shows **10** for all forty of the forty subjects; there is no difference at all between min and max. At the same time, **17** of the forty subjects have overrun their capacity, and the real load spreads between **21** and **164**. The lever is working correctly — by its own definition. **The tail does not show up at the dimension the lever measures.** Moving to the application layer solves part of this. Once requests are distributed, load tightens to between **54** and **55**; now the gauge actually shows what is being distributed. But overrunning subjects drop from **17** to **16**, overrun units from **438** to **310**. The gain across forty subjects is **a single subject.** In a set of forty subjects, the smallest measurable difference is **1/40 = 0.025**; a one-subject gain is exactly at that threshold, and a smaller difference cannot be defended with this set. The reason the gain stays small is clear. The application layer distributes **equal requests**, but the subjects are not equal: capacities run between **22** and **99**. Dropping 54 requests on everyone is a threefold overrun for a subject with capacity 22, and roughly half-idle for one with capacity 99. Raising the lever's resolution **moved** the tail, it did not remove it. The third line says this more clearly. The class pool uses the real capability the application layer buys: reading the request's path and picking the subject. The result is **worse** — overrunning subjects **18**, overrun units **558**. The reason sits in the three lines above: the standby class carries **9** subjects and **410** capacity but takes **838** requests; the batch class carries **20** subjects and **1346** capacity for **700** requests. Splitting into pools made subjects that had capacity unusable. Idle units climb from **453** to **701**. This is why the fourth column is part of the measurement. In all three levers, **subjects with no load at all is 0**: no subject is left out, no one is standing at the door outside. And yet **581**, **453**, and **701** units sit idle in the system. A gauge that counts only overrun never shows this column and never says a quarter of capacity went unused. The only reason we can write this table is that we **know** the capacities. In operations, such a column does not sit ready-made: the balancer does not measure a subject's capacity, it counts the work it gives it. The overrun column can only be computed once capacity is brought in from somewhere else — a separate per-subject measurement; without that, the gauge stays flat and stays no less wrong. In this lesson's fiction we hold the oracle in hand, so we can put the two columns side by side, and this is exactly what the rest of the course asks: **where is the difference between the lever's gauge and the lever's result read from?** The next two topics count this difference — first in the traffic itself, then in telemetry. #### Summary - Operations' unit is not a single object but a **lever**: the operator writes a rule, the rule gets applied to forty subjects at once, and what is measured is the minority the lever serves wrongly — the **tail.** - The transport-layer balancer distributes **connections** and cannot see the requests a connection carries; its own gauge shows **10** for all forty subjects while the real load is between **21** and **164** and **17** subjects have overrun capacity. - The application-layer balancer distributes **requests** and tightens load to a **54–55** range; overrunning subjects drop only to **16**, overrun units to **310**. The gain across forty subjects is **one subject.** - Distributing equal requests is not equal load: because capacities run between **22** and **99**, the same share is overrun for one subject and idle for another. - Splitting into class pools grows the tail — overrunning subjects **18**, idle units **701** — because pool capacity and the requests the pool receives do not match. - A health check is also a single lever, and its tail has two ends: a loose threshold keeps sending work to a slowed subject, a tight threshold drops high-delay subjects and piles their load onto the rest; because the forty subjects' delays all differ, a single threshold cannot close both ends at once. #### Next Step In this lesson, the rule was the same across all three measurements: round-robin distribution. The only thing that changed was the unit of distribution, and changing the unit moved the tail by about one-fortieth. So the real question is not the unit but **the rule itself**: does a rule that splits load by capacity, or picks the least-loaded subject at every step, actually remove the tail? The next lesson puts three more distribution rules side by side against round-robin on the same forty subjects at two separate loads, and counts what improving the rule does to the tail — whether it removes it, or only chooses who ends up in it. ### Load Balancing Algorithms Source: https://academia.sh/en/courses/network-operations/load-balancing-algorithms Four distribution rules are run on the same forty subjects at two separate loads; at load 1200 overrunning subjects come out 6 / 0 / 0, at load 2600 23 / 40 / 40 and overrun units 533 / 245 / 264, 1160 units sit idle at 1200, and what makes the proportional rule's overrun column look clean is the 19 units it never distributes. The previous lesson held the rule constant and changed only the unit of distribution: request instead of connection. The gain across the forty-subject set was a single subject. So the real question is not in the unit but in **the rule itself.** Does a rule that splits load by capacity, or picks the least-loaded subject at every step, actually remove the tail? This is what this lesson measures, and the answer comes in a single table at two separate loads. A rule being "good" turns out true at one load and misleading at another. #### Four Rules A distribution rule is a function that binds an incoming job to one of forty subjects. Four rules are measured, and all four differ from each other in **what they read.** **Round robin** reads nothing. It cycles through the subjects in turn and gives every job to the next one. It keeps no state, does not know capacity, does not know instantaneous load. **Proportional** distribution allocates each subject a share proportional to its capacity. It reads capacity but not instantaneous load: it computes the shares up front and distributes accordingly. **Least connections** picks the currently least-loaded subject at every step. The form measured here is weighted: the comparison looks not at the raw connection count but at **the ratio of connection count to capacity.** It reads both capacity and instantaneous load, which makes it the most expensive of the four rules — it has to see all forty subjects' state for every decision. **Consistent hashing** drops a key of the job — a session ID, a client address — onto a ring and gives it to the first subject clockwise on the ring. It reads neither capacity nor instantaneous load; the only thing it reads is the key. As long as the pool does not change, the same key keeps going to the same subject; when a subject is added or removed, only a small share of the keys moves — the measurement below counts that share. Comparing these rules was done in the Traffic Layer course of the System Design and Distributed Systems curriculum, and the algorithm list is not repeated here. The difference is one sentence: **there the rules' design criteria were compared, here the tail a single rule leaves on forty subjects with unequal capacity is counted.** A rule is a configuration line, and all four are written in the same place: ```text # taught rule syntax, not run pool main { rule round-robin } pool main { rule proportional weight subject-01 22 weight subject-09 99 } pool main { rule least-connections metric open-connections / weight } pool main { rule consistent-hashing key session-id virtual 128 } ``` #### Two Loads, One Set The forty subjects' total capacity is **2336** units. The measurement runs on both sides of this number. **Load 1200** is a bit more than half the total capacity. Here the system has plenty of room, and overrun is entirely distribution's fault: if split correctly, no subject would overrun. **Load 2600** is above total capacity. Here overrun is unavoidable, and no rule can zero it out. The only question that can be asked is **who** the overrun falls on and **how much.** In between, **load 2000** is also measured; it is below capacity, but congestion has begun. The measurement's assumptions: - **TM7** — The forty subjects are the course's fixed set and are not changed in this lesson; capacities are between **22** and **99**, and the oracle is the set itself. - **TM8** — The load consists of divisible units; one unit consumes one unit of capacity on a subject. The load's class or delay does not enter distribution in this lesson. - **TM9** — All four rules run on **the same load and the same set.** The only difference between the rules is the field they read. - **TM10** — Proportional distribution's shares **round down** to whole numbers; the remaining units are not redistributed. This is part of the rule's own definition and is counted as a separate column in the measurement. - **TM11** — The least-connections rule picks the lower-numbered subject on a tie; the selection is deterministic and gives the same result when the run is repeated. - **TM12** — Consistent hashing represents every subject with **128** virtual points on the ring and uses the job's sequence number as the key. The hash function is deterministic. - **TM13** — The measurement is not a network measurement; no connection is opened, no request is sent. What is counted is the distribution in the fiction itself. #### The Measurement ```python """Four distribution rules on the same forty subjects: the tail moves, it does not vanish.""" import bisect SEED = 20260812 CLASSES = ("interactive", "batch", "standby") RING = 1 << 20 def generator(seed): d = seed % 2147483646 + 1 def r(n): nonlocal d d = (d * 48271) % 2147483647 return d % n return r def subjects(count=40, seed=SEED): r, result = generator(seed), [] for i in range(count): result.append({"no": i + 1, "capacity": 20 + r(81), "delay": 5 + r(45), "class": CLASSES[r(3)], "special": r(9) == 0}) return result def ring_hash(x): x = (x * 2654435761) & 0xFFFFFFFF x ^= x >> 16 x = (x * 2246822507) & 0xFFFFFFFF x ^= x >> 13 return x % RING def distribute(load, subs, rule, virtual=128): """A single rule distributes load across forty subjects; also returns key -> subject.""" share, mapping = {s["no"]: 0 for s in subs}, {} if rule == "round-robin": for i in range(load): mapping[i + 1] = subs[i % len(subs)]["no"] share[mapping[i + 1]] += 1 elif rule == "proportional": total = sum(s["capacity"] for s in subs) for s in subs: share[s["no"]] = load * s["capacity"] // total elif rule == "least-connections": for i in range(load): least = min(subs, key=lambda s: (share[s["no"]] / s["capacity"], s["no"])) mapping[i + 1] = least["no"] share[least["no"]] += 1 elif rule == "consistent-hashing": ring = sorted((ring_hash(s["no"] * 7919 + k), s["no"]) for s in subs for k in range(virtual)) points = [p for p, _ in ring] for a in range(1, load + 1): mapping[a] = ring[bisect.bisect_left(points, ring_hash(a)) % len(ring)][1] share[mapping[a]] += 1 return share, mapping def overrun(share, subs): n = units = 0 for s in subs: over = share[s["no"]] - s["capacity"] if over > 0: n, units = n + 1, units + over return n, units RULES = ("round-robin", "proportional", "least-connections", "consistent-hashing") subs = subjects() print(f"subjects {len(subs)} | total capacity {sum(s['capacity'] for s in subs)} | " f"smallest {min(s['capacity'] for s in subs)} largest {max(s['capacity'] for s in subs)}") print() print(f"{'rule':<19s} {'load':>5s} {'over':>10s} {'over units':>11s} " f"{'idle':>12s} {'undistributed':>13s} {'no load at all':>16s}") for rule in RULES: for load in (1200, 2000, 2600): share, _ = distribute(load, subs, rule) n, units = overrun(share, subs) print(f"{rule:<19s} {load:5d} {n:10d} {units:11d} " f"{sum(max(0, s['capacity'] - share[s['no']]) for s in subs):12d} " f"{load - sum(share.values()):13d} " f"{sum(1 for s in subs if share[s['no']] == 0):16d}") remaining = [s for s in subs if s["no"] != 20] print() print(f"{'rule':<19s} {'share of subject 20':>20s} {'moved when removed':>19s}") for rule in ("round-robin", "least-connections", "consistent-hashing"): share, before = distribute(1200, subs, rule) _, after = distribute(1200, remaining, rule) print(f"{rule:<19s} {share[20]:20d} " f"{sum(1 for k in before if before[k] != after[k]):19d}") ``` ``` subjects 40 | total capacity 2336 | smallest 22 largest 99 rule load over over units idle undistributed no load at all round-robin 1200 6 24 1160 0 0 round-robin 2000 14 239 575 0 0 round-robin 2600 23 533 269 0 0 proportional 1200 0 0 1156 20 0 proportional 2000 0 0 357 21 0 proportional 2600 40 245 0 19 0 least-connections 1200 0 0 1136 0 0 least-connections 2000 0 0 336 0 0 least-connections 2600 40 264 0 0 0 consistent-hashing 1200 7 66 1202 0 0 consistent-hashing 2000 13 277 613 0 0 consistent-hashing 2600 22 559 295 0 0 rule share of subject 20 moved when removed round-robin 30 1181 least-connections 15 1181 consistent-hashing 29 29 ``` #### A Good Rule Did Not Remove the Tail The load-1200 rows say improving the rule works. Round-robin distribution overruns **6** subjects; proportional distribution and least-connections overrun **none.** Overrun units drop from **24** to **0.** A reading that stopped here would conclude the correct rule removed the tail. The load-2600 rows reverse this. Round-robin distribution overruns **23** subjects; proportional distribution and least-connections overrun **all forty of the forty.** The two rules said to be good took the overrunning-subject count from zero to forty. The overrun-units column reads in exactly the opposite direction: round-robin **533**, proportional **245**, least-connections **264**. That is, the round-robin rule overruns a small number of subjects a lot; the others overrun all subjects a little. **The tail did not lift; the pain spread to everyone.** Which of these two readings is correct is not the measurement's call. The choice between twenty-three subjects' heavy overrun and forty subjects' light overrun is a **policy** decision: should forty subjects slow down a little, or should seventeen of them go untouched while twenty-three take a heavy hit? The measurement does not answer this question, it only puts the two options' numbers side by side. Where load is above total capacity, there is also a lower bound. Total capacity is **2336**, load is **2600**; the difference is **264** units, and no rule that places all of the load can go below this. The least-connections rule's overrun units are exactly **264** — meaning this rule has found the best possible distribution of what can be placed. There is nothing better. #### What the Overrun Column Does Not See So what is proportional distribution's **245**? A number below the lower bound can only come up when not all of the load gets placed. The **undistributed** column to the right says this. The proportional rule rounds each subject's share down to a whole number and gives the remaining units to no one: **19** units at load 2600, **21** at 2000, **20** at 1200 never reach any subject. These units do not show up in the overrun column, nor in the idle column — because they were never distributed at all. The conclusion is this: what makes proportional distribution's overrun column look its best is **the work the rule does not do.** If this column were absent while the measurement put the two rules side by side, the comparison **245 < 264** would lead to a wrong conclusion. **Undistributed units are a tail too, and they never show up in the lever's overrun gauge.** The same column has another reading. At load 1200, while round-robin distribution overruns **6** subjects, **1160** units sit idle in the system — half of total capacity. Overrun and idleness exist at the same time. **A gauge that counts only overrun cannot see that half of capacity went unused**, and a gauge that counts only utilization cannot see that six subjects overran. The two columns have to be read separately. #### What Least Connections Cannot See The least-connections rule gave the measurement's best result, but that result depends on two conditions, and both sit outside the measurement. The first is cost. Round-robin increments a counter, consistent hashing computes a hash; least connections compares all forty subjects' instantaneous state for every decision and repeats this for every incoming job. The decision itself turns into work. The second, and heavier, is this: **least loaded within which set?** The rule in the measurement sees all forty subjects and every unit distributed up to that instant. In operations, the balancer is usually not a single one; more than one balancer stands in front of the same pool, and each can count only the connections **it itself** opened. Each balancer picks the least-loaded subject by its own view, all of them pick the same subject at the same time, and that subject overruns without showing up in anyone's gauge. This is another form of the previous lesson's flat gauge. There the lever measured the **wrong dimension**; here it measures the right dimension over an **incomplete set.** The table's best result of **264** units rests on the assumption that the rule sees all forty subjects — `TM9`. When the assumption falls, so does the result, and there is no column in the table that says it fell. #### Where Consistent Hashing Pays The consistent-hashing rows are not good for distribution: at load 1200, **7** subjects overrun, one more than round-robin; at 2600, overrun units are **559**, above round-robin's **533**. This is an expected result, because this rule also does not read capacity, and on top of that it adds the hash distribution's own variability. Where the rule pays off is in the lower table. When a single subject is removed from the pool, **1181** keys change place under round-robin, and again **1181** under least-connections — nearly all of twelve hundred. Under consistent hashing, the number of keys that change place is **29**, and that is exactly the removed subject's own share. The remaining thirty-nine subjects' keys stay in place. Why does the difference matter? Because a key moving does not just mean one request going to a different subject. Whatever had been accumulated on that subject depending on the key — session state, a warmed cache — is not on the new subject. Under round-robin, one subject leaving the pool invalidates this accumulation on all forty of the forty subjects. **Consistent hashing does not improve distribution; it bounds the tail's length when membership changes.** Removing a subject from the pool is also a lever move — and one of the most frequently pulled ones in this course, because maintenance, failure, and version changes all pass through here. The rule's choice determines up front how many of the forty subjects that move will disturb. #### Summary - At load 1200, round-robin distribution overruns **6** subjects by **24** units, proportional distribution and least-connections overrun **none**; improving the rule genuinely pays off here. - At load 2600, the same two rules overrun **all forty of the forty** subjects, round-robin only **23**; overrun units are **533**, **245**, and **264** respectively. The tail did not lift, the pain spread to everyone, and which is correct is a policy decision. - When total capacity is **2336** and load is **2600**, overrun's lower bound is **264** units; the least-connections rule finds exactly this number, and there is nothing better. - Proportional distribution's **245** is below the lower bound because the rule rounds shares down and never distributes **19** units; undistributed units show up in neither the overrun column nor the idle column. - At load 1200, while six subjects overrun, **1160** units sit idle in the system; overrun and idleness exist at the same time, and no single gauge says both. - Consistent hashing is not better than round-robin for distribution, but when a subject is removed, the keys that move are **29** instead of **1181** — where the rule pays off is not distribution, it is membership change. #### Next Step In these two lessons, the balancer stood in the same place throughout: in front of the forty subjects, deciding on their behalf. Whoever sent the load, the lever was on the subjects' side, and its purpose was to protect them. The same intermediary can be turned around: it can stand in front of the clients and decide **on their behalf.** The next lesson puts these two stances — reverse proxy and forward proxy — side by side, and counts how a single number, the same limit, produces two separate tails depending on which side it stands on. ### Reverse Proxy and Forward Proxy Source: https://academia.sh/en/courses/network-operations/reverse-proxy-and-forward-proxy The same limit number produces two separate tails on two proxies; at limit 30 the reverse proxy leaves 6 subjects unprotected and 1160 units idle, the forward proxy counts that same 1160 units as throttled legitimate demand, and because it never sees 9 of the forty subjects and 430 units, its own report shows only 993 units. In the previous two lessons, the balancer stood in the same place throughout: in front of the forty subjects, deciding on their behalf. Whoever sent the load, the lever was on the subjects' side, and its purpose was to protect them. An intermediary is not condemned to this stance. The same mechanism can be turned around: it can stand in front of the clients and decide **on their behalf.** This lesson's question is not what the proxy does — it is **on whose behalf** it does it, and how that choice turns the same number into two separate tails. #### The Same Intermediary, Two Stances Both are **intermediaries**: they take a request, apply their own decision, pass it to the other side. The distinction is where they stand. The **reverse proxy** stands in front of the subjects. To an outside viewer, it is the only address; the forty subjects behind it are not directly visible. The reasons the server side puts it there can be listed: feeding the request into a distribution rule, terminating the encrypted connection here, caching the response, and closing off the topology behind it from the outside. The decision is made on the subjects' behalf. It is made without asking the subject whether it wants this — the subject is not, after all, the party that chose the intermediary in front of it. The **forward proxy** stands in front of the clients. Every outgoing request passes through it; the address the destination side sees is the proxy's, not the client's. The reasons the client side puts it there can also be listed: limiting which destinations can be reached, logging outgoing requests, caching a frequently requested response, and hiding the client from the destination. The decision is made on the clients' behalf. The relationship between the load balancer and the reverse proxy was covered in the System Design and Distributed Systems curriculum and the Server-Side Fundamentals course, and the reverse proxy's production configuration in the Server Security and Going to Production course; that account is not repeated here. The difference is one sentence: **there the proxy's role and configuration were measured, here the tail a single limit number leaves on forty subjects.** On the configuration side, the two resemble each other, and this is exactly why they get confused: ```text # taught configuration transcript, not run reverse-proxy { listen *:443 upstream subject-01 ... subject-40 limit concurrent 30 / subject purpose protect the subject } forward-proxy { listen internal-net:3128 egress every destination limit rate 30 / client purpose limit and hide the client } ``` #### The Lever Is a Single Number The lever is this: **a single limit number**, applied to forty subjects at once. On the reverse proxy this number is a **concurrency limit** — at most this much work passes to a subject at the same time. On the forward proxy it is a **rate limit** — a client can put out at most this much work at the same time. The question of why forty separate limits are not written is a fair one, and its answer is outside the measurement. Forty separate numbers could be written; the moment they are, forty separate maintenance burdens are born. When a subject's capacity changes, the line belonging to it also has to change; if it does not, the limit silently turns wrong and no gauge reports this. Operations gravitating toward a single number is not a shortcut, it is the direct result of maintenance cost — a single line sits in one place and watches forty subjects at once. The lever-to-subject ratio this course measures comes from exactly this: a single line's cost is a tail, and forty lines' cost is that, over time, forty lines stop genuinely describing forty subjects. The tail exists in both stances, but its name and its owner change. On the reverse proxy, a limit **larger than a subject's capacity** does not protect it: the subject overruns its own capacity before reaching the limit, and the limit never engages. A limit **smaller than a subject's capacity** keeps it below capacity; the difference is unused capacity. On the forward proxy, the same two conditions read in reverse. A client whose demand is **above** the limit gets throttled, and the difference is rejected legitimate demand. A client whose demand stays **below** the limit is not touched by the limit at all. There is one more difference, and it produces the measurement's sharpest column. **The reverse proxy is not bypassed**: it is the only path to the subjects, all forty of the forty pass in front of it. This is the measurement's assumption and an operating condition at the same time — if a subject's address stays directly reachable, the proxy can be cut out of the path, and every promise a reverse proxy makes rests on that address being closed. **The forward proxy can be bypassed**; a client using it is a configuration choice, and some subjects are set up to exit directly. The measurement's assumptions: - **TM14** — The forty subjects are the course's fixed set and are not changed. - **TM15** — In the reverse-proxy fiction, the subject is a server and the `capacity` field is the amount of work it can serve — the same reading as in the previous two lessons. - **TM16** — In the forward-proxy fiction, the subject is a client and the `capacity` field is read as its **legitimate demand.** **The set does not change, the reading does**; this is the condition for the two stances to be comparable on the same numbers. - **TM17** — The limit is a single number applied per subject; separate limits are not written for individual subjects. The lever's singleness rests in this assumption. - **TM18** — The subjects that bypass the forward proxy are the fiction's `special`-marked devices. Bypassing is not a flaw, it is a recorded configuration choice; the proxy neither limits nor sees their traffic. - **TM19** — The reverse proxy is the single entry point; no subject can bypass it. - **TM20** — The measurement is not a network measurement; no connection is opened. What is counted is the limit's arithmetic over forty subjects. #### The Measurement ```python """Same limit, two proxies: whose behalf the decision is on changes the tail.""" SEED = 20260812 CLASSES = ("interactive", "batch", "standby") def generator(seed): d = seed % 2147483646 + 1 def r(n): nonlocal d d = (d * 48271) % 2147483647 return d % n return r def subjects(count=40, seed=SEED): r, result = generator(seed), [] for i in range(count): result.append({"no": i + 1, "capacity": 20 + r(81), "delay": 5 + r(45), "class": CLASSES[r(3)], "special": r(9) == 0}) return result def reverse(subs, limit): """Subject is a server; the limit is set on its behalf. Tail: unprotected and idle.""" unprotected = sum(1 for s in subs if s["capacity"] < limit) idle = sum(max(0, s["capacity"] - limit) for s in subs) return unprotected, idle def forward(subs, limit, passing): """Subject is a client; the limit is set on its behalf. Tail: throttled and never-seen.""" g = [s for s in subs if s["no"] in passing] throttled = sum(1 for s in g if s["capacity"] > limit) units = sum(max(0, s["capacity"] - limit) for s in g) return throttled, units subs = subjects() passing = {s["no"] for s in subs if not s["special"]} allsub = {s["no"] for s in subs} print(f"subjects {len(subs)} | total demand {sum(s['capacity'] for s in subs)} | " f"passing forward proxy {len(passing)} | bypassing proxy {len(subs) - len(passing)} " f"({sum(s['capacity'] for s in subs if s['special'])} units)") print() print(f"{'limit':>6s} | {'reverse proxy':^22s} | {'forward proxy (31 pass)':^24s} | " f"{'if all forty passed':^27s}") print(f"{'':>6s} | {'unprotect.':>10s} {'idle':>11s} | " f"{'throttled':>8s} {'throttled units':>15s} | {'throttled':>8s} {'units':>11s} " f"{'equal':>5s}") for limit in (30, 45, 60, 75, 90): k, b = reverse(subs, limit) ki, bi = forward(subs, limit, passing) kt, bt = forward(subs, limit, allsub) print(f"{limit:6d} | {k:10d} {b:11d} | {ki:8d} {bi:15d} | {kt:8d} {bt:11d} " f"{sum(1 for s in subs if s['capacity'] == limit):5d}") print() print(f"subjects not passing the reverse proxy: {len(subs) - len(allsub)} " f"(it is the single entry point, all forty pass) | " f"subjects the forward proxy never sees: {len(subs) - len(passing)}, " f"{sum(s['capacity'] for s in subs if s['special'])} units") ``` ``` subjects 40 | total demand 2336 | passing forward proxy 31 | bypassing proxy 9 (430 units) limit | reverse proxy | forward proxy (31 pass) | if all forty passed | unprotect. idle | throttled throttled units | throttled units equal 30 | 6 1160 | 27 993 | 33 1160 1 45 | 12 711 | 23 620 | 28 711 0 60 | 23 354 | 15 326 | 17 354 0 75 | 29 128 | 10 125 | 11 128 0 90 | 34 17 | 4 17 | 4 17 2 subjects not passing the reverse proxy: 0 (it is the single entry point, all forty pass) | subjects the forward proxy never sees: 9, 430 units ``` #### The Same Arithmetic, Two Names The table's most important feature is that two of its columns are **identical, digit for digit.** The reverse proxy's "idle" column reads **1160, 711, 354, 128, 17**; the rightmost "if all forty passed" units column also reads **1160, 711, 354, 128, 17**. The same numbers, because both are the same sum: the positive part of the difference between each subject's capacity and the limit. The only thing that changes is that sum's **name.** On the reverse proxy, this number is capacity left unused for the sake of protection: the subject could have served more, the limit did not allow it. On the forward proxy, the exact same number is rejected legitimate demand: the client would have asked for more, the limit did not allow it. **The measure did not change; whose behalf it was set on did.** This has a direct consequence. Raising the limit **reduces protection** on the reverse proxy — unprotected subjects go from **6** at 30 to **34** at 90. The same move **reduces throttling** on the forward proxy — throttled subjects drop from **33** at 30 to **4** at 90. Turning the lever the same direction carries an opposite meaning in the two stances, and in a single dashboard the two show up as the same number. In practice, this means the limit's correct value cannot be read off from the measurement. The answer to "thirty or ninety" depends on which column in the table is deemed acceptable — and what makes that decision is not the lever itself, it is whose behalf you pull the lever on. This is also why the two sides cannot agree on the same number: what one calls protection, the other calls rejected demand. #### The Complementary Tail The two tails are each other's complement, and their totals can be verified in the table. At limit 30, there are **6** unprotected, **33** that would have been throttled if all forty passed, and **1** subject exactly equal to the limit: six plus thirty-three plus one makes forty. At limit 90, **34 + 4 + 2 = 40.** None of the three rows in between has a tie, and the two columns fill out to forty directly: **12 + 28**, **23 + 17**, **29 + 11**. What this means is: every one of the forty subjects, **at every limit value**, is in exactly one of the two tails. A subject staying below the limit is unprotected on the reverse proxy; one staying above it is throttled on the forward proxy. There is no limit value at which a subject escapes both, and there is no number that rescues all forty at once. Subjects exactly equal to the limit show up in neither column — one or two subjects in a set of forty, that is, one or two times the smallest measurable difference (**1/40 = 0.025**). The only thing a single-number lever can do is choose **where** to split the forty subjects between these two tails. Not splitting is not an option the lever has. #### What the Forward Proxy Cannot See The last line gives the second difference. Because it is the only path to the subjects, the reverse proxy sees all forty at once; the number bypassing it is **0.** The forward proxy, on the other hand, carries the traffic of only **31** of the forty subjects: **9** subjects are set up to exit directly, and their **430** units of demand never pass through the proxy at all. The report's counterpart to this is the middle columns. At limit 30, the forward proxy's own report says **993** units were throttled. If all forty subjects passed through the proxy, this number would be **1160.** The **167**-unit gap in between is not missing — it was **never counted.** It shows up in the proxy's report neither as throttled nor as free, because that traffic was never touched at all. The same gap exists in the throttled-subject column too: the report says **27**, the real number is **33.** This is the structural weakness of a lever that decides on the client's behalf — **for it to be able to enforce its decision, the client has to agree to pass through it.** A lever deciding on the server's behalf does not need this consent, because it is already sitting on the path. It should also be added that the nine subjects bypassing is not a flaw; it is a recorded choice, and it has a reason. What is a flaw is that **the report never mentions these nine at all**: a coverage report that does not say what falls outside its coverage shows the wrong total even if it counts what it does cover correctly. #### The Distinction the Proxy Erases The counterpart to address hiding on the other side is also a lever problem, and it is often overlooked in operations. The **31** subjects passing through the forward proxy show up as **a single address** in the message the destination side sees. The destination side applies its own lever — a rate limit, a concurrency limit, an access rule — to that single address. Thirty-one of the forty subjects have landed on one subject in the other side's ledger. A destination wanting to throttle its heaviest client throttles all thirty-one at once; wanting to block one of them, it blocks all thirty-one at once. The ratio this course measures — one lever, how many subjects — breaks a **second time** behind the proxy, and the other side does not know how many subjects it is touching. On the reverse proxy, the same erasure works in the opposite direction. To an outside viewer, the forty subjects behind it are a single address; which subject answered does not appear in the message. Closing off the topology is already the desired outcome, but its side effect hits the measurement directly: which subject a response came from cannot be extracted from the message. If this information is not carried separately, the forty subjects' tail collapses behind a single average, and none of the tables in the previous two lessons could be built. The two proxies' caching ability also carries the same distinction. The reverse proxy caches on the subjects' behalf: a response produced once is given to many clients. The forward proxy caches on the clients' behalf: a response received once is given to many clients' requests to the same destination. In both cases, the cached copy can go to a request it was **not produced for.** The per-person feature flag governing this decision was measured in the Application Layer Protocols course and is not repeated here; what matters in this lesson is that the caching decision, too, is a single lever applied to all forty subjects at once. #### Summary - The reverse proxy decides on the subjects' behalf, the forward proxy on the clients'; both are intermediaries and their configurations look alike, the distinction is where they stand. - A single limit number produces **unprotected subjects** and **idle capacity** on the reverse proxy, **throttled subjects** and **rejected legitimate demand** on the forward proxy. - Idle capacity and rejected demand are **the same sum**: both columns read **1160, 711, 354, 128, 17**; only the number's name changes. - The two tails are complementary: at every limit value, unprotected, throttled, and exactly-at-limit subjects add up to forty (**6 + 33 + 1**, **34 + 4 + 2**); no number rescues all forty at once. - In the measurement the reverse proxy is not bypassed and sees all forty; **9** subjects bypass the forward proxy, **430** units never pass through it, and its report shows **993** instead of **1160**, **27** instead of **33**. - The proxy also erases the address distinction: the **31** subjects behind the forward proxy land on a single address at the destination side, and the destination's lever touches all thirty-one at once; behind the reverse proxy, which subject a response came from does not survive in the message. #### Next Step In these three lessons, the lever stood at a single point throughout: where the request was forced to pass. The reverse proxy in front of the subjects, the forward proxy in front of the clients, but both at **a single** point. The next lesson multiplies this point: if content gets distributed across many copies standing close to the client, the lever no longer adjusts distribution but **what stands where.** The measure changes too — the question is not who gets throttled, but **which subject the edge never sees**: content waiting for its first request under the pull model, content never requested at all under the push model. ### Content Delivery Networks Source: https://academia.sh/en/courses/network-operations/content-delivery-networks Under the pull model, the edge carries only what has been requested and its cost is 33 misses; under the push model, misses are zero but 402 units of never-requested content sit at the edge; once an 800-unit limit is placed on the edge, pull gives 245 misses, push that does not know popularity gives 379, and in both models the object the edge never sees at all climbs to 25. In the previous three lessons, the lever stood at a single point throughout: where the request was forced to pass. The reverse proxy in front of the subjects, the forward proxy in front of the clients, but both at **a single** point, and one with access to all of the content. A **content delivery network** multiplies this point: copies of the same content are kept in many **edge caches** standing close to the clients, and the request does not travel all the way to the source server. The lever changes too. What gets adjusted is no longer how load gets split, but **what stands where** — and this lever's tail is the content the edge never sees. #### What the Edge Measures The edge cache's operating models, which layers can be cached, and distribution's design decisions were covered in the Traffic Layer course of the System Design and Distributed Systems curriculum; that discussion is not repeated here. The difference is one sentence: **there the distribution model was compared as a design, here the tail a single placement decision leaves on forty objects is counted.** In this lesson's fiction, the forty subjects are **content objects.** The set is the same set; the only thing that changes is how the `capacity` field is read — here that field is the object's **size.** The forty objects' total size is the same number as the total capacity in the previous lessons. The edge's defining property is that it is **smaller than the source server.** If it were not smaller, it would not be an edge; it would be a second source server carrying all the content. This is why every edge cache has to make a choice: which of the forty objects will sit there? The lever is this choice's rule, and it has two basic forms. #### Pull and Push Under the **pull** model, the edge starts empty. When an object is requested for the first time, it is not at the edge; the request goes to the source server, and as the response comes back it gets cached at the edge, and subsequent requests are served from there. The edge never carries an object that has not been requested — but every object's **first request** inevitably misses. Under the **push** model, content is placed at the edge without waiting to be requested. The first request too is served from the edge, and misses can be zeroed out. In return, the edge also carries content that will never be requested, and the space set aside for it cannot be given to anything else. On the configuration side, both are a few lines: ```text # taught configuration transcript, not run edge { model pull source source-server cache on-first-request size 800 units eviction least-recently-used } edge { model push source source-server place at-publish-time size 800 units selection object-order } ``` The last line carries the push model's real question: if the edge cannot take all the content, **which** does it take? The pull model does not ask this question, because it reads the answer from the request stream. The push model has to ask it, and has to give the answer **in advance.** The measurement's assumptions: - **TM21** — The forty subjects are the course's fixed set. In this lesson, each subject is a content object, and the `capacity` field is read as its **size**; the set does not change, the reading does. - **TM22** — The request stream consists of six hundred requests, and its popularity is skewed: a small number of objects take most of the requests, and objects in the tail are never requested. - **TM23** — **The popularity order is independent of the object number.** The order is shuffled with a separate seed, so a push rule that selects by object number carries no information about popularity. - **TM24** — There is no miss for an object sitting at the edge; a request for an object that is not there goes to the source server and counts as a miss. Objects' freshness lifetime is infinite in this measurement; a cached copy never goes stale. - **TM25** — On a limited edge, the eviction rule is **least recently used**: when room needs to be made, the object that has gone longest unrequested is dropped. - **TM26** — Limited push selects objects **by number order**; this is the plainest form of a placement that does not know popularity. - **TM27** — The measurement is not a network measurement; no request is sent, no edge is set up. What is counted is the placement in the fiction itself. #### The Measurement ```python """Edge cache: pull waits on the first request, push carries what is never requested.""" SEED = 20260812 CLASSES = ("interactive", "batch", "standby") EDGE = 800 def generator(seed): d = seed % 2147483646 + 1 def r(n): nonlocal d d = (d * 48271) % 2147483647 return d % n return r def subjects(count=40, seed=SEED): r, result = generator(seed), [] for i in range(count): result.append({"no": i + 1, "capacity": 20 + r(81), "delay": 5 + r(45), "class": CLASSES[r(3)], "special": r(9) == 0}) return result def popularity(subs, seed=SEED + 17): """Popularity order is independent of object number.""" r, order = generator(seed), [s["no"] for s in subs] for i in range(len(order) - 1, 0, -1): j = r(i + 1) order[i], order[j] = order[j], order[i] return order def requests(subs, count=600, seed=SEED + 23): """Popularity-skewed request stream: objects in the tail go never requested.""" r, order, result = generator(seed), popularity(subs), [] for _ in range(count): result.append(order[min(r(len(subs)), r(len(subs)), r(len(subs)))]) return result def pull(subs, reqs, limit): """Object enters the edge on first request; if limited, the oldest use gets evicted.""" size = {s["no"]: s["capacity"] for s in subs} edge, used, misses = [], 0, 0 for n in reqs: if n in edge: edge.remove(n) edge.append(n) continue misses += 1 if limit is not None: while used + size[n] > limit and edge: used -= size[edge.pop(0)] if used + size[n] > limit: continue edge.append(n) used += size[n] return set(edge), used, misses def push(subs, reqs, limit): """Forty objects are pushed ahead of time; if limited, selected by number order.""" size, edge, used = {s["no"]: s["capacity"] for s in subs}, set(), 0 for s in subs: if limit is None or used + s["capacity"] <= limit: edge.add(s["no"]) used += s["capacity"] return edge, used, sum(1 for n in reqs if n not in edge) subs = subjects() reqs = requests(subs) size = {s["no"]: s["capacity"] for s in subs} requested = set(reqs) print(f"objects {len(subs)} | total size {sum(size.values())} | edge limit {EDGE} units") print(f"requests {len(reqs)} | requested objects {len(requested)} | " f"never requested {len(subs) - len(requested)} " f"({sum(size[s['no']] for s in subs if s['no'] not in requested)} units)") print() print(f"{'model':<20s} {'at edge':>10s} {'used':>11s} {'misses':>6s} " f"{'hits':>7s} {'edge never sees':>18s} {'wasted idle':>11s}") for name, edge, used, misses in ( ("pull, unlimited", *pull(subs, reqs, None)), ("pull, limited", *pull(subs, reqs, EDGE)), ("push, unlimited", *push(subs, reqs, None)), ("push, limited", *push(subs, reqs, EDGE))): print(f"{name:<20s} {len(edge):10d} {used:11d} {misses:6d} {len(reqs) - misses:7d} " f"{len(subs) - len(edge):18d} " f"{sum(size[n] for n in edge if n not in requested):11d}") ``` ``` objects 40 | total size 2336 | edge limit 800 units requests 600 | requested objects 33 | never requested 7 (402 units) model at edge used misses hits edge never sees wasted idle pull, unlimited 33 1934 33 567 7 0 pull, limited 15 788 245 355 25 0 push, unlimited 40 2336 0 600 0 402 push, limited 15 798 379 221 25 30 ``` #### The Cost of the First Request The pull model's unlimited row shows the model's pure form. **33** of the six hundred requests miss, and this number is no coincidence: the requested-object count is also **33.** Under the pull model, every object misses **exactly once** — on first request. The remaining **567** requests are served from the edge. This model's tail is therefore a very specific tail: **each object's first requester.** That request travels all the way to the source server, sees the longest delay, and clears the path for the next requester. Being in the tail here is not a flaw, it is how the model works; but who is in the tail matters. This tail is not distributed equally. On a popular object, a single miss gets lost among hundreds of hits, and almost none of the people requesting that object pay the cost. On a rarely requested object, on the other hand, **every** requester is, in practice, the first requester: if the time between two requests is longer than the cache's lifetime, the copy gets fetched fresh every time. So the party that actually pays the pull model's cost is the minority that already receives the least service — and a gauge that looks at the overall hit rate never shows this minority, because **33** misses against **567** hits is a good ratio. The column to the right gives the second number: the object the edge never sees is **7.** Seven of the forty objects were not requested even once across six hundred requests, and the pull model never brought them to the edge at all. By the model's own measure, this is flawless behavior — wasted idle units are **0.** Whatever is at the edge has been requested. #### The Cost of Never Being Requested The push model's unlimited row zeroes out the cost the pull model pays: misses are **0**, all six hundred of the six hundred requests are served from the edge. There is no tail left called the first requester. The cost sits in the column to the right. The edge carries all **2336** units, but **402** of them belong to seven objects that are never requested. Roughly a sixth of the edge's space is filled with content that goes unread even once across six hundred requests. The two models' tails are therefore each other's opposite. The pull model's tail is **in time**: the first request until an object enters the edge. The push model's tail is **in space**: content sitting at the edge that is never read. A model that zeroes out one grows the other, and there is no model that zeroes out both — because there is no way to know in advance what will be requested. #### When a Limit Is Placed The two rows above assumed the edge was unlimited, and that assumption goes against the edge's own definition. The rows below repeat the same measurement with a realistic **800**- unit limit. Under the pull model, misses rise from **33** to **245.** The cause is eviction: when the edge fills up, the least-recently-used object gets dropped, and when the evicted object is requested again, it misses again. The rule "every object misses once" no longer holds; an object misses as many times as it enters and leaves the edge. The object the edge never sees also rises from **7** to **25** — because there are now objects that never find room at the edge. Under the push model, the same limit charges a heavier price: misses are **379.** More than half of the six hundred requests go to the source server. The cause is clear, and `TM23` and `TM26` were put in place exactly to measure this: the push rule selects objects by number order, and number order has nothing to do with popularity. The edge fills to **798** units and carries **15** of the forty objects — almost the same occupancy as pull — but carries **the wrong fifteen objects.** On top of that, limited push still carries **30** units of never-requested content. Even when space is scarce, the lever spends part of that scarce space on something no one wants. The real lesson here concerns the placement rule itself. **The push model requires a prediction, and the prediction's wrongness is invisible unless it is measured.** The pull model requires no prediction; in return, it pays a miss for every new object, and this cost is always visible. In a set of forty objects, the smallest measurable difference is **1/40 = 0.025**; the gap between **245** and **379** is far above this, and the two models' ranking can be defended with this set. #### The Edge Is Not Singular The measurement modeled a single edge cache, and this goes against the content delivery network's own definition: a network is a network because it multiplies the edge. Multiplication's effect on both models is direct. Under the pull model, the first request's cost is paid **per edge.** Every edge misses its own first request, because the copy one edge caches is not on another edge. The **33** misses in the measurement are a single edge's cold start; as the edge count grows, this cost is multiplied along with it. If two clients requesting the same object land on separate edges, both are the first requester. This is the pull model's most commonly overlooked cost: adding an edge lowers the delay the client sees but **raises** the number of misses falling on the source server. Increasing the edge count does not remove the tail, it multiplies it — the same pattern as in previous lessons. Under the push model, the multiplication shows up in space. The **402** never-requested units sit separately on every edge; pushing forty objects to a large number of edges means carrying content no one wants that many times over. The intermediate form that builds a balance here is edges pulling from each other, or from a shared upper tier, rather than from the source server. The first requester still misses, but the miss goes to a nearer copy, not to the source server. The lever is still single: which edge pulls from whom is written by a single rule and applied to all forty of the forty objects at once. #### The Cached Copy Going Stale `TM24` assumed in the measurement that the cached copy does not go stale, and it has to be said that this assumption is not real. If the content changes, the copy at the edge turns wrong, and the edge does not know this on its own. What closes this gap is a third lever: **invalidation.** When content changes, its copies at the edges have to be dropped, and this too is a single decision — which objects get dropped at which edges is written by a single rule. Its tail has two ends, and by now it is familiar. If the rule is written narrow, the changed content's stale copy stays at the edge and **every** requester gets the wrong answer; worse, such a copy gives no sign of it unless someone actually looks. If the rule is written broad — say, the entire edge is dropped on every publish — objects that never changed also leave the edge, and the pull model's first-request cost gets paid all over again. The **33** misses in the measurement, under a broad invalidation rule, are reborn on every publish; the limited edge's **245** grows even further as eviction and invalidation stack on top of each other. Freshness lifetime, validator fields, and the cache directives themselves were measured in the Server-Side Fundamentals and Application Layer Protocols courses; they are not repeated here. What matters on this lesson's axis is that invalidation, too, takes the form of **one decision, many subjects**, and its tail's two ends do not close together. #### Summary - The content delivery network lever adjusts not load but **placement**: which of the forty objects sit at the edge. The tail is the content the edge never sees. - Under the pull model, every object misses exactly once: **33** requested objects, **33** misses, **567** hits, wasted idle units **0**; the edge carries only what has been requested. - Under the push model, misses are **0**, but **402** never-requested units sit at the edge; the two models' tails are each other's opposite — one in time, the other in space. - An **800**-unit limit raises pull's misses from **33** to **245**; an evicted object misses again when requested again, and the object the edge never sees rises from **7** to **25.** - At the same limit, push that does not know popularity gives **379** misses: filled to **798** units, the same occupancy as pull, but with the wrong **15** objects, plus **30** units of never-requested content. - Push requires a prediction, and the prediction's wrongness is invisible unless measured; pull requires no prediction and always pays its cost visibly. #### Next Step The four levers up to this lesson all asked a single question in different forms: **where** should the work go? The balancer to which subject, the proxy to whose ownership, the edge to which copy. None of them accounted for work potentially being of **different importance**; the forty subjects all waited in the same line. The next lesson lifts this assumption: traffic gets split into classes, one class gets moved ahead, the other gets made to wait. The measure directly asks this too — the delay a class gains is taken **from whose tail**, and the two numbers are always written together. ### Quality of Service Source: https://academia.sh/en/courses/network-operations/quality-of-service Three scheduling rules are run on the same forty subjects, and the 433 units that cannot be served stay the same across all three; strict priority brings the interactive class's wait from 1.13 rounds to 0.00 while sending the standby class from 1.28 to 5.76 rounds and into a 433-unit tail, and the overall average drops from 1.19 to 0.40, appearing to improve. The four levers so far all asked the same question in different forms: **where** should the work go? The balancer to which subject, the proxy on whose behalf, the edge to which copy. None of them accounted for work being able to have **different importance**; the forty subjects were all waiting in the same line. **Quality of service** lifts this assumption. Traffic gets split into classes, one class gets moved ahead, the other gets made to wait. The lever now adjusts not the work's location but its **order.** And on a link where capacity is not enough, order is a zero-sum thing: the delay a class gains is taken from another class's tail. This lesson's rule is that these two numbers are **always written together.** #### Three Steps Quality of service is not a single mechanism but three steps that follow each other. **Classification** gives every incoming unit a class label. The label can be read from a field in the message, a port, the arriving interface, or a mark placed beforehand in advance. This step makes no decision; it only says who is in which set. **Prioritization** builds an order among the classes. It determines which class's tail gets pulled from when the link frees up. If capacity meets demand, this step has no effect at all; its effect shows up only under congestion. **Shaping** caps from above the share a class can take. Its difference from prioritization is this: priority says **when** a class gets served, shaping says **at most how much** it can take. Used together, a class can be both prioritized and capped — it moves ahead but cannot swallow the whole link. None of the three steps creates capacity, and this is quality of service's most commonly misunderstood side. If the link meets demand, all three steps do nothing at all. The rules' effect shows up only at the **bottleneck**; this is why a quality-of-service rule is written not at every point traffic passes through, but at the point where capacity falls short. A rule written somewhere else sits in the configuration, never shows up in the measurement, and gives a false sense of security. On the configuration side, all three sit in the same place: ```text # taught rule syntax, not run classify { interactive mark = urgent batch mark = batch standby remaining } schedule { rule strict-priority order interactive, batch, standby } shape { interactive at-most 45% batch at-most 40% standby at-most 15% leftover redistribute } ``` #### Classification's Own Tail Before moving to the measurement, one warning is needed. The fiction below **knows the classes correctly**: the forty subjects' class comes from the fiction itself, and it is the oracle. This is not how it is in operations. The classifier is a lever too, and it has its own tail. It gives the label by looking at what is written in the message; nothing verifies what is written — the observation left by the Application Layer Protocols course holds here too. When a unit is misclassified, the next two steps produce a wrong result **while working flawlessly**: the priority rule is applied correctly, shaping is applied correctly, but to the wrong subject. And this leaves no sign in the result itself. The tail's two ends are here. If classification is written narrow, traffic that is genuinely urgent falls into the remaining class and gets no benefit from priority at all. If it is written broad, the urgent class swells; prioritization's meaning depends on the set it sets aside staying small. A schedule where everyone is prioritized is the same as having no schedule. The measurement's assumptions: - **TM28** — The forty subjects are the course's fixed set; each subject's class comes from the fiction, and **classification is taken to be flawless.** What is measured is the schedule's result, not the label's accuracy. - **TM29** — Twenty rounds are run. In every round, every subject generates a variable demand; demand size depends on the subject's capacity. - **TM30** — The link carries **150** units per round. Total demand is above this capacity; **the number of units that cannot be served is fixed, independent of the rule.** - **TM31** — Wait is the difference between the round a unit arrives and the round it gets served, counted in rounds. A unit served in the same round it arrives has a wait of zero. - **TM32** — Under the classless rule there is a single tail, and units get served in arrival order. Under strict priority, the lower class is not touched until the higher class's tail is empty. - **TM33** — Under shaping, each class is given a per-round cap; capacity left over from the caps is distributed round-robin in a second pass, so the link never sits idle. - **TM34** — The measurement is not a network measurement; no packet is sent. What is counted is the schedule in the fiction itself. #### The Measurement ```python """Quality of service: one class's gained delay is another class's tail.""" SEED = 20260812 CLASSES = ("interactive", "batch", "standby") ROUNDS, LINK = 20, 150 SHARE = {"interactive": 45, "batch": 40, "standby": 15} def generator(seed): d = seed % 2147483646 + 1 def r(n): nonlocal d d = (d * 48271) % 2147483647 return d % n return r def subjects(count=40, seed=SEED): r, result = generator(seed), [] for i in range(count): result.append({"no": i + 1, "capacity": 20 + r(81), "delay": 5 + r(45), "class": CLASSES[r(3)], "special": r(9) == 0}) return result def demands(subs, seed=SEED + 31): """Every round, every subject generates a variable demand.""" r, stream = generator(seed), [] for _ in range(ROUNDS): rnd = [] for s in subs: for _ in range(1 + r(s["capacity"] // 8 + 1)): rnd.append(s) stream.append(rnd) return stream def run(subs, stream, rule): """A single scheduling rule pushes forty subjects' units through the link.""" tail = {c: [] for c in CLASSES} wait = {c: [] for c in CLASSES} served = {s["no"]: 0 for s in subs} common = [] def pull(items, cap): n = 0 while n < cap and items: arrival, s = items.pop(0) wait[s["class"]].append(t - arrival) served[s["no"]] += 1 n += 1 return n for t in range(ROUNDS): for s in stream[t]: (common if rule == "classless" else tail[s["class"]]).append((t, s)) remaining = LINK if rule == "classless": remaining -= pull(common, remaining) elif rule == "strict-priority": for c in CLASSES: remaining -= pull(tail[c], remaining) else: for c in CLASSES: remaining -= pull(tail[c], min(LINK * SHARE[c] // 100, remaining)) for c in CLASSES: remaining -= pull(tail[c], remaining) for arrival, s in common: tail[s["class"]].append((arrival, s)) return tail, wait, served subs = subjects() stream = demands(subs) print(f"rounds {ROUNDS} | link capacity {LINK}/round, total {ROUNDS * LINK} units | " f"total demand {sum(len(t) for t in stream)} | " f"unservable {sum(len(t) for t in stream) - ROUNDS * LINK}") for c in CLASSES: print(f" {c:12s} subjects {sum(1 for s in subs if s['class'] == c):2d} " f"demand {sum(1 for t in stream for s in t if s['class'] == c):5d}") print() print(f"{'rule':<16s} {'class':<12s} {'served':>7s} {'avg wait':>13s} " f"{'longest':>8s} {'tail':>9s} {'subjects in tail':>16s}") for rule in ("classless", "strict-priority", "shaping"): tail, wait, served = run(subs, stream, rule) for c in CLASSES: w = wait[c] print(f"{rule:<16s} {c:<12s} {len(w):7d} " f"{sum(w) / len(w) if w else 0:13.2f} {max(w) if w else 0:8d} " f"{len(tail[c]):9d} " f"{len({s['no'] for _, s in tail[c]}):16d}") total = [x for c in CLASSES for x in wait[c]] print(f"{'':<16s} {'-- overall':<12s} {len(total):7d} " f"{sum(total) / len(total):13.2f} {max(total):8d} " f"{sum(len(tail[c]) for c in CLASSES):9d} " f"{len({s['no'] for c in CLASSES for _, s in tail[c]}):16d}") ``` ``` rounds 20 | link capacity 150/round, total 3000 units | total demand 3433 | unservable 433 interactive subjects 11 demand 829 batch subjects 20 demand 1974 standby subjects 9 demand 630 rule class served avg wait longest tail subjects in tail classless interactive 714 1.13 3 115 11 classless batch 1736 1.19 3 238 20 classless standby 550 1.28 3 80 9 -- overall 3000 1.19 3 433 40 strict-priority interactive 829 0.00 0 0 0 strict-priority batch 1974 0.03 1 0 0 strict-priority standby 197 5.76 13 433 9 -- overall 3000 0.40 13 433 9 shaping interactive 829 0.00 0 0 0 shaping batch 1731 1.14 3 243 20 shaping standby 440 2.85 7 190 9 -- overall 3000 1.08 7 433 29 ``` #### Taken From Whom The first column of the table that should be read is not the rightmost one, but the three `-- overall` rows' **tail** value. All three are **433.** Total demand is **3433**, the link carries **3000** units over twenty rounds, and the gap between them closes with no rule. This is this topic's last form of the course's second claim. **A scheduling rule does not destroy the unservable unit; it chooses who carries it.** None of the three rules grew the link, none shrank the demand; they only distributed the same 433 units over different subjects. The distribution itself reads directly. Under the classless rule, the tail splits three ways: interactive **115**, batch **238**, standby **80.** Moving to strict priority zeroes out interactive's **115** and batch's **238**, and both pile onto the standby class: **80 + 115 + 238 = 433.** The standby class's tail climbs from **80** to **433**, and only **197** of that class's **630**-unit demand gets served. The same arithmetic shows up in the wait, in rounds. The interactive class's average wait drops from **1.13** rounds to **0.00** — not a single one of its units waits at all, and its longest wait is also **0.** The gain is exactly **1.13** rounds. In the same move, the standby class's average wait climbs from **1.28** to **5.76** rounds, its longest wait from **3** to **13.** **The 1.13 rounds interactive gains is 4.48 rounds standby pays.** Written separately, both numbers are correct; written alone, the rule looks either flawless or catastrophic. The shaping rows give the middle ground, and show that it is exactly that — a middle ground. Interactive again waits **0.00** — the upper class's gain is preserved. But standby's wait is **2.85** instead of **5.76**, its tail **190** instead of **433.** The difference is the share the batch class takes on: its tail climbs from **238** to **243**, and its wait drops from **1.19** to **1.14.** The same 433 units are split this time as **243 + 190.** What shaping does is spread the load strict priority dumps on a single class across two classes. It does not remove the tail; it adjusts **who the tail accumulates on** — the same result that has come out of every lever since the start of the course. #### Waiting Versus Dropping The shaping in the measurement **waited** the units above the cap: the unit stayed in the tail and got served when its turn came. This is only one of two options. The second is **dropping** the units above the cap — in this course this is called **policing.** The lever is the same lever, the cap is the same cap; the only thing that changes is what happens to the unit above the cap. The distinction's effect on the tail is direct. The waiting rule holds the unservable **433** units in the tail; these units get served, late as it is, and show up in the table as wait time. The dropping rule never shows them at all: the tail column reads **0**, wait times drop, the gauge looks clean. The lost units are in no column at all. On top of that, dropping may not eliminate the dropped unit. If an upper layer notices the loss and resends, the same job comes to the link a **second time** — this time later and with extra load. The retransmission behavior measured in the Transport Layer topic of the Network Models and Protocols course becomes an operations problem here: enforcing the cap by dropping can raise demand instead of lowering it. There is a limit in the opposite direction too. Waiting may always look better, but the tail itself is a cost. As the tail deepens, wait grows, and in a deep shared tail, an urgent unit sits behind a large number of batch units that arrived before it. The classless rule in the measurement is exactly this situation: the interactive class's **1.13**-round wait comes not from its own traffic, but **from other classes' units ahead of it.** This is also classification's reason for existing — without splitting the tail, the tail's depth charges everyone the same cost. #### What the Overall Average Does Not Say The last reading is in the `-- overall` rows, and it is a dashboard trap. Overall average wait is **1.19** rounds under the classless rule, **0.40** rounds under strict priority. A gauge that looks only at this number would say moving to strict priority improves overall delay by **more than threefold.** The number is not wrong: the average across the three thousand units served genuinely dropped, because most of the weight now sits in two classes that never wait at all. The two columns next to the same row say the opposite. Longest wait climbs from **3** rounds to **13**; the number of subjects carrying a tail drops from **40** to **9.** Forty subjects waiting a little turns into nine subjects waiting a lot, and this is exactly why the average drops. The **40 → 9** drop in the subjects-in-tail column can read, on its own, like an improvement — the sentence "the number of subjects waiting in the tail fell to a quarter" is true and completely misleading. In a set of forty subjects, the smallest measurable difference is **1/40 = 0.025**; the change here is thirty-one times that, and its direction is set by policy, not by the measure. The shaping row softens this trap too: overall average **1.08**, longest wait **7**, subjects carrying a tail **29.** None of the three numbers is the best, but all three together leave no single class alone. Which row gets chosen is not the measurement's decision — the measurement only says who pays. #### Summary - Quality of service is three steps: classification gives the label, prioritization builds the order, shaping sets the cap; if the first step is wrong, the next two produce a wrong result while working flawlessly. - Total demand is **3433**, link capacity **3000**; the unservable **433** units are the same across all three rules. A scheduling rule does not destroy the tail, it chooses who carries it. - Under strict priority, interactive's **115**-unit and batch's **238**-unit tails pile onto the standby class: **80 + 115 + 238 = 433**, and only **197** of standby's **630**-unit demand gets served. - The **1.13**-round wait interactive gains is **4.48** rounds standby pays; if the two numbers are not written together, the same rule looks either flawless or catastrophic. - Shaping splits the same **433** units into two classes as **243 + 190**; it preserves the upper class's **0.00** gain but holds the lower class's wait at **2.85** instead of **5.76.** - The overall average **drops** from **1.19** to **0.40** under strict priority — while at the same time the longest wait climbs from **3** to **13** and subjects carrying a tail fall from **40** to **9.** #### Next Step In these five lessons, the lever genuinely managed traffic: it distributed, it limited, it placed, it scheduled. In every lesson we could also count the tail — how many subjects overran, how many units sat idle, how many objects were never seen, who paid whose delay. But there was a single reason we could write down all of these numbers: we built the fiction ourselves, and the oracle was in our hands. We knew capacity, popularity, class, and real demand from the start. None of these are known in operations. As seen in the last lesson, the lever's own gauge can say it improved while the tail grew, and the gauge is not lying. So the next question is not about the lever: **how do we know what the lever is doing?** The next topic takes on this question, and starts from its most basic point — it defines which records, which counters, the network reports its own state with. ## Observability Flow record and counter collection, the distinction between poll-based monitoring and event notification, what capture filters see and fail to see, and which subject end-to-end health indicators never report. ### Network Telemetry Source: https://academia.sh/en/courses/network-operations/network-telemetry The sampling rate is a single lever applied to forty subjects at once: dropping from one-in-one to one-in-twenty takes records from 400 to 20, seen heavy events from 22 to 2, 20 heavy events go unrecorded, and 26 of the forty subjects never appear in the report. The previous lesson built Quality of Service. Classification, prioritization, and shaping were a single lever; the delay one class gained was another class's tail, and the two numbers were written together. This lesson asks where those two numbers come from. The lever was managing traffic — but **how do we know** what the lever is doing? The operator does not have the network itself in hand; what they have are the records the network produces about itself. **Telemetry** is the collection of these records. The lesson's question is not what telemetry says, but **what it does not say** — because the setting that governs collection is itself a lever, and it has a tail of its own. #### What Telemetry Collects The data a network device produces splits into two separate classes, and each answers a different question. A **flow record** is a summary. The device counts packets that share the same address–port–protocol five-tuple as a single logical flow and writes one line when the flow ends or a duration elapses: who talked to whom, how long it lasted, how many packets and bytes passed. A flow record **names the subject**. A **counter** is an accumulation. The device keeps values like incoming bytes, outgoing bytes, and dropped packets on each interface by incrementing them; the reader takes the difference between two readings. A counter does not name — it says how much traffic passed, not whose it was. The **gauge**, which carries an instantaneous value, and the **histogram**, which carries a distribution, belong to the same class. The distinction between metric types was established in the **Observability and Reliability** course, and time-series querying and aggregation in the **Observability and Operations** course; neither is repeated here. In this lesson the subject is not a service but a **network device**, and what is measured is not the result of aggregation but **the number of subjects that never entered aggregation at all.** The real tension is this: a counter is cheap and misses no event, but it names no event either. A flow record names, but it is expensive — for every flow, the device's processor, memory, and reporting link do work. Because it is expensive it gets **reduced**, and the name for that reduction is sampling. #### The Shape of a Flow Record The dump below shows the fields carried by a flow record and an interface counter. This is a format example; it is not a measurement, and it has not been executed. ```text # taught dump , not executed flow record observing device device-07 source station.example destination example.test protocol TCP packets / bytes 143 / 20416 start 00:00:12.400 duration 840 ms sampling rate 1-in-10 interface counter (cumulative) bytes in 918233100 bytes out 640112940 packets dropped 1204 ``` The `sampling rate` field in the record matters: the record itself reports that it is a sample. If the reader misses this field, they take the record for a full count. There is no such field on the counter side, and none is needed — a counter is not sampled. #### The Record's Journey There are two distances between a record's production and its reading, and both affect the measurement. The first is **time**. A flow record is written when the flow ends; a transfer that runs for minutes produces no line until the moment it finishes. Devices loosen this with an **active timeout**: once a set duration elapses, an intermediate record is written even if the flow is still running, and the flow continues under the same identity. If no intermediate record is produced, a long flow stays invisible exactly while it is causing trouble; if it is produced, the same flow is represented by multiple lines in the report, and a dashboard that counts flows mistakes it for several flows. The second is the **path**. The record travels from the device to the collector over the network — meaning telemetry uses the very network it measures. This has two consequences. When the network degrades, record loss happens at **the same moment** as the event that was meant to be measured; the minute that needs the most information delivers the fewest records. The second consequence is quieter: if transport is unreliable, the content of a dropped record does not come back. The loss itself need not stay hidden — flow record protocols put a sequence number on the record stream, and the collector derives from the gap how many records were dropped; what it cannot derive is **what** they were. The counter diverges again here: a counter is cumulative, and however many readings are missed between two reads, the difference stays correct. #### Sampling Is a Lever The sampling rate is a single number. The operator sets it once, and the setting applies to **forty subjects at once**. The lever's tail — the minority it serves wrongly — shows up here in two separate forms. The first is a **missed heavy event**: a problem that is never known because it was never recorded. The second is subtler: an **invisible subject**. If a device does not appear in even one line of the report, there is neither a high value nor a low value for it on the dashboard; there is no value at all, and the gap looks the same color as health. The measurement's assumptions: - **OB1** — The forty subjects are the course's fixed population and are not changed in this lesson; subjects differ from one another in capacity, latency, and class. - **OB2** — Four hundred events pass in the observation window. Every event belongs to a subject, and a portion of the events is marked **heavy**; the heaviness comes from the setup itself. - **OB3** — The oracle is the setup: we know which event is heavy and which subject it belongs to because we wrote them. The measurement does not use this information, it only compares the result against it. - **OB4** — Sampling is deterministic: an event whose sequence number is divisible by the `one-in-N` value is recorded. Random sampling would give a different record set, but it would not change the record count or the trend below. - **OB5** — Every flow record is a fixed **64 bytes**. The bytes column shows the relative size of the record volume; it is not a network measurement, it is a multiple of the record count. - **OB6** — The estimate is made with this rule: the number of observed heavy events is multiplied by the sampling rate. This is the usual path taken when generalizing from a sample to the total. #### Measurement ```python """Network telemetry: the sampling rate is a single lever, what is measured is the tail. Part 1 - forty subjects and four hundred events. Part 2 - five sampling rates: records, observed heavy events, missed, invisible subjects. """ SEED = 20260812 RECORD_BYTES = 64 def rng(seed): state = seed % 2147483646 + 1 def draw(n): nonlocal state state = (state * 48271) % 2147483647 return state % n return draw def subjects(count=40, seed=SEED): draw, items = rng(seed), [] for i in range(count): items.append({"id": i + 1, "capacity": 20 + draw(81), "latency": 5 + draw(45), "class": ("interactive", "batch", "standby")[draw(3)], "special": draw(9) == 0}) return items def events(subs, count=400, seed=SEED + 3): draw, items = rng(seed), [] for i in range(count): s = subs[draw(len(subs))] items.append({"id": i + 1, "subject": s["id"], "heavy": draw(25) == 0}) return items def sample(evts, one_in): """One in every `one_in` events is recorded.""" return [e for e in evts if e["id"] % one_in == 0] def visible(evts, one_in): """Record count, observed heavy events, missed heavy events, observed subject set.""" k = sample(evts, one_in) heavy = sum(1 for e in k if e["heavy"]) return len(k), heavy, sum(1 for e in evts if e["heavy"]) - heavy, {e["subject"] for e in k} subs = subjects() evts = events(subs) print(f"subject {len(subs)} | event {len(evts)} | heavy event {sum(e['heavy'] for e in evts)} | " f"event-producing subject {len({e['subject'] for e in evts})}") print() print(f"{'1-in-N':>7s} {'records':>7s} {'bytes':>6s} {'heavy seen':>10s} " f"{'heavy missed':>12s} {'heavy estimate':>14s} {'invisible subjects':>19s}") for one_in in (1, 2, 5, 10, 20): k, heavy, missed, seen = visible(evts, one_in) print(f"{one_in:7d} {k:7d} {k * RECORD_BYTES:6d} {heavy:10d} {missed:12d} " f"{heavy * one_in:14d} {len(subs) - len(seen):19d}") seen10 = {e["subject"] for e in sample(evts, 10)} counts = {s["id"]: sum(1 for e in evts if e["subject"] == s["id"]) for s in subs} missing = [n for n in counts if n not in seen10] print() print(f"1-in-10 | invisible subjects {len(missing)}" f" | average events: invisible " f"{sum(counts[n] for n in missing) / len(missing):.2f}, " f"visible {sum(counts[n] for n in counts if n in seen10) / len(seen10):.2f}") print(f"invisible subject's highest event count {max(counts[n] for n in missing)}" f" | lowest-event subject {min(counts.values())} events" f" | highest-event subject {max(counts.values())} events") ``` ``` subject 40 | event 400 | heavy event 22 | event-producing subject 40 1-in-N records bytes heavy seen heavy missed heavy estimate invisible subjects 1 400 25600 22 0 22 0 2 200 12800 9 13 18 1 5 80 5120 7 15 35 8 10 40 2560 2 20 20 17 20 20 1280 2 20 40 26 1-in-10 | invisible subjects 17 | average events: invisible 8.18, visible 11.35 invisible subject's highest event count 13 | lowest-event subject 5 events | highest-event subject 18 events ``` #### Reading the Numbers The first line gives the measurement's baseline. At full count, records are **400**, observed heavy events **22**, missed **0**, invisible subjects **0**. All forty of the forty subjects produced at least one event; invisibility, then, is not a property of the population but a result the lever produces. **Missed heavy events.** At one-in-two, records fall by half and observed heavy events drop from 22 to **9**: **13** heavy events are missed. The disproportion here is notable — while records fall by half, heavy-event visibility falls by more than half. The reason is that heavy events are few: how twenty-two of them land against a fixed-stride filter is left to chance. At one-in-ten, observed heavy events fall to **2** and **20** heavy events are missed. At one-in-twenty, records fall by half again, but observed heavy events stay at **2**; the decline is not smooth, because what is being counted is no longer statistics but coincidence. **The estimate misses.** The usual path from sample to total is multiplying the observed count by the sampling rate. The column gives this estimate: while the true value is **22**, the estimate comes out to **22**, **18**, **35**, **20**, and **40** in turn. At one-in-five the estimate is one and a half times the truth; at one-in-twenty, nearly double. If the same multiplication were applied to the record count, it would give exactly **400** on every line — because the total is large and evenly distributed. The rule is this: **sampling preserves the total, not the minority.** A heavy event is a minority, and the minority is exactly what the measurement is after. **Invisible subjects.** The last column is this lesson's main number. At one-in-two, **1** subject is absent from every line of the report; at one-in-five, **8**; at one-in-ten, **17**; at one-in-twenty, **26**. Twenty-six of the forty subjects — more than two-thirds — are entirely absent from the dashboard. The last two lines say who this vanishing touches. At one-in-ten, the **17** invisible subjects' average event count is **8.18**; the visible twenty-three's is **11.35**. The loss is not random: a subject that talks less falls out first. But the filtering is not a sharp threshold either — one of the invisible subjects has **13** events, meaning a device that produced more than twice the traffic of the population's least active subject (**5** events) could still drop out of the report. The loss is both biased and coincidental, and because both are at play, the operator cannot estimate who is invisible just by looking at the report. The result is the course's third claim: **the lever renders the tail invisible.** The sampling rate is set as a performance dial, and it genuinely does bring the record volume down from 25,600 bytes to 1,280 bytes. But the same setting also decides which subject gets reported, and nobody made that decision explicitly. The smallest measurable difference across forty subjects is **1/40 = 0.025**; 26 subjects is twenty-six times that, and well inside the measurement band. #### The Gap in the Dashboard The most dangerous thing about an invisible subject is that the dashboard does **not show it as missing**. The table format below makes this concrete; it is not a measurement, and it has not been executed. ```text # taught dashboard format , not executed top traffic-producing devices (window: last 5 min, source: flow record) rank device flows bytes heavy event 1 device-31 14 1.2 GB 1 2 device-07 12 0.9 GB 0 3 device-22 11 0.8 GB 1 ... 23 device-16 1 0.1 GB 0 total devices listed : 23 ``` The table lists twenty-three devices and is correct — every listed line rests on a record that was genuinely produced. Nothing in it is wrong. What is missing is that nowhere does it say the table was produced from a **population of forty devices**. The `total devices listed` field says 23; there is no field called `devices in inventory`. A ranking list can only rank what **produced data**, because the input to ranking is the record, and a device with no record never enters the ranking at all. The distinction is this: a zero-valued row is a piece of information, **a row that does not exist is not.** The first says "no traffic passed from this device"; the second says nothing at all, and the reader confuses it with the first. Until the inventory's total count is written onto the dashboard, this distinction stays invisible. #### Narrowing the Invisibility The sampling lever's tail cannot be removed — if records are being reduced, something will go unseen. What the tail **consists of** can, however, be chosen. **First narrowing — take heavy events outside sampling.** An event marked heavy is recorded regardless of the sampling rate. This brings missed heavy events down to **0** and raises the record volume by only as many as there are heavy events; at one-in-twenty, that is 42 records instead of 20. Its cost is that heaviness must be determined **on the device** — the device has to know what counts as heavy. **Second narrowing — at least one record per subject.** The sampling rate is kept, but at least one record is made mandatory for every subject in the window. Invisible subjects become **0**, and record volume grows by at most forty records. This buys visibility from **coverage**, not volume. **Third narrowing — keep the counter alongside the flow record.** A counter is not sampled. Even if the flow record never shows a subject, that subject's interface counter keeps being read and reports the presence of traffic. A counter does not say who spoke, but it says **someone spoke**; reading the two sources together lets silence and invisibility be told apart. What the three share is that they take the decision out of being implicit. Left alone, the sampling rate applies a visibility policy anyway; the difference is that the policy is unwritten. #### Summary - Telemetry comes from two sources: the flow record names the subject but is sampled because it is expensive, the counter is not sampled but does not name the subject. - The sampling rate is a single lever applied to forty subjects at once; dropping from one-in-one to one-in-twenty takes records from **400** to **20**, and record volume from 25,600 bytes to 1,280 bytes. - Observed heavy events fall from **22** to **2** and **20** heavy events are missed; the decline is not smooth, because a heavy event is a minority and how it lands against the filter is left to chance. - An estimate made from a sample preserves the total but not the minority: while the truth is 22, the estimate comes out to **22 / 18 / 35 / 20 / 40**. - Invisible subjects rise from **0** to **26**; the lever is set as a performance dial, but it also chooses who gets reported, and the gap is the same color as health on the dashboard. #### Next Step In this lesson, the device produced the records and the collector received them; who spoke when was never asked. Yet collection can be set up in two separate ways: the collector **asks** devices, or the device **reports** on its own. The two models' round-trip cost was measured in the Application Layer Protocols course and will not be repeated here. The next lesson takes up the same distinction with this course's measure: in poll-based monitoring and event notification, **how many subjects never get reported**, and are the subjects the two models miss the same subjects? ### Device Monitoring Protocols Source: https://academia.sh/en/courses/network-operations/device-monitoring-protocols Poll-based monitoring and event notification lose the same forty devices in different ways: within a seven-hundred-millisecond scan budget, polling sees 29 devices at best, at a nine-event notification threshold 25 devices speak, and the number of devices neither model ever sees is 6. The previous lesson read telemetry from the record itself: flow records and counters were produced on the device and went to a collector. **Who initiated the record** was never asked. Yet collection can be set up in two separate ways, and the choice is itself a lever. In the first form the manager asks: it walks the devices in order, requests values from each, and writes down the reply. In the second form the device reports: it sends a message to the manager on its own when a threshold is crossed or a state changes. The two models' **round-trip cost and capture latency** were measured in the Application Layer Protocols course and will not be repeated here. This lesson's question is different: on the same forty devices, **how many subjects does each model never report** — and are the subjects they fail to report the same subjects? #### The Two Models' Message Formats In poll-based monitoring, the device presents an **object tree**. Every measurable value has an object identifier, and the manager requests the value by giving the identifier. The tree is ordered, so the manager can also walk a branch it does not know by saying "give me the next one." In event notification the direction is reversed: the device sends the same object identifier and value without being asked. Notification has two forms, acknowledged and unacknowledged; the difference between them was measured in the previous course and is only mentioned here. A third form sets up a **subscription**: the manager declares a path and an interval, and the device streams that path on a regular schedule or on every change. ```text # taught message format , not executed poll-based monitoring manager -> device query object identifier 1.3.6.1.2.1.2.2.1.10.3 device -> manager reply value 918233100, type counter32 manager -> device next next branch in the tree event notification device -> manager notification object identifier, value, timestamp device -> manager acknowledged notification manager acknowledges, device retries if none arrives subscription manager -> device subscribe path /interfaces/interface/counters, interval 10 s device -> manager stream on every interval or on every change ``` The format itself does not enter the measurement. What enters the measurement is the setting **that decides what each model reports**. #### Polling's Budget In the polling model, the manager walks the list from the start and asks each device in order. The scan has to finish within a period: when the next period begins, the scan starts over from the beginning. This scan amounts to a **time budget**. Two things spend the budget. A device that responds takes up time equal to its own latency. A device that does not respond takes up time equal to the **timeout** — and gives no data at all. The setting in the manager's hands is this timeout threshold, and it pulls in two directions at once. If the threshold is kept low, a slow device is cut off before it can answer and goes unreported; in exchange, every cutoff is cheap, and the scan reaches the end of the list. If the threshold is raised, slow devices can answer too, but every answer takes up more time and the budget runs out before the scan reaches the end of the list. In that case, devices at the **tail** of the list never even get asked. The two causes must be counted separately: a device that times out was asked but did not answer, while a device that is never asked is left out without the manager even knowing about it. #### Notification's Threshold The notifying model has no scan, and therefore no time budget. In its place, a **notification threshold** sits on the device: the device does not speak on every event, only when it sees activity crossing the threshold. Without the threshold, the manager would receive every event that falls in the window as a separate message, and that would be a far bigger load than the scan budget. The silence the threshold produces has two meanings, and the message does not tell them apart. If a device is quiet, either it genuinely did not cross the threshold, or it is in no condition to speak. In polling, silence is at least a silence that was **asked for**; in notification, it was not even asked. The measurement's assumptions: - **OB7** — The forty subjects are the course's fixed population; each subject has a response latency, and this latency is the query's time cost. - **OB8** — The scan period budget is **700 ms**. The sum of the forty devices' latencies is above this budget, meaning the scan cannot cover the whole list even in the best case. - **OB9** — The scan walks the list in subject order and starts over from the beginning every period; devices at the tail never enter the round at all. If the order is shuffled, which subjects are invisible changes, but not how many. - **OB10** — A query that times out takes up time equal to the timeout from the budget and returns no value. - **OB11** — The notification threshold is defined over the number of events produced in the window: a device sends a notification only if it produced as many events as the threshold or more, and when it sends, it sends all of its events. - **OB12** — All of a reported device's heavy events are visible; all of an unreported device's heavy events are missed. The oracle is the setup, and which event is heavy is known because we wrote it. - **OB13** — In the row comparing the two models, a timeout of 30 ms was chosen for polling and a threshold of 9 events for notification; these two settings produce blind spots of a similar size. #### Measurement ```python """Device monitoring: polling vs. notification, what is measured is what neither reports. Part 1 - polling: fixed time budget, lever is the timeout threshold. Part 2 - notification: lever is the notification threshold (events needed in the window). Part 3 - intersection of the two models' invisible-subject sets. """ SEED = 20260812 SCAN_BUDGET = 700 # ms, one scan period def rng(seed): state = seed % 2147483646 + 1 def draw(n): nonlocal state state = (state * 48271) % 2147483647 return state % n return draw def subjects(count=40, seed=SEED): draw, items = rng(seed), [] for i in range(count): items.append({"id": i + 1, "capacity": 20 + draw(81), "latency": 5 + draw(45), "class": ("interactive", "batch", "standby")[draw(3)], "special": draw(9) == 0}) return items def events(subs, count=400, seed=SEED + 3): draw, items = rng(seed), [] for i in range(count): s = subs[draw(len(subs))] items.append({"id": i + 1, "subject": s["id"], "heavy": draw(25) == 0}) return items def poll(subs, timeout, budget=SCAN_BUDGET): """The scanner walks the list from the start; each query costs as much time as the device's latency, an unresponsive device costs as much as the timeout. The scan stops when the budget runs out.""" spent = asked = 0 seen = set() for s in subs: cost = min(s["latency"], timeout) if spent + cost > budget: break spent += cost asked += 1 if s["latency"] <= timeout: seen.add(s["id"]) return asked, spent, seen def notify(subs, evts, threshold): """A device sends a notification only if the event count in the window reaches the threshold.""" counts = {s["id"]: 0 for s in subs} for e in evts: counts[e["subject"]] += 1 speaking = {n for n, c in counts.items() if c >= threshold} return sum(counts[n] for n in speaking), speaking def missed(evts, seen): return sum(1 for e in evts if e["heavy"] and e["subject"] not in seen) subs = subjects() evts = events(subs) print(f"subject {len(subs)} | event {len(evts)} | heavy event {sum(e['heavy'] for e in evts)}" f" | latency {min(s['latency'] for s in subs)}–{max(s['latency'] for s in subs)} ms" f" | full scan time {sum(s['latency'] for s in subs)} ms") print() print(f"{'timeout':>9s} {'asked':>7s} {'spent ms':>9s} " f"{'responded':>10s} {'timed out':>10s} {'never asked':>12s} " f"{'heavy missed':>13s}") poll_seen = {} for za in (10, 20, 30, 40, 50): s, g, seen = poll(subs, za) poll_seen[za] = seen print(f"{za:9d} {s:7d} {g:9d} {len(seen):10d} {s - len(seen):10d} " f"{len(subs) - s:12d} {missed(evts, seen):13d}") print() print(f"{'threshold':>10s} {'speaking':>9s} {'messages':>9s} " f"{'never reported':>15s} {'heavy missed':>13s}") notify_seen = {} for esik in (1, 6, 9, 12, 15): msgs, spk = notify(subs, evts, esik) notify_seen[esik] = spk print(f"{esik:10d} {len(spk):9d} {msgs:9d} {len(subs) - len(spk):15d} " f"{missed(evts, spk):13d}") print() a, b = poll_seen[30], notify_seen[9] ya, yb = set(s["id"] for s in subs) - a, set(s["id"] for s in subs) - b print(f"polling (timeout 30) missed {len(ya)} | notification (threshold 9) missed {len(yb)}") print(f"missed by both {len(ya & yb)} | missed by only one {len(ya ^ yb)}" f" | seen by either {len(subs) - len(ya & yb)}") print(f"heavy events missed by both together {missed(evts, a | b)}") ``` ``` subject 40 | event 400 | heavy event 22 | latency 6–48 ms | full scan time 935 ms timeout asked spent ms responded timed out never asked heavy missed 10 40 385 10 30 0 18 20 40 634 18 22 0 9 30 36 698 24 12 4 8 40 29 664 26 3 11 5 50 29 678 29 0 11 4 threshold speaking messages never reported heavy missed 1 40 400 0 0 6 37 385 3 1 9 25 301 15 3 12 13 184 27 15 15 5 83 35 16 polling (timeout 30) missed 16 | notification (threshold 9) missed 15 missed by both 6 | missed by only one 19 | seen by either 34 heavy events missed by both together 1 ``` #### Reading the Polling Table The sum of the forty devices' latencies is **935 ms**, and the budget is **700 ms**. Even with no timeouts at all, the scan cannot cover the whole list; this means the lever's range of motion is limited from the very start. At a **10 ms** timeout, the scan is cheap: all forty of the forty devices get asked, and a total of **385 ms** is spent. But only **10** devices respond — thirty could not fit under the threshold. The report shows a quarter of the forty devices, and **18** heavy events are missed. When the threshold is raised to **50 ms**, timed-out devices fall to **0**; now every device that gets asked responds. In exchange, the scan stops at **29** devices and **11** devices never get asked. The number of respondents has risen from 10 to **29**, but it never reaches forty. The **30 ms** row sitting in between is where both causes act at once: **12** devices time out, **4** devices never get asked, leaving **24** respondents. The same report carries two separate reasons for invisibility, and on the dashboard both sit as the same gap. The heavy-events-missed column follows this trend but never falls to zero on any row: **18, 9, 8, 5, 4**. Even at the most generous setting, four of the twenty-two heavy events never enter a single query. #### Reading the Notification Table At a notification threshold of **1**, no device stays quiet: all forty speak, **0** devices stay invisible, and no heavy event is missed. The cost sits in the column — **400** messages. Polling, at its most expensive row, was firing forty queries; notification here produces ten times that many messages. Notification load depends not on the number of subjects but on the **number of events**, and this is the two models' most fundamental structural difference. When the threshold is pulled down to **9**, messages fall to **301** and **25** devices speak; **15** devices are never reported. At threshold **15**, the load drops as low as **83** messages, but now only **5** devices are speaking and **35** devices are silent. **16** of the twenty-two heavy events are missed. What stands out is that the threshold drops the load faster than linearly: as it falls from 400 to 83, visible devices fall from 40 to 5. The threshold keeps the minority that produces many events and drops the rest. The device polling lost was the **slow** device; the device notification loses is the **quiet** device. #### The Blind Spots Are Not the Same The last three lines are this lesson's real conclusion. At a 30 ms timeout, polling never reports **16** devices; at a threshold of 9, notification never reports **15**. The numbers are close to each other; but **the devices neither one sees number only 6**. The thirty-one blind spots fall on twenty-five distinct devices, and **nineteen** of those devices are blind to one model only. This has a direct operational consequence. When the two models run together, the number of visible devices rises to **34**, and the heavy events missed by both together fall from **22** to **1**. The best polling setting alone was missing 4 heavy events, and the best notification setting only worked at the cost of making all forty devices speak; laid on top of each other, everything becomes visible except a single heavy event. The reason for this is not the quality of the models but the fact that **their blind spots are independent.** Polling filters by latency, notification filters by activity; a device being both slow and quiet is a separate coincidence, and it happens in only six of the forty devices. The tail does not disappear — six devices are still absent from every report — but who the tail consists of becomes measurable. #### Inventory and Liveness There is a place where the measurement knows all forty subjects from the start, and this is a convenience the setup grants the operator. In the field, both models need an **inventory**, but their needs point in opposite directions. Polling uses the inventory as **input**: a device not on the list is never asked, so a gap in the inventory turns directly into an invisible subject. There is an upside to this — because the list is known, **who did not respond** is also known. In the table above, the "timed out" column is exactly this information, and it is the name of a problem. Notification produces the inventory as **output**: the manager comes to know the devices that send it messages. If a device not in the inventory speaks, the manager learns about it — something polling cannot do. But the reverse is also true: a device that never speaks never appears in the inventory at all, and **its absence does not show up as a problem.** As with the ranking list in the previous lesson, a row that does not exist carries no information. The common mechanism that closes this asymmetry is a **heartbeat**: the device sends a short message at regular intervals even when it has nothing to say. The manager can now measure silence, because silence has become a missing message. Its cost is a calculable load — forty devices sending a notification once a minute means 2,400 messages an hour, and this load is independent of the event count. Its gain, though, is directly visible in the measurement: at a notification threshold of 15, **35** devices are silent; once a heartbeat is added, all thirty-five of them become visible, carrying not the information of **what they did**, but that **they exist**. #### Summary - In poll-based monitoring the manager asks, in event notification the device speaks; the two models' round-trip cost was measured in the previous course, what is measured here is **how many subjects never get reported**. - Polling's lever is the timeout threshold, and it cuts from two directions at once: when the threshold is low, **30** devices time out; when it is high, **11** devices never get asked; the respondent count is at most **29** within a 700 ms budget. - Notification's lever is the threshold, and it ties the load to the event count: at threshold 1, **400** messages with **0** invisible devices; at threshold 15, **83** messages with **35** invisible devices. - Polling loses the **slow** device, notification loses the **quiet** device; because the two models' blind spots are independent, the intersection of two sets of 16 and 15 is only **6** devices. - When the two models are used together, visible devices rise to **34** and missed heavy events fall to **1**; the tail does not disappear, but who is in it is now known. #### Next Step Both of these models relied on the device's **own account**: the device produced both the reply and the notification, and the device decided their content. If a device has a behavior it never reports, there is no way to learn it by asking the device. The next lesson therefore changes the source and looks at the packet itself: once a filter is set up to sift the bytes crossing the wire, how much does the filter **fail to see** — how many flows does a narrow filter miss, and how many bytes does a wide one? ### Packet Analysis Tools Source: https://academia.sh/en/courses/network-operations/packet-analysis-tools The capture filter is a single lever, and it costs at both ends: in a window of ninety-eight flows, the narrowest filter never sees 91 flows and misses 21 of 22 heavy events, while unfiltered capture drops 776,196 bytes because they do not fit the buffer. The previous two lessons both relied on the device's **own account**. The device produced the flow record, the device answered the query, the device sent the notification. If a device has a behavior it never reports, there is no way to learn it by asking the device. This lesson changes the source and looks at the bytes crossing the wire itself. An analyzer placed at a point copies and records the packets passing through it. Copying is not unlimited: a **capture filter** chooses which packets get copied, and the copies are written into a limited buffer. The filter is a lever — a single rule applies to all forty subjects' traffic at once. What this lesson measures is not what the filter sees, but what it **fails to see**. The use of analyzer tools and the capture procedure is the subject of the **Security Tooling and Penetration Testing** course and is not repeated here; nor are tool names written. What is measured here is not the tool, but the **rule** itself. #### Capture's Two Limits Capture loses at two separate points, and the two work in opposite directions. **The first limit is the filter.** A packet that does not pass the filter is never copied at all. This is not a loss but a choice — until the behavior being sought falls outside the filter. A narrow filter protects the buffer by filtering out irrelevant traffic; in exchange, it can never bring back what it filtered out, because a packet that was never copied does not exist. **The second limit is the buffer.** Packets that pass the filter are written to a buffer, and once the buffer fills, the next incoming packet is dropped. This drop is **silent**: there is no line in the record that says "a packet was dropped here," only packets that are not there. A wide filter fills the buffer quickly, and the drop happens right in the middle of the traffic the filter cares about. Both limits are tied to the same lever. When the filter is narrowed, the first limit grows and the second shrinks; when it is widened, the reverse happens. There is no loss-free setting in between, only a choice of **where** the loss falls. #### Filter Syntax Capture filters are written over the packet's header fields. The dump below shows the taught syntax; it has not been executed and carries no numeric claim. ```text # taught filter syntax , not executed single field tcp looks at the protocol field port 443 source or destination port src port 443 source port only host station.example source or destination host compound tcp and port 443 tcp and (port 80 or port 443 or port 8080) not port 22 in-packet position tcp[13] & 2 != 0 segments with the SYN flag set ip[6:2] & 0x1fff = 0 unfragmented first fragment ``` Two things deserve attention. First, in practice the filter addresses **header fields**; reading a byte inside the payload is possible too, but it requires computing the variable-length header's size inside the rule, and the rule quickly becomes unreadable. The real limit is elsewhere: the decision is made packet by packet, and the flow as a whole cannot be looked at. Second, a negation like `not port 22` widens the filter — a rule that looks narrow is actually a wide one if what it excludes is small. The measure of a filter's width is not the length of its syntax but **the number of matching flows.** #### Snap Length and Flow Reassembly The **snap length** determines how much of each packet is kept. If only headers are to be kept, a few dozen bytes per packet is enough; if the full payload is wanted, the entire packet is written, and the buffer fills ten times faster in the same span. Snap length directly limits **flow reassembly**. Reassembly is merging the packets belonging to the same flow by their sequence numbers to recover the original byte stream. Headers are enough if the question is "when did this flow start, how many bytes did it carry, how did it close." If the question is "what did this flow carry," headers are not enough, and a truncated record can never answer that question. Reassembly's second condition is **continuity**: if one of the flow's packets was dropped, the flow cannot be fully reconstructed. This is why, in the measurement, a flow can land in one of three separate outcomes — fully captured, left half-done, or never fit into the buffer at all. The measurement's assumptions: - **OB14** — The forty subjects are the course's fixed population; each subject produces between one and four flows in the window, and the flows are ordered by start time. - **OB15** — The capture buffer is **1,200,000 bytes** and is not flushed during the window. In a real capture, the buffer is flushed to disk; keeping it fixed here isolates the drop's dependence on filter width. - **OB16** — Snap length is fixed at **96 bytes/packet** and is not a lever in this lesson. Only the filter is changed. - **OB17** — The buffer fills in time order. A flow that ends by the moment it fills is fully captured, a flow that straddles that moment is left half-done, and a flow that starts afterward is never written at all. - **OB18** — A dropped packet leaves no trace; the measurement reads the dropped bytes from the oracle, not from the record. In a real capture, this number can only be estimated from a counter. - **OB19** — A heavy event is the oracle's knowledge. If even a single flow of a subject was captured, that subject's heavy events count as visible; if none of its flows were captured, all of them are missed. - **OB20** — The filters are nested: every filter matches all the flows the previous one matched. Width is thus measured on a single one-directional axis. - **OB21** — The observation point sees all forty subjects' flows. This is a convenience with no counterpart in the field; the measurement therefore only counts the loss from the filter and the buffer, not the loss from position. #### Measurement ```python """Packet capture: the capture filter is a single lever, what is measured is what it fails to see. Part 1 - forty subjects' flows, ordered by start time. Part 2 - five filter widths: how many flows a narrow filter misses, how many bytes a wide one drops. """ SEED = 20260812 BUFFER = 1_200_000 # bytes, capture buffer SNAP = 96 # bytes, most kept per packet def rng(seed): state = seed % 2147483646 + 1 def draw(n): nonlocal state state = (state * 48271) % 2147483647 return state % n return draw def subjects(count=40, seed=SEED): draw, items = rng(seed), [] for i in range(count): items.append({"id": i + 1, "capacity": 20 + draw(81), "latency": 5 + draw(45), "class": ("interactive", "batch", "standby")[draw(3)], "special": draw(9) == 0}) return items def events(subs, count=400, seed=SEED + 3): draw, items = rng(seed), [] for i in range(count): s = subs[draw(len(subs))] items.append({"id": i + 1, "subject": s["id"], "heavy": draw(25) == 0}) return items def flows(subs, seed=SEED + 7): """Each subject produces between one and four flows; the list is ordered by start time.""" draw, items = rng(seed), [] for s in subs: for _ in range(1 + draw(4)): items.append({"subject": s["id"], "protocol": ("TCP", "UDP", "ICMP")[draw(3)], "port": (22, 53, 80, 123, 443, 8080)[draw(6)], "packets": 20 + draw(400), "size": 64 + draw(1400), "start": draw(1000)}) return sorted(items, key=lambda a: (a["start"], a["subject"])) def capture(fl, filt, buffer=BUFFER, snap=SNAP): """Flows passing the filter are written to the buffer; once it fills, the remaining bytes drop.""" filled = matched = whole = half = overflow = dropped = 0 seen = set() for a in fl: if not filt(a): continue matched += 1 size = a["packets"] * min(a["size"], snap) if filled + size <= buffer: filled, whole = filled + size, whole + 1 seen.add(a["subject"]) elif filled < buffer: half += 1 dropped += filled + size - buffer filled = buffer seen.add(a["subject"]) else: overflow += 1 dropped += size return matched, whole, half, overflow, dropped, seen FILTERS = { "TCP port 443": lambda a: a["protocol"] == "TCP" and a["port"] == 443, "TCP three ports": lambda a: a["protocol"] == "TCP" and a["port"] in (80, 443, 8080), "TCP only": lambda a: a["protocol"] == "TCP", "TCP and UDP": lambda a: a["protocol"] in ("TCP", "UDP"), "unfiltered": lambda a: True, } subs = subjects() evts = events(subs) fl = flows(subs) print(f"subject {len(subs)} | flow {len(fl)} | packet {sum(a['packets'] for a in fl)}" f" | heavy event {sum(e['heavy'] for e in evts)}") print(f"buffer {BUFFER} bytes | snap {SNAP} bytes/packet" f" | snapped total {sum(a['packets'] * min(a['size'], SNAP) for a in fl)} bytes" f" | unsnapped {sum(a['packets'] * a['size'] for a in fl)} bytes") print() print(f"{'filter':<15s} {'matched':>8s} {'outside':>8s} {'whole':>6s} {'half':>5s} " f"{'overflow':>9s} {'bytes dropped':>13s} {'invisible':>10s} {'heavy missed':>13s}") for name, f in FILTERS.items(): e, whole, half, over, dropped, seen = capture(fl, f) missed = sum(1 for x in evts if x["heavy"] and x["subject"] not in seen) print(f"{name:<15s} {e:8d} {len(fl) - e:8d} {whole:6d} {half:5d} {over:9d} " f"{dropped:13d} {len(subs) - len(seen):10d} {missed:13d}") ``` ``` subject 40 | flow 98 | packet 20631 | heavy event 22 buffer 1200000 bytes | snap 96 bytes/packet | snapped total 1976196 bytes | unsnapped 15032397 bytes filter matched outside whole half overflow bytes dropped invisible heavy missed TCP port 443 7 91 7 0 0 0 33 21 TCP three ports 17 81 17 0 0 0 23 13 TCP only 38 60 38 0 0 0 15 6 TCP and UDP 65 33 58 1 6 188868 11 4 unfiltered 98 0 65 1 32 776196 5 2 ``` #### What the Narrow Filter Misses The first row is the narrowest rule, and it works exactly as intended: all seven of the matching **7** flows are captured intact, not a single byte drops from the buffer, the record is small, and it is easy to read. Nothing is broken on this row. What is broken sits to the row's right. **91** of the ninety-eight flows are left outside the filter, and **33** of the forty subjects never appear in the record at all. **21** of the twenty-two heavy events are missed. The record is clean, because only what was meant to be looked at entered it; that is exactly the problem — the record was built on the assumption that what would be asked of it was already known. As the filter widens, this side improves. With `TCP three ports`, invisible subjects fall to **23**; with `TCP only`, to **15**; missed heavy events become **13** and **6**. Through the third row, the buffer is never strained at all: bytes dropped is **0**. In this range, the lever looks free. #### What the Wide Filter Misses In the fourth row, the second limit kicks in. The `TCP and UDP` filter matches **65** flows, but only **58** of them are captured whole: **1** flow is left half-done and **6** flows never fit in the buffer at all. Bytes dropped is **188,868**. In unfiltered capture, the table flips. The filter now leaves nothing outside — the `outside` column is **0**. In exchange, only **65** of the ninety-eight flows are captured whole, **32** never fit in the buffer at all, and **776,196** bytes drop. The snapped total is **1,976,196** bytes; that is, roughly two-fifths of the data meant to be captured never enters the record at all. The most striking column is invisible subjects: in unfiltered capture, this number is **5**, and it is not zero. Even though the filter excluded no subject, five subjects are absent from the record, because their flows started after the moment the buffer filled. **Widening the filter did not increase visibility — it moved the loss from the filter to the buffer.** The qualitative difference between the two losses matters. A flow the filter excludes was excluded **knowingly**; whoever reads the rule can say what was left out. A flow dropped from the buffer, by contrast, was dropped without anyone knowing, and the record itself carries no field showing this. The second loss is more dangerous even when it carries away fewer bytes than the first, because whoever reads the record does not notice the gap. #### The Uncounted Third Loss The measurement assumes the observation point sees all forty subjects' flows. This assumption does not hold in the field, and where it fails to hold, a third loss is born. The analyzer is placed at one point and only sees packets that **pass through** that point. If two subjects talk to each other on the near side of that point, the traffic between them is never copied at all — no matter how wide the filter or how large the buffer. In the same way, an analyzer attached to one port of a switch does not see traffic exchanged among the other ports; that requires the traffic to be separately mirrored, and mirroring carries its own limit: if the mirrored port's speed is lower than the combined speed of the sources, packets drop during mirroring too. This loss differs from the other two in that it **cannot be estimated by reading the filter rule.** What the narrow filter excludes is written into the rule itself; the amount dropped from the buffer can be read off a counter. What the position fails to see, though, can only be inferred from the topology, and it leaves no trace in the capture record at all. The three losses share one consequence: **an empty record does not mean nothing happened there.** #### Narrowing the Losses The lever's tail cannot be removed; three narrowings make it measurable. **First narrowing — count the drops.** The capture interface keeps a counter for dropped packets. The counter is not sampled and is not affected by the filter; if the record is incomplete, it is the only field that says so. Reading the counter every time the record is opened gives the field counterpart of the `overflow` column. **Second narrowing — choose the snap length by the question.** If the full packets in the measured window had been kept, the total would be **15,032,397** bytes; with a 96-byte snap, it falls to **1,976,196** bytes, roughly one-seventh. The same buffer, with the same filter, then covers a window seven times as long. If the question is a flow's existence and timing, this trade is close to free. If the question is content, the snap must be removed, but then the window must be shortened; the product of the two settings is constant. **Third narrowing — write the filter by the oracle, not the question.** In the measurement above, the narrowest filter missed twenty-one of twenty-two heavy events. Because which ports the heavy events pass through is unknown, a narrow filter is gambling. The workable rule is this: the capture filter is written to cover the subjects **invisible in telemetry**. The previous two lessons were counting who is invisible; this lesson says where that list gets written. #### Summary - Capture loses in two places: a packet that does not pass the filter is never copied, and a packet that does not fit the buffer drops silently; the two losses are opposite directions of the same lever. - The narrowest filter captures **7** flows and drops no bytes at all, but it leaves **91** flows outside, never shows **33** subjects, and misses **21** of 22 heavy events. - Unfiltered capture leaves no flow outside, but **32** flows never fit the buffer and **776,196** bytes drop; roughly two-fifths of the snapped total never enters the record. - Even in unfiltered capture, **5** subjects stay invisible: flows that start after the buffer fills never enter the record, even though the filter excluded nothing. - A flow the filter excludes is excluded knowingly; a flow dropped from the buffer drops unknowingly; the second loss is more dangerous even when it is smaller, because it leaves no trace in the record. - A third loss sits outside the measurement and comes from the observation point: traffic that does not pass through that point is invisible at any filter width, and this gap can be read from neither the rule nor a counter — only inferred from the topology. #### Next Step The three measurements so far all looked from **inside** the network: the device's record, the device's reply, the packet crossing the wire. All three had individual devices as their subject. The real question asked in operations, though, is asked end to end — can one end reach the other, and how long does it take? The next lesson builds this single indicator and asks it backwards: which of the forty subjects does an end-to-end health value **fail to represent**, and how many subjects stay in the tail while the indicator reads green? ### Reachability and Latency Measurement Source: https://academia.sh/en/courses/network-operations/reachability-and-latency-measurement The end-to-end health indicator is a sample: with a four-target probe list, 36 of the forty subjects are never probed, availability swings between 37.5 and 62.5 while the true value sits at 57.5, and the indicator's resolution is set by the target count, not the subject count. The previous three lessons all looked from inside the network: the record a device produces, the reply a device gives, the packet crossing the wire. All three had individual devices as their subject, and in all three, what was measured was how many devices stayed invisible. The real question asked in operations, though, is asked from outside, and it is one sentence: **can one end reach the other, and how long does it take?** The tests that answer this question are collapsed into a single number, and that number is written at the very top of the dashboard. This lesson's question is who that single number **speaks on behalf of**. #### Two Separate Words In this lesson, two concepts that get confused with each other are called by separate names, and the distinction is binding. **Reachability** is whether one end can get to a target or not. What measures it is a **reachability test**: an echo request is sent, a reply is awaited, the result comes out binary. The test itself, and how many separate causes a non-response can fit, were established in the **Linux Network Administration and Troubleshooting** course; here the test is used not as a procedure but as a **data source**. **Availability**, by contrast, is a ratio: how many of the tests succeeded. It is derived from a binary result and written as a percentage. This is the number we see on the dashboard. The difference sits at the center of the measure. Reachability is **about a subject**; availability is **about a set**, and the probe list determines who that set is. The list is a lever: a single configuration file chooses which of the forty subjects gets spoken for. Indicator design, alert thresholds, and error budgets are the subject of the **Observability and Reliability** course and are not repeated here. What is measured here is not how to design an indicator, but which subject an already-designed indicator **fails to represent.** #### How a Test Is Set Up ```text # taught probe definition and dashboard format , not executed probe definition name end-to-end-reach target list device-03, device-11, device-27, device-38 procedure reachability test, 5 attempts, 1 s interval success rule at least 4 attempts got a reply latency value median of the attempts window 24 h dashboard availability ....% average latency .... ms slowest target .... ms target count 4 ``` The definition's most important line is `target list`, and it never appears on the dashboard. Even though the dashboard has a `target count` field, it does not say **who** the targets are or **how many subjects** the inventory contains. The indicator reports the numerator, not the denominator. #### The Indicator Is a Sample The probe list selects a subset of the forty subjects, and the indicator measures only that subset. This has the same shape as the sampling rate from the first lesson — there, events were being selected; here, subjects are. The lever's tail is doubled in the same way: a subject not on the list never enters the indicator, and if a subject not on the list is broken, **the indicator stays green.** The second, less noticed, consequence is **resolution**. In a test with four targets, the ratio can only take the values 0, 25, 50, 75, and 100; the smallest measurable difference is **100/4 = 25 points**. Putting a one-percent threshold on such an indicator asks it to report a difference below its own measurement floor. The measurement's assumptions: - **OB22** — The forty subjects are the course's fixed population and are not changed in this lesson. - **OB23** — A subject counts as unreachable if it produced at least one heavy event in the window. The oracle is the setup; the test does not use this information, it is only compared against the result. - **OB24** — Every target is probed once, and the probe gives a binary result. Retrying attempts and the success rule do not change the result, they only reduce noise; the measurement therefore takes a single attempt. - **OB25** — The measured latency is the target's own latency. Contributions from devices along the path are not modeled in this lesson; if they were, the measured values would grow, but the trend below would not change. - **OB26** — The probe list's order is a fixed sequence coming from the setup, and the lists are nested: the eight-target list contains the whole of the four-target list. If the order changes, which subjects are invisible changes, but the counts stay similar. - **OB27** — `Availability` is computed only over the probed targets; an unprobed subject enters neither the numerator nor the denominator. The calculation in the field is the same. - **OB28** — The smallest measurable difference across forty subjects is 1/40 = 0.025; in a four-target list, it is 1/4 = 0.25. The indicator's resolution depends on the list, not the inventory. - **OB29** — The probes are taken as independent: each target's result comes only from its own state, path sharing is not modeled. If it were, the indicator's swing would grow. #### Measurement ```python """End-to-end health: the probe list is a single lever, what is measured is what the indicator fails to represent. Part 1 - forty subjects; a subject producing a heavy event counts as unreachable. Part 2 - five probe lists: availability, latency indicator, and the tail. """ SEED = 20260812 def rng(seed): state = seed % 2147483646 + 1 def draw(n): nonlocal state state = (state * 48271) % 2147483647 return state % n return draw def subjects(count=40, seed=SEED): draw, items = rng(seed), [] for i in range(count): items.append({"id": i + 1, "capacity": 20 + draw(81), "latency": 5 + draw(45), "class": ("interactive", "batch", "standby")[draw(3)], "special": draw(9) == 0}) return items def events(subs, count=400, seed=SEED + 3): draw, items = rng(seed), [] for i in range(count): s = subs[draw(len(subs))] items.append({"id": i + 1, "subject": s["id"], "heavy": draw(25) == 0}) return items def probe_order(subs, seed=SEED + 11): """The probe list's order; its prefixes give nested target sets.""" draw, remaining, order = rng(seed), [s["id"] for s in subs], [] while remaining: order.append(remaining.pop(draw(len(remaining)))) return order subs = subjects() evts = events(subs) lat = {s["id"]: s["latency"] for s in subs} heavy = {s["id"]: 0 for s in subs} for e in evts: if e["heavy"]: heavy[e["subject"]] += 1 unreachable = {n for n, c in heavy.items() if c >= 1} order = probe_order(subs) print(f"subject {len(subs)} | heavy event {sum(heavy.values())} | unreachable subject " f"{len(unreachable)} | true availability " f"{100 * (len(subs) - len(unreachable)) / len(subs):.1f}%") print(f"true latency: average {sum(lat.values()) / len(subs):.1f} ms, " f"slowest {max(lat.values())} ms, fastest {min(lat.values())} ms") print() print(f"{'targets':>7s} {'unprobed':>8s} {'availability':>12s} {'resolution':>10s} " f"{'avg ms':>7s} {'slowest':>8s} {'invisible':>9s} {'heavy missed':>13s}") for n in (4, 8, 16, 32, 40): h = order[:n] ok = [x for x in h if x not in unreachable] print(f"{n:7d} {len(subs) - n:8d} {100 * len(ok) / n:11.1f}% " f"{100 / n:9.1f}p {sum(lat[x] for x in h) / n:7.1f} " f"{max(lat[x] for x in h):8d} {len(unreachable - set(h)):9d} " f"{sum(c for x, c in heavy.items() if x not in h):13d}") ``` ``` subject 40 | heavy event 22 | unreachable subject 17 | true availability 57.5% true latency: average 23.4 ms, slowest 48 ms, fastest 6 ms targets unprobed availability resolution avg ms slowest invisible heavy missed 4 36 50.0% 25.0p 22.0 40 15 20 8 32 37.5% 12.5p 21.6 40 12 15 16 24 62.5% 6.2p 21.1 40 11 14 32 8 56.2% 3.1p 26.0 48 3 3 40 0 57.5% 2.5p 23.4 48 0 0 ``` #### The Subject Not Represented With the four-target list, the indicator reads **50.0%**. Behind this number are two successful and two failed tests — four observations in total. At the same time, **36** of the forty subjects were never probed at all, **15** unreachable subjects never appear in a single test, and **20** heavy events never enter the indicator at all. The structure here is the same as the first lesson's result, read backwards. There, an unreported subject left a gap on the dashboard; here it does not even leave a gap, because the indicator is a single percentage, and a gap does not show inside a percentage. **An unprobed subject neither raises nor lowers the indicator; it sits outside the indicator altogether.** At sixteen targets, unprobed subjects fall to **24**; at thirty-two targets, to **8**. Invisible unreachable subjects do not fall at the same rate: 15, 12, 11, 3. On the third row, doubling the list's target count only drops the invisible unreachable subjects one at a time — because the targets added to the list are mostly subjects that are already working. Growing the coverage does not shrink the tail proportionally. #### The Indicator's Own Noise As important as the columns on the right is how the percentage on the left behaves. As the probe list grows, the indicator reads **50.0**, **37.5**, **62.5**, **56.2**, and **57.5** in turn. The true value is **57.5** across the entire window and never changed. With nothing happening on the network, the indicator swings **between 37.5 and 62.5**, a band of twenty-five points. The reason is written in the resolution column. In a four-target list, a single test's result moves the indicator by **25 points**; at eight targets, **12.5**; at forty targets, **2.5** points. If an alert threshold is set below this resolution, the alert measures the list, not the network. This has a direct operational consequence: **an availability ratio computed from a small target list cannot be read as a trend.** Seeing 50 and then 75 in two consecutive windows is not an improvement, it is a single subject changing state. The measure's band is the divisor itself. #### What the Average Hides The latency columns show a third tail. At four targets, the measured average is **22.0 ms**; the forty subjects' true average is **23.4 ms**. The gap is small, and the indicator therefore looks correct. The misleading column is not the average, it is the **slowest**. Across the four-, eight-, and sixteen-target lists, the measured slowest value is **40 ms** in all three. The population's true slowest subject is **48 ms**, and it only shows up in the thirty-two-target list. So up to sixteen targets, the indicator underreports the tail by eight milliseconds — and it does so without giving any sign of it. The relationship between the average and the slowest also reverses: at thirty-two targets, the average **rises** to **26.0** ms, because slow subjects have entered the list. The indicator looks like it has gotten worse, but the network has not changed; only the field of view has widened. **An indicator getting worse does not mean the thing it measures has gotten worse.** #### The Probes Are Not Independent The measurement counts every target as a separate observation and computes the percentage accordingly. In the field, this assumption only partly holds, because the probes all start **from the same point** and share their paths. Every target leaving a single vantage point shares the first few devices. If a shared device breaks, every target fails at once, and the indicator drops to **0%**. Even if thirty-nine of the forty subjects are sound, the table reports a total outage. The reverse case is subtler: when a device far from the vantage point breaks, only one or two targets go through it, so the indicator drops by only a few points and stays under the threshold. This is why a percentage measured from a single point does not give information about a **location**. A drop in the percentage does not mean "this much of the network is broken," it means "from this point, this much is not visible." The way to tell locations apart is not to tweak the divisor, but to **multiply the vantage points**: when the same target list is run from two separate points, targets that fail from only one point show a problem specific to that point, and targets that fail from both show a problem in the target itself. This distinction was not modeled in the measurement; the probes were taken as independent, and every target's result came only from its own state. If it had been modeled, the swing in the availability column would have **grown**, because a shared device's state would push multiple targets in the same direction at once. The independence assumption makes the indicator look steadier than it is. #### Active and Passive Measurement The measurement above is **active measurement**: the side doing the measuring produces the probe traffic. This topic's first three lessons, by contrast, were **passive measurement** — reading from traffic that already existed, without adding any packets. The two have their tails in opposite places, and this is the reason for reading all four together. Passive measurement only sees a subject that is **talking**. A device with no traffic produces no flow record, never enters a capture, and sends no notification; this is why the twenty-six subjects lost in the first lesson were lost. Passive measurement, in exchange, measures real user traffic: the latency it sees is a latency someone actually experienced. Active measurement does the reverse. Because it produces the probe traffic itself, it can also measure a silent subject; a device placed on the list is probed even if it is never used at all. Its cost is twofold. The latency it measures is a latency nobody experienced — probe packets can differ from the real load's path, the real load's size, and the real load's class. Second, the measurement is **bound by the list**: active measurement has to know in advance what to measure, while passive measurement can see even what it does not know about. The two methods have no shared gap, and this is not a coincidence. The subject passive measurement fails to see is the silent one; the subject active measurement fails to see is the one not on the list. A device being both silent and off the list is a separate deficiency, and it comes from the inventory itself. The same result came out of the comparison between the monitoring models: **when independent blind spots are laid on top of each other, the tail shrinks, but a subject the inventory does not see appears in neither.** #### Narrowing the Tail Three narrowings make it visible who the indicator speaks on behalf of. **First narrowing — write the denominator.** A `subjects in inventory` field is placed on the dashboard alongside `target count`. With a four-target list, the table says "50%, 4/40 subjects," and the reader sees what was not measured. This is a correction made without adding any measurement. **Second narrowing — tie the resolution to the threshold.** The alert threshold is chosen larger than the list's resolution. For a four-target list, a threshold smaller than 25 points cannot be defined; if it needs to be defined, the list must be grown. The threshold and the divisor are decided together. **Third narrowing — feed the target list from the previous lessons' output.** A subject never visible in telemetry, a subject never reported in monitoring, and a subject never captured — these are the probe list's **first candidates**. All three lessons were counting who is invisible; this lesson says where that list gets written. A list fed this way closes more of the tail while staying smaller than a randomly chosen list. #### Summary - Reachability is about a subject and gives a binary result; availability is a ratio, and the probe list determines its set — the list is a lever. - With a four-target list the indicator reads **50.0%**, while **36** of the forty subjects are never probed, **15** unreachable subjects stay invisible, and **20** heavy events never enter the indicator at all. - As the list grows, unprobed subjects fall from 36 to 0, but invisible unreachable subjects do not fall at the same rate (15, 12, 11, 3): growing the coverage does not shrink the tail proportionally. - While the true value is fixed at **57.5%**, the indicator swings between **37.5** and **62.5**; resolution is **25** points at four targets and **2.5** at forty, and the alert threshold cannot be set below this band. - The measured slowest value stays at **40 ms** through sixteen targets; the true slowest is **48 ms**; at thirty-two targets the average rises because the field of view changed, not the network. #### Next Step Every lever touched throughout this topic was a lever of **observation**. The sampling rate, the timeout threshold, the capture filter, and the probe list — all four decided what would be seen, but none of them touched the network itself. A badly set sampling rate drops a device out of the report; it does not drop a connection. This is where the whole weight of the measurement came from: a subject left in the tail was not harmed, only unseen. The next topic removes this distinction. What happens when the lever is no longer observation's but **configuration itself**? A single rule again applies to all forty subjects at once, but this time the moment it is applied, the devices' behavior changes, and a subject left in the tail falls out not of the report, but of the service. ## Automation and Cloud Networking Manual configuration's error modes, defining device configuration with a data model, protocol-based configuration access, bulk change and validation, cloud networking's core objects, hybrid connectivity options, and validating topology in a lab. ### The Case for Network Automation Source: https://academia.sh/en/courses/network-operations/case-for-network-automation In manual configuration, error grows with touch count: forty devices take 122 field touches, producing 12 errors and 10 faulty devices. The template brings touches down to 1, but expected faulty devices rises from 9.76 to 11.34, and the six-device tail outside the template stays exactly where it was. The previous lesson measured end-to-end health indicators, and every lever touched so far was the same kind: the lever of **observation**. Sampling rate, capture filter, polling interval — all three determined not what the network did, but how much of it we saw. A misconfigured sampling rate does not break a device, it only hides it; turn the lever back and the device is still there. This topic changes the lever. The lever is now **configuration itself**. A misconfigured template does not hide a device, it **changes** it; and once applied, turning the lever back does not restore the old state, because the old state no longer exists anywhere. This lesson's question is not whether automation is a good idea — it is **what** manual configuration's error grows with, **how** automation cuts that growth, and **what tail remains** after it does. #### The Touch: The Unit of Manual Configuration The unit of manual configuration is not the device, it is the **field**. An operator configuring forty devices does not make forty decisions; on each device, they make and write as many decisions as that device requires. Because devices are not identical to one another, this count is not the same either. ```text # taught configuration transcript, not run device 14 device 33 interface : main interface : main address : /24 address : /24 scheduler : 400 ms priority : interactive local_rule : present ``` Two devices, not four and four, but three and four fields. A high-capacity device wants an aggregation field, a high-latency device wants a scheduler, a device in the interactive class wants a priority field, and a special device that must fall outside the template wants a local rule no template carries. The total surface of configuration is not the device count, it is the sum of these fields. The direct consequence: **in manual configuration, error grows with touch count, not device count** — and since touch count grows with device count, the result lands at the same door. A forty-device change does not produce four times the error of a ten-device change; it produces however many times the touch count grew. #### Manual Configuration's Error Modes Four modes can be distinguished, and each can be counted separately. **A field written wrong.** The value is written, the syntax is correct, the device accepts it, and it is wrong. The device reports no fault because there is no fault; the gap between intent and what was written is a gap the device cannot see. **A skipped device.** Thirty-nine of forty devices change, one does not. It keeps running for a long time, and the difference only becomes visible when an event singles it out from the others. **Order-dependent partial application.** If the change is interrupted halfway, some devices are left with the new configuration and some with the old. The network behaves inconsistently between these two sets. **Documentation and device diverging.** The written configuration says one thing, the actual configuration on the device says another. This gap is called **configuration drift**, established in the Wireless Networks and Network Security course as the gap between a boundary's intended state and what actually passes. Here, the same gap is measured in the **configuration itself**. What the four have in common is that none of them is reported by the device. The device applies what is written to it; whether what was written is correct lies outside the device's knowledge. #### The Template Is a Single Touch Automation makes a single structural claim: **making touch count independent of device count.** The configuration is written once, reviewed once, and applied to forty devices in a single operation. What is written is not a sequence of commands but the **desired end state** — this approach is called **declarative**; it defines not what the tool will do, but what the result will be. The infrastructure-as-code procedure and tool selection are the subject of the Infrastructure as Code course and are not repeated here. Here the object is not a server or a cloud resource, it is a **network device**; and what is measured is not the procedure's efficiency, but **the minority that falls outside the template.** Bringing the touch count down to one has a cost: the **spread of error**. In manual configuration, a field written wrong breaks one device. In a template, a field written wrong breaks forty at once. The probability of error does not fall, the **size** of the error changes: manual configuration produces many small errors, a template produces few very large ones. The assumptions the measurement rests on: - **AC1** — The forty-device subject set is the course's constant and is not changed in this lesson. The oracle is the setup itself: we know which device is special because we wrote it that way. - **AC2** — The fields a device must have written by hand are derived from its attributes: interface and address on every device, priority in the interactive class, a scheduler if latency is above 30 units, aggregation if capacity is 80 units or above, a local rule on a special device. - **AC3** — Every hand-written field has an error probability of 80 per thousand, and fields are independent. This rate is not a measurement but an assumption of the setup; if it changes, every row scales in the same direction. - **AC4** — The template carries four fields and is written once. The template's probability of coming out faulty is computed at the same per-field rate; if the template is faulty, all forty of forty devices are faulty at once. - **AC5** — The template is written for a device class. A special device whose class does not match the template's deviates; the deviation is not the device's fault, it is the template's scope decision. - **AC6** — The report is built on the interactive class's indicators; a deviation in another class never appears in the report. This is an assumption, not a measurement. #### The Measurement ```python """Manual configuration vs. the template's error model. Part 1 - manual campaign: touch count grows with subject count, errors with it. Part 2 - the template is a single touch; error spreads, the expected count does not fall. Part 3 - the template's own tail: what falls outside it and what the report cannot see. """ SEED = 20260812 ERROR_PER_MILLE = 80 def make_rng(seed): d = seed % 2147483646 + 1 def r(n): nonlocal d d = (d * 48271) % 2147483647 return d % n return r def subjects(count=40, seed=SEED): r, out = make_rng(seed), [] for i in range(count): out.append({"no": i + 1, "capacity": 20 + r(81), "latency": 5 + r(45), "class": ("interactive", "batch", "standby")[r(3)], "special": r(9) == 0}) return out def fields(o): """The configuration fields a device requires to be written by hand.""" f = ["interface", "address"] if o["class"] == "interactive": f.append("priority") if o["latency"] > 30: f.append("scheduler") if o["capacity"] >= 80: f.append("aggregation") if o["special"]: f.append("local_rule") return f def manual(subj, count, seed): """Count devices are touched one by one; each field has ERROR_PER_MILLE error.""" r, faulty = make_rng(seed), set() touches = errors = 0 for o in subj[:count]: for _ in fields(o): touches += 1 if r(1000) < ERROR_PER_MILLE: errors += 1 faulty.add(o["no"]) return touches, errors, len(faulty) def apply_template(subj, template_class): """A single template is applied to forty devices; special devices deviate.""" matched = deviated = hidden = 0 for o in subj: if o["class"] == template_class or not o["special"]: matched += 1 else: deviated += 1 if o["class"] != "interactive": hidden += 1 return matched, deviated, hidden subj = subjects() rate = ERROR_PER_MILLE / 1000 print(f"subjects {len(subj)} | total fields to write by hand " f"{sum(len(fields(o)) for o in subj)} | error per field {rate:.3f}") print() print(f"{'campaign':>9s} {'touches':>8s} {'errors':>6s} {'faulty devices':>14s} " f"{'errors per touch':>17s}") for count in (10, 20, 40): t, e, a = manual(subj, count, SEED + 7) print(f"{count:9d} {t:8d} {e:6d} {a:14d} {e / t:17.3f}") print() p = 1 - (1 - rate) ** 4 print(f"{'procedure':<12s} {'touches':>8s} {'faulty prob':>12s} " f"{'affected devices':>17s} {'expected faulty devices':>24s}") t, e, a = manual(subj, 40, SEED + 7) print(f"{'manual':<12s} {t:8d} {'-':>12s} {a:17d} {t * rate:24.2f}") print(f"{'template':<12s} {1:8d} {p:12.3f} {len(subj):17d} {p * len(subj):24.2f}") print() print(f"{'template class':<15s} {'matched':>7s} {'deviated':>8s} " f"{'hidden from report':>19s}") for c in ("interactive", "batch", "standby"): m, d, h = apply_template(subj, c) print(f"{c:<15s} {m:7d} {d:8d} {h:19d}") ``` ``` subjects 40 | total fields to write by hand 122 | error per field 0.080 campaign touches errors faulty devices errors per touch 10 30 3 3 0.100 20 60 5 5 0.083 40 122 12 10 0.098 procedure touches faulty prob affected devices expected faulty devices manual 122 - 10 9.76 template 1 0.284 40 11.34 template class matched deviated hidden from report interactive 34 6 6 batch 34 6 3 standby 34 6 3 ``` #### What Manual Configuration Costs The top table gives the claim directly. As the campaign grows from ten devices to forty, touches climb from **30** to **122**, and errors from **3** to **12**. The right-hand column, though, stays nearly constant: errors per touch of **0.100**, **0.083**, and **0.098**. This constancy is the measurement's real result — error is not a skill problem, it is a **multiplication**. When the operator is more careful, the rate falls; when device count rises, the multiplier grows, and no indicator says how much attention has dropped. The count of faulty devices is smaller than the count of errors: **12** errors spread across **10** devices, so two devices carry more than one wrong field. This matters in operations, because the unit of repair is the device, not the field; you go back to ten devices, not twelve. None of the three rows reads zero. Shrinking the campaign does not zero out error, only shrinks it — and the cost is splitting the same change into more pieces. #### What Automation Costs The middle table puts the two procedures side by side, and they meet at an unexpected place. Manual configuration leaves **10** faulty devices across **122** touches; the expected value is **9.76**. The template brings touches down to **1**, but the template's probability of coming out faulty is **0.284**, and if the template is faulty, the affected devices number **40**. Expected faulty devices: **11.34**. So automation did not lower the expected error count — in this setup, it raised it slightly. What changed is the **distribution**. Manual configuration produces around ten faulty devices every time. The template produces **zero** most of the time and **forty** the rest. Same average, different operation: the first is a steady, bearable repair load; the second is rare but covers the entire network when it happens. So the case for automation is not "fewer errors." The case rests on three things: error sits in **one place** and can be reviewed there; the change can be tested **before** it is applied; and it can be rolled back **the same way**. Manual configuration has none of the three, because there is no written form of the intent. #### The Tail Automation Does Not Remove The bottom table carries this lesson's real result, unchanged for the rest of the topic. Whichever class the template is written for, matched devices are **34** and deviated devices are **6**. Nine of the forty devices are special, and the six that fall outside the three matching the template's class are outside the lever every time. The deviated count does not change with which class the lever targets — the deviation does not come from the template being written wrong, it comes from **a single template claiming to represent forty different devices**. The right-hand column, though, does change: deviations hidden from the report are **6**, **3**, and **3**. In the template written for the interactive class, every deviation is in another class, and **none** appear in the report; in the other two templates, half of the six are visible. The tail's **size** is constant; what is **visible** is not. This distinction is the topic's axis, paid off in full in the fourth lesson. In a set of forty subjects, the smallest measurable difference is 1/40 = 0.025. The gap between six and three is one hundred twenty times that, comfortably inside the measurement band. #### The Limit of the Inventory The three tables above share the same silent assumption: all forty of the forty devices are known. The measurement runs over forty subjects because the subject set **is** the setup; in operations, this set comes from an **inventory**, and the inventory itself is written either by hand or through a discovery procedure. This concerns both procedures: a device absent from the inventory is touched by neither the operator nor the template. In manual configuration, this device at least has a chance of being found by accident one day. In a network managed by templates, the device never appears in any report, application summary, or deviation list, because all of these documents are produced by walking the inventory. **The lever does not reach beyond the inventory, and because it falls outside the inventory, it does not tell anyone that it does.** This is the lever's tail in its second form. The first was the six deviated devices measured above: a known minority that does not fit the template. The second is a minority not known at all; in this lesson's setup its count is zero, because we wrote the inventory ourselves. In a real network that count is not zero, and measuring it means comparing the inventory against an independent discovery — asking what one lever does by using another lever. #### Irreversibility and Three Protections Every bulk change in this topic is an **irreversible operation**: a template applied to forty devices erases the previous state on all forty the moment it is applied. A change that empties a device's configuration, shuts down its interface, or cuts off management access can split the network in two, and the other half may become unreachable for repair. This course does not write such a command in runnable form; what is written is what it does and that it is irreversible. Three protections exist against this, used together. **Dry run** computes the change's outcome without applying it: it lists in advance which field will change on which device. It is the only procedure that can show the six deviated devices above even where the report cannot, because a dry run compares intent against the device's actual state, not against a report. **Staged rollout** applies the change in batches instead of all forty at once. The goal is not to prevent error, it is to limit how many devices an error affects. **Rollback** keeps an applicable copy of the previous state. Without a copy there is no rollback; rollback is therefore not an operation, it is a decision made **before** the change. Credentials, keys, and access tokens are not written in any example in this topic; where access to a device comes from is a separate operational decision, and it does not live inside the configuration text. #### Summary - The unit of manual configuration is not the device, it is the field; forty devices require **122** field touches, and error grows with touch count — **3/5/12** errors across **30/60/122** touches. - Errors per touch stay nearly constant at **0.100 / 0.083 / 0.098**; error is not a skill problem, it is a multiplication by touch count. - The template brings touches down to **1** but spreads the error: expected faulty devices rises from **9.76** to **11.34**. What changes is not the average, it is the distribution — **0** most of the time, **40** the rest. - The case for automation is not fewer errors; it is that intent sits in one place, can be tested before it is applied, and can be rolled back the same way. - Whichever class the template targets, matched devices are **34** and deviated devices are **6**; the only thing that changes is deviations hidden from the report being **6, 3,** and **3**. #### Next Step This lesson's template rested on an assumption: that forty devices' configuration can be described with the same fields. The measurement showed this does not hold for six of them, and the reason was the last row in the field list — the local rule a special device wants, a field no template carries. The next lesson follows that field: when device configuration is defined by a data model, what can the model represent, which devices do not fit it, and where does the field outside the model go. ### Configuration Models Source: https://academia.sh/en/courses/network-operations/configuration-models When device configuration is defined by a data model, as the model widens, covered fields climb from 80 to 113 and partially represented devices fall from 30 to 9; the remaining nine fields enter no model at all, because their value source is the device itself. The previous lesson built its template on an assumption: that forty devices' configuration can be described with the same fields. The measurement also showed where that assumption did not hold — the last row in the field list, the local rule a special device wanted, and no template carried it. Where that row went was left open. This lesson's question is exactly that. When a device's configuration stops being text and is defined by a **data model**, what can the model represent? Which device does not fit it? And does the field left outside the model disappear, or does it merely vanish from view? What is measured is not what the model covers, it is **what it does not**. #### A Model Is a Contract Manual configuration's output is text, and the text's meaning lives inside the device that reads it. Two different management tools can interpret the same text two different ways; there is nowhere to say which one is right. A **configuration model** closes this gap. The model declares, in a place independent of any device, what fields configuration consists of, each field's type, under what condition it is required, and what range of values it accepts. What declarative configuration calls the "desired end state" can only be written with such a model; without a model, there is no grammar in which to write the desired state at all. The model does three jobs. **Validation:** a configuration that does not conform to the model is rejected before it reaches the device. **Comparison:** two configurations can be compared field by field, because both are written with the same field set — configuration drift can only be measured through this comparison. **Generation:** the same model can generate different values for different devices. ```text # taught model schema, not run device: interface : text , required , shared value address : address , required , per-device priority : number 0-7 , required if class interactive , shared value scheduler : number (ms) , required above latency thresh. , per-device aggregation : list , required above capacity thresh., shared value # local_rule : has no counterpart in the model ``` The schema's last row sits inside a comment, and that is what this lesson measures. #### The Value's Source: Three Classes A field's presence in the model does not say where that field's value comes from. Three sources can be distinguished, and the distinction sets the model's boundary. **Shared value.** The field is the same across every device. The template carries the value once and writes it to forty devices. This is where the lever works most efficiently. **Per-device value.** The field exists in the model, but its value derives from the device's own attribute — the address comes from the inventory, the scheduler is computed from latency. Here the template carries not a value but a **rule**; the rule runs forty separate times. The lever is still singular, its output is forty separate values. **Unrepresentable field.** The field has no counterpart in the model and cannot have one. Adding a rule that belongs to only one device would push the model into forty separate shapes; at that point the model would stop being a contract and become the sum of forty separate configurations. A field left outside the model is not a deficiency, it is **the consequence of the model's own definition**. The assumptions the measurement rests on: - **AC7** — The forty-device subject set is the course's constant and is not changed; field derivation is the same rule as the previous lesson. - **AC8** — A model is a set of field names. A device is **fully represented** only if every field it requires is present in the model; if a single field is missing, it is **partially represented**. - **AC9** — The three model widths are nested: each adds fields on top of the previous one, none removes a field. The width order is core, extended, full. - **AC10** — `local_rule` is in no model. This is not a flaw but a decision of the setup; its rationale is the third class above. - **AC11** — Every field has a single value source: shared, per-device, or unrepresentable. The source depends on the field's name, not on the device. - **AC12** — The template is written for a device class, and the report is built on the interactive class; a special device whose class does not match deviates, and a deviation in another class never appears in the report. #### The Measurement ```python """Configuration model: the device the model cannot represent. Part 1 - three model widths: covered fields, uncovered fields, and devices. Part 2 - a field's value source: shared, per-device, unrepresentable. Part 3 - the tail: what falls outside the template and what the report cannot see. """ SEED = 20260812 SOURCE = {"interface": "shared", "address": "per-device", "priority": "shared", "scheduler": "per-device", "aggregation": "shared", "local_rule": "unrepresentable"} MODELS = { "core": ("interface", "address"), "extended": ("interface", "address", "priority", "scheduler"), "full": ("interface", "address", "priority", "scheduler", "aggregation"), } def make_rng(seed): d = seed % 2147483646 + 1 def r(n): nonlocal d d = (d * 48271) % 2147483647 return d % n return r def subjects(count=40, seed=SEED): r, out = make_rng(seed), [] for i in range(count): out.append({"no": i + 1, "capacity": 20 + r(81), "latency": 5 + r(45), "class": ("interactive", "batch", "standby")[r(3)], "special": r(9) == 0}) return out def fields(o): """The fields a device's configuration requires.""" f = ["interface", "address"] if o["class"] == "interactive": f.append("priority") if o["latency"] > 30: f.append("scheduler") if o["capacity"] >= 80: f.append("aggregation") if o["special"]: f.append("local_rule") return f def measure(subj, model): """Fields and devices the model covers and leaves outside.""" covered = uncovered = full = partial = 0 for o in subj: d = [a for a in fields(o) if a not in model] covered += len(fields(o)) - len(d) uncovered += len(d) if d: partial += 1 else: full += 1 return covered, uncovered, full, partial def apply_template(subj, template_class): """A single template is applied to forty devices; special devices deviate.""" matched = deviated = hidden = 0 for o in subj: if o["class"] == template_class or not o["special"]: matched += 1 else: deviated += 1 if o["class"] != "interactive": hidden += 1 return matched, deviated, hidden subj = subjects() total = sum(len(fields(o)) for o in subj) print(f"subjects {len(subj)} | total field instances {total} | distinct field names " f"{len(SOURCE)} | special devices {sum(o['special'] for o in subj)}") print() print(f"{'model':<10s} {'field names':>11s} {'covered':>9s} {'uncovered':>9s} " f"{'full rep.':>9s} {'partial rep.':>12s}") for name, model in MODELS.items(): c, u, f, p = measure(subj, model) print(f"{name:<10s} {len(model):11d} {c:9d} {u:9d} {f:9d} {p:12d}") print() print(f"{'field':<12s} {'devices':>7s} {'value source':>16s} " f"{'first covering model':>21s}") for field, source in SOURCE.items(): count = sum(1 for o in subj if field in fields(o)) first = next((name for name, m in MODELS.items() if field in m), "none") print(f"{field:<12s} {count:7d} {source:>16s} {first:>21s}") print() for source in ("shared", "per-device", "unrepresentable"): count = sum(1 for o in subj for a in fields(o) if SOURCE[a] == source) print(f"{source:<17s} field instances {count:4d} {count / total:.3f} of total") print() print(f"{'template class':<15s} {'matched':>7s} {'deviated':>8s} " f"{'hidden from report':>19s}") for c in ("interactive", "batch", "standby"): m, d, h = apply_template(subj, c) print(f"{c:<15s} {m:7d} {d:8d} {h:19d}") ``` ``` subjects 40 | total field instances 122 | distinct field names 6 | special devices 9 model field names covered uncovered full rep. partial rep. core 2 80 42 10 30 extended 4 105 17 23 17 full 5 113 9 31 9 field devices value source first covering model interface 40 shared core address 40 per-device core priority 11 shared extended scheduler 14 per-device extended aggregation 8 shared full local_rule 9 unrepresentable none shared field instances 59 0.484 of total per-device field instances 54 0.443 of total unrepresentable field instances 9 0.074 of total template class matched deviated hidden from report interactive 34 6 6 batch 34 6 3 standby 34 6 3 ``` #### What the Model Can Close The top table gives what widening the model buys. The two-field **core** model fully represents only **10** of the forty devices; at least one field of **30** devices falls outside it, and the uncovered field instances number **42**. When the model grows to four fields, uncovered fields drop to **17**, and partially represented devices to **17**. In the five-field **full** model, uncovered fields are **9**, partially represented devices **9**. The shape of the gain is instructive. Going from two fields to four brings uncovered fields down from **42** to **17** — a gain of twenty-five fields for two fields added. Going from four to five buys only **8** fields. Each new field concerns fewer devices than the last, because fields are not distributed evenly: interface and address are needed on all forty devices, scheduler on **14**, priority on **11**, aggregation on **8**. This gives the practical rule of model design. The decision to widen a model is not made by field count, it is made by **how many devices that field removes from partial representation** — and this number shrinks at every step. #### What the Model Cannot Close In the last row, uncovered fields hold at **9**, and widening the model further cannot bring this number down. The middle table gives the reason: the `local_rule` field's first covering model is **none**. These nine field instances belong to the forty devices' **9** special devices, and each is meaningful only to its own device. The bottom table shows the same thing converted into shares. Of the one hundred twenty-two field instances, **59** carry a shared value — **0.484** of the total. **54** are per-device, fields defined in the model but with a value computed separately forty times; their share is **0.443**. The remaining **9** field instances, **0.074** of the total, enter no model at all. The sum of these three shares describes what the model actually is. The portion where the lever truly carries a single value does not even reach half the total. The second share is where the lever carries a rule and produces forty separate results — the model works here, but without the simplicity of writing one value to forty devices. The third share is where the lever does not reach at all. The bottom table's last rows give this the template side's counterpart: whichever class the template targets, matched devices are **34**, deviated **6**. All six deviated devices are special, meaning all carry `local_rule`. But what deserves attention is the others: **three** of the nine special devices match the template's class every time and **accept** the template — even while carrying a field the model cannot represent. Deviations hidden from the report are again **6, 3,** and **3**. #### The Model's Two-Way Limit The measurement counts one direction of the model's limit: a field the device wants is missing from the model. There is a second direction, and it is not measured in this setup — **the device does not accept a field the model declares.** The model defines five fields, the device recognizes four and rejects the fifth; or it recognizes the field but does not accept the full range the model permits. The two directions do not produce the same outcome. The gap in the first direction is clear at **write time**: while the model is being written, the field's absence is seen, and the decision is made deliberately. The gap in the second direction surfaces at **apply time**, and not on all forty devices at once — the devices that accept the field vary, the ones that do not stay fixed. The result is one of the error modes counted in the previous lesson: order-dependent partial application. Part of the network is left with the new configuration, part with the old, and the two behave inconsistently. This is why validating a model is two-staged. The first stage checks whether the model is internally consistent, and never reaches the device. The second stage asks whether the model is applicable on a specific device, and this can only be learned by asking the device. Doing the second stage before applying the change is what later lessons measure as **dry run**; the model's own validation does not stand in for it. #### Where a Field Outside the Model Goes A field outside the model does not vanish; it keeps sitting on the device. Two policies exist, and both open a surface. **Hands-off policy.** The management tool never touches a field the model does not recognize. The device's configuration splits in two: the part managed by the model and the part managed by hand. In this setup, what stays manual is **9** field instances. Configuration drift lives exactly in this region, because comparison can only be done on fields the model knows — the model cannot see a field it does not know has changed. **Free-form field policy.** The model keeps a field with no structure to carry configuration it does not recognize, and writes its content to the device as-is. The field is now carried but **not validated**: type checking, range checking, and comparison do not operate on it. Nine field instances appear inside the model and receive none of its guarantees. The counted surfaces and their narrowing follow. **First surface — unvalidated configuration text.** A free-form field is a path that can reach forty devices without passing through validation; an error in its content catches on none of the model's checks. **Narrowing:** the free-form field is named per device and made valid only on that device; its content becomes part of the device's inventory record, not the template's. This way, an error in one place stays on one device. **Second surface — the region the model cannot see.** Under the hands-off policy, part of the device never enters any comparison, and configuration drift in that region is silent. **Narrowing:** the unmanaged region's boundary is declared explicitly in the model, and that region's existence is turned into a line in the report — a line reading "this device has this many fields outside management" does not show the drift itself, but shows **where it can be**. **Third surface — secret information entering the model.** Credentials, keys, and access tokens are not carried inside the configuration model. The model sits in a repository, gets compared, copied, and reviewed; none of these operations are appropriate for a secret value. **Narrowing:** the model carries only a **reference**, and the value itself sits somewhere else; where that somewhere is, is a separate operational decision and not the subject of this course. #### Summary - A configuration model declares fields' names, types, and requirements independent of any device; validation, comparison, and generation can only be done through this declaration. - As the model widens, covered fields go **80 → 105 → 113**, and partially represented devices **30 → 17 → 9**; each new field's gain is smaller than the last. - Fields' value sources split into three: shared **59** instances (**0.484**), per-device **54** instances (**0.443**), unrepresentable **9** instances (**0.074**). - Nine fields enter no model, because their value is meaningful only to their own device; widening the model that far would turn it into the sum of forty separate configurations. - On the template side, matched devices are **34**, deviated **6**, hidden from the report **6 / 3 / 3**; three of the nine special devices accept the template and keep carrying a field the model cannot represent. #### Next Step This lesson measured what the model can say, and assumed the model stays where it is written. But for a model to be of any use, it has to reach the device: a request goes out from somewhere, the device receives it, applies it, and returns a response. This path has its own rules, and the most important one is this — what does sending the same request twice do? The next lesson measures the programmable interfaces that provide access to configuration, and what a device being **idempotent** or not changes across forty devices. ### Programmable Interfaces Source: https://academia.sh/en/courses/network-operations/programmable-interfaces When the same change is applied through three request forms, the append form changes state on a second application in all forty of forty devices; when nine devices with no response are retried, append breaks five of them, the two idempotent forms break neither, but full replace erases nine devices' out-of-model field. The previous lesson measured what a configuration model can say, and assumed the model stays where it is written. For a model to be of any use, it has to reach the device: a request goes out from somewhere, the device receives it, applies it, and returns a response. This path has its own rules. The most important one comes down to a single question: **what does sending the same request twice do?** The question looks theoretical, but it grows out of operations' most common situation — the response never arrives, and the operator cannot know whether the request was applied. What this lesson measures is not the interface's speed or elegance, it is the answer to this question. #### What the Interface Provides The oldest way to send configuration to a device is opening a shell session and typing commands. This path has three flaws from automation's point of view, and all three come from the same root: **the output is formatted for a human.** A script can only learn whether a command succeeded by parsing the output text. If the text's format changes, the script silently reads it wrong. What comes back on failure is not an error code, it is a sentence. And there is no concept of multiple commands applying together — if the third command is rejected, the first two are already in effect. A **programmable interface** closes all three at once. Both the request and the response are structured according to the previous lesson's model; fields arrive by name, and there is no text to parse. An error returns as a processable structure: it states which field violated which constraint. And the change can be wrapped in a **transaction boundary** — the request is first accumulated as a **candidate configuration**, validated, then committed in a single step. ```text # taught request/response transcript, not run request target : device/34 op : merge body : { priority: 3 , rule: [ base , template_rule ] } response status : accepted changed : [ priority , rule ] txn : 7f ``` The response's `changed` list is the interface's most useful field: it tells you what actually changed on the device. If the list returns empty, the request was applied but changed nothing — the device was already in the desired state. #### Three Request Forms The same change can be requested in three separate forms, and the three do not leave the same result on the device. **Append.** The request says "add this rule to the list." The device puts one more row in the list. Sent twice, the list gets two rows. **Merge.** The request says "these fields have this value." The device matches fields by key and overwrites them; it does not repeat the same entry in a list. Sent twice, the second does nothing. **Full replace.** The request gives the entire configuration. The device discards what it has and sets up exactly what the request says. Sent twice, the second does nothing — but the first sending has also erased every field not present in the request. This is what makes an interface **idempotent**: if applying the request once and applying it twice leave the same state, the interface is idempotent. Idempotence is not a property of speed or correctness; it is a property of **retryability**. #### Response Loss and Retry Why idempotence is an operational problem shows up in a single setup. The request is sent, the device receives and applies it, but the response is lost on the way back. The operator has no information: the request may never have reached the device, or it may have been applied. The two are indistinguishable from outside. In this situation, the only thing to do is retry. On an idempotent interface, retrying is free — if the request was applied, the second one does nothing, and if it was not, it applies it. On an interface that is not idempotent, retrying **breaks the devices that already succeeded**. And what breaks are exactly the devices where the request worked. The assumptions the measurement rests on: - **AC13** — The forty-device subject set is the course's constant. Each device's state before the change is derived from its attributes: interface, address, and a base rule; on a special device, additionally a local rule the model does not carry. - **AC14** — The three request forms carry the **same** change: adding a rule and setting priority. The difference is only in the request's form, not its content. - **AC15** — The idempotence criterion is a comparison of states: if applying the request once and applying it twice leave the same state, the interface is idempotent. The measurement does not look at the interface's own claim. - **AC16** — Response loss is the setup's parameter: one in five devices gets no response, and roughly half of those with no response have actually had the request applied. This is not a measurement; if the rate changes, every row scales in the same direction. - **AC17** — The retry sends the same request again to every device that got no response. The operator cannot know which ones were applied; if they could, there would be no need to retry. - **AC18** — The template is written for a device class, and the report is built on the interactive class; a special device whose class does not match deviates, and a deviation in another class never appears in the report. #### The Measurement ```python """Programmable interface: what sending the same request twice does. Part 1 - three request forms, the difference a second application leaves. Part 2 - when the response is lost, what a retry breaks. Part 3 - the tail: what falls outside the template and what the report cannot see. """ SEED = 20260812 EXTRA = "template_rule" FORMS = ("append", "merge", "full replace") def make_rng(seed): d = seed % 2147483646 + 1 def r(n): nonlocal d d = (d * 48271) % 2147483647 return d % n return r def subjects(count=40, seed=SEED): r, out = make_rng(seed), [] for i in range(count): out.append({"no": i + 1, "capacity": 20 + r(81), "latency": 5 + r(45), "class": ("interactive", "batch", "standby")[r(3)], "special": r(9) == 0}) return out def state(o): """The device's configuration before the change.""" d = {"interface": "main", "address": f"a{o['no']}", "rule": ["base"]} if o["special"]: d["local_rule"] = f"k{o['no']}" return d def request(d, form): """The same change is applied through three separate request forms.""" y = {k: (list(v) if isinstance(v, list) else v) for k, v in d.items()} if form == "append": y["rule"] = y["rule"] + [EXTRA] elif form == "merge": y["rule"] = [k for k in y["rule"] if k != EXTRA] + [EXTRA] elif form == "full replace": y = {"interface": "main", "address": d["address"], "rule": ["base", EXTRA]} y["priority"] = "3" return y def apply_template(subj, template_class): """A single template is applied to forty devices; special devices deviate.""" matched = deviated = hidden = 0 for o in subj: if o["class"] == template_class or not o["special"]: matched += 1 else: deviated += 1 if o["class"] != "interactive": hidden += 1 return matched, deviated, hidden subj = subjects() print(f"subjects {len(subj)} | devices carrying an out-of-model field " f"{sum(o['special'] for o in subj)} | request forms {len(FORMS)}") print() print(f"{'request form':<14s} {'second application changes':>27s} {'idempotent':>10s} " f"{'out-of-model field erased':>26s}") for b in FORMS: changed = erased = 0 for o in subj: d = state(o) first = request(d, b) second = request(first, b) if second != first: changed += 1 if "local_rule" in d and "local_rule" not in first: erased += 1 print(f"{b:<14s} {changed:27d} {'no' if changed else 'yes':>10s} " f"{erased:26d}") print() r = make_rng(SEED + 11) no_response, actually_applied = [], set() for o in subj: if r(5) == 0: no_response.append(o["no"]) if r(2) == 0: actually_applied.add(o["no"]) print(f"devices with no response {len(no_response)} | of these, request actually " f"applied {len(actually_applied)}") print() print(f"{'request form':<14s} {'retried':>7s} {'reached correct state':>22s} " f"{'broken':>7s}") for b in FORMS: correct = broken = 0 for no in no_response: o = subj[no - 1] d = state(o) target = request(d, b) before = target if no in actually_applied else d after = request(before, b) if after == target: correct += 1 else: broken += 1 print(f"{b:<14s} {len(no_response):7d} {correct:22d} {broken:7d}") print() print(f"{'template class':<15s} {'matched':>7s} {'deviated':>8s} " f"{'hidden from report':>19s} {'accepted template, carries out-of-model field':>47s}") for c in ("interactive", "batch", "standby"): m, d, h = apply_template(subj, c) risky = sum(1 for o in subj if o["special"] and o["class"] == c) print(f"{c:<15s} {m:7d} {d:8d} {h:19d} {risky:47d}") ``` ``` subjects 40 | devices carrying an out-of-model field 9 | request forms 3 request form second application changes idempotent out-of-model field erased append 40 no 0 merge 0 yes 0 full replace 0 yes 9 devices with no response 9 | of these, request actually applied 5 request form retried reached correct state broken append 9 4 5 merge 9 9 0 full replace 9 9 0 template class matched deviated hidden from report accepted template, carries out-of-model field interactive 34 6 6 3 batch 34 6 3 3 standby 34 6 3 3 ``` #### The Measure of Idempotence The top table separates the three forms. In the **append** form, the second application changes state in all forty of forty devices; the interface is not idempotent. In the **merge** and **full replace** forms, the second application changes nothing on any device; both are idempotent. All forty of forty is notable. A violation of idempotence is not a minority event: in the append form, **every** device breaks on the second application, because the cause of breaking is not a device attribute, it is the request's form. This is the opposite of the other tails measured in this course — here there is no tail, the lever misserves all forty of forty the same way. The right-hand column gives the second distinction, and shows that idempotence alone is not enough. **Full replace** is idempotent, but it erases **9** devices' out-of-model field. Since the request does not carry that field, the device treats it not as missing but as **extra**, and discards it. The previous lesson's nine `local_rule` fields vanish here — and the interface reports no fault at all, because this is exactly what the requested operation is defined to do. So idempotence and safety are two separate properties. An interface that is not idempotent breaks on retry; an interface that is idempotent but uses full replace erases on the **first** attempt. The form standing at the intersection of the two is **merge**: it is both retryable and preserves the fields it does not touch. #### What Retrying Costs The middle table runs the setup. **9** of the forty devices get no response, and **5** of those actually had the request applied. The operator has no information to make this distinction; all nine of the nine have to be retried. In the append form, devices reaching the correct state after retry are **4**, broken **5**. The five broken devices are exactly the ones where the request worked the first time. The shape of this result is the worst possible from an operational standpoint: the procedure does damage exactly where it succeeded, and the damage scales with the success. In the merge and full replace forms, all nine of the nine devices reach the correct state, broken **0**. This is exactly what idempotence buys, and nothing more: it makes retrying **safe to do without thinking**. In a set of forty subjects, the smallest measurable difference is 1/40 = 0.025. Five broken devices are a share of **0.125**, comfortably inside the band. #### What Idempotence Does Not Protect Against Idempotence protects the operator against **their own** repetition. There is a situation it does not protect against, and it arises from the interface's read side. A script's typical loop is three steps: read the device's state, compute the change on top of it, write the result. Reading is idempotent, writing can be idempotent; but the three together are not a single unit. If two operators run this loop on the same device at the same time, the second writes after the first has read, and overwrites the first's write without ever seeing it. The result is not the sum of two changes, it is the second one's result alone; and no request returns an error, because each request is valid on its own. Idempotence does nothing here. Sending the same request twice is not the problem; the problem is **two different requests interleaving**. The interface has two tools against this. The first is the transaction boundary: while the change accumulates in the candidate configuration, the device holds a lock, and the second operator's write is rejected. The second is a **stamp**: every configuration gets a stamp, the request says "I saw this stamp, I am writing on top of it," and if the stamp has changed, the request is rejected. Neither solves the problem, both **make it visible** — a rejected request is not an error, it is a notification. #### Surfaces the Interface Opens and Their Narrowing A programmable interface is a path that can change a device's configuration; by definition, this makes it a surface. The counted surfaces and their narrowing follow. **First surface — the management path itself.** If the interface is open, the device's configuration can be changed over the network. **Narrowing:** the interface connects to a separate management network and is unreachable from the path data traffic takes; access is restricted by source address; read authority and write authority are granted separately. These three narrowings do not substitute for one another, they stack. **Second surface — where the credential sits.** A script uses an identity to connect to the device, and that identity sits somewhere. Credentials, keys, and access tokens are not written in any example in this course. **Narrowing:** the value does not sit inside the script, the template, or the configuration repository; the script carries only a reference and gets the value from its runtime environment. **Third surface — the field full replace erases.** In the measurement, **9** devices lose a field the request does not carry. This is not an attack, it is the interface's defined behavior, and that is exactly why it is dangerous — it produces no warning. **Narrowing:** full replace is used only when the entire configuration is genuinely known; when it is not, merge is used. Devices carrying an out-of-model field are listed in advance, and the operation is halted when full replace's scope intersects that list. **Fourth surface — a change that cuts off its own access.** A request that shuts down the management interface itself, changes the management network's address, or filters access makes the device unreachable the moment it is applied. Such a request's command is not written in runnable form in this course; what can be said is that it is irreversible. **Narrowing:** the change is first written as a candidate configuration, its outcome is computed with a dry run, then it is committed with a **rollback timer** — if the operator does not confirm within a set period, the device automatically reverts to the previous configuration. The timer is the only tool left in the hands of an operator who has lost access. #### Summary - A programmable interface provides structured requests and responses, processable errors, and a transaction boundary; it closes all three flaws of a script parsing shell output at once. - The append form is not idempotent: a second application changes state in **40** of forty devices. Merge and full replace are idempotent, **0** in both. - **5** of the **9** devices with no response actually had the request applied; on retry, the append form breaks these **5**, the two idempotent forms break **0**. - Idempotence and safety are separate: full replace is idempotent but erases **9** devices' out-of-model field on the first application, and reports no fault. - On the template side, matched devices are **34**, deviated **6**, hidden from the report **6 / 3 / 3**; **3** of the devices accepting the template carry an out-of-model field, and they are exactly the ones that would lose it under full replace. #### Next Step This lesson measured what a single request leaves on a single device. In operations, a request is never singular: a script connects to forty devices in sequence, applies the same template to each, and writes a report at the end. The next lesson measures that script itself — a bulk change precomputed with a dry run, spread in batches, and rolled back. And it is there that we answer the real question: when the template is applied, how many devices fall outside it, and how many of those devices appear in the report that gets written. ### Managing Devices with Scripts Source: https://academia.sh/en/courses/network-operations/managing-devices-with-scripts Whichever class the template targets, deviated devices are 6 every time; the only thing that changes is the count hidden from the report being 6, 3, and 3. Dry run finds all six before applying, the report finds 0, 3, and 3, and under one template staged rollout never stops. The previous lesson measured what a single request leaves on a single device. In operations, a request is never singular: a script walks the inventory, connects to forty devices in sequence, applies the same template to each, and writes a report at the end. Once written, the script carries the same decision to all forty of forty devices — this is exactly what the lever is. This lesson's question is not how to write the script, it is **what the script cannot see**. When the template is applied, how many devices fall outside it, how many of these devices appear in the report that gets written, and what decision opens the gap between the two numbers. #### What the Script Does A bulk change is seven steps, and the order of the steps determines the measurement's result. ```text # taught bulk-change procedure, not run 1 read the inventory -> 40 devices 2 dry run the template -> per-device diff list 3 review the diff -> flag what falls outside the template 4 apply batch 1 -> first 10 devices 5 read the report -> any deviation visible 6 if not, batch 2, 3, 4 -> rollout continues 7 if so, stop and roll back -> only what was applied ``` The second step is **dry run**, the loop from four through seven is **staged rollout**, the second half of seven is **rollback**. The three are not three separate protections; they are the same protection in three different moments — before applying, while applying, and after applying. The infrastructure-as-code procedure and tool discussion is the subject of the Infrastructure as Code course and are not repeated here. Here the object is a network device, and what is measured is not the procedure's efficiency, it is **the minority that falls outside the template.** #### Two Sources of Verification There are two ways for a script to know a change is going right, and the two do not measure the same thing. **Dry run compares intent against the device's state.** It computes what the template would write, reads the current configuration from the device, takes the difference between the two, and **writes to no device**. Its coverage is the entire inventory: it looks at all forty of forty devices, because looking requires no indicator. **The report comes from indicators.** Metrics collected after applying are read, and devices deviating from expectation are listed. Its coverage is the **indicators' coverage** — and the indicator set never sees all forty of forty devices at equal detail. In this setup, the report is built on the interactive class's indicators; a deviation in another class never appears in it. This is not a flaw in whoever wrote the report. An indicator set is always a coverage decision: which class's which metric, collected at what frequency. The question asked when making that decision is "which devices matter"; the question not asked is "what deviation are we becoming unable to see." The assumptions the measurement rests on: - **AC19** — The forty-device subject set is the course's constant and is not changed in this lesson. - **AC20** — The template is written for a device class. A special device whose class does not match deviates; the deviation arises from the template's scope decision, not from a device fault. - **AC21** — Dry run compares intent against the device's actual state and writes to no device. It does not use the oracle; it reads deviation from the device's own state. - **AC22** — The report is built on the interactive class's indicators. A deviation in another class never appears in it. - **AC23** — Staged rollout batches devices in number order. The report is read at the end of every batch; if a deviation visible in the report exists, rollout stops. - **AC24** — When rollout stops, deviated devices applied up to that point are rolled back; deviated devices in later batches were never touched at all. - **AC25** — When batch size is 40, rollout is a single step, and there is no interval at which a stopping decision could be made. #### The Measurement ```python """Bulk change: deviated devices constant, visible deviation variable. Part 1 - the template is applied: matched, deviated, visible and hidden in the report. Part 2 - dry run against the report's coverage, side by side. Part 3 - staged rollout: changed and prevented deviations until it stops. """ SEED = 20260812 CLASSES = ("interactive", "batch", "standby") def make_rng(seed): d = seed % 2147483646 + 1 def r(n): nonlocal d d = (d * 48271) % 2147483647 return d % n return r def subjects(count=40, seed=SEED): r, out = make_rng(seed), [] for i in range(count): out.append({"no": i + 1, "capacity": 20 + r(81), "latency": 5 + r(45), "class": ("interactive", "batch", "standby")[r(3)], "special": r(9) == 0}) return out def apply_template(subj, template_class): """A single template is applied to forty devices; special devices deviate.""" matched = deviated = hidden = 0 for o in subj: if o["class"] == template_class or not o["special"]: matched += 1 else: deviated += 1 if o["class"] != "interactive": hidden += 1 return matched, deviated, hidden def dry_run(subj, template_class): """Before applying, intent is compared against the device's state.""" return [o["no"] for o in subj if not (o["class"] == template_class or not o["special"])] def in_report(subj, template_class): """The report is built on the interactive class's indicators.""" return [no for no in dry_run(subj, template_class) if subj[no - 1]["class"] == "interactive"] def staged(subj, template_class, batch): """Applied batch by batch; stops at the first deviation visible in the report.""" applied = deviated_applied = wave = 0 for start in range(0, len(subj), batch): wave += 1 visible = 0 for o in subj[start:start + batch]: applied += 1 if not (o["class"] == template_class or not o["special"]): deviated_applied += 1 if o["class"] == "interactive": visible += 1 if visible: return wave, applied, deviated_applied return 0, applied, deviated_applied subj = subjects() print(f"subjects {len(subj)} | special devices {sum(o['special'] for o in subj)} | " f"class distribution " + " ".join(f"{s} {sum(1 for o in subj if o['class'] == s)}" for s in CLASSES)) print() print(f"{'template class':<14s} {'matched':>5s} {'deviated':>6s} {'visible in report':>16s} " f"{'hidden from report':>19s} {'report coverage':>16s}") for s in CLASSES: m, d, h = apply_template(subj, s) print(f"{s:<14s} {m:5d} {d:6d} {d - h:16d} {h:19d} " f"{(d - h) / d:16.3f}") print() print(f"{'template class':<14s} {'dry run finds':>19s} {'report finds':>12s} " f"{'found only by dry run':>31s}") for s in CLASSES: k, r = dry_run(subj, s), in_report(subj, s) print(f"{s:<14s} {len(k):19d} {len(r):12d} {len(k) - len(r):31d}") for s in CLASSES: print(f" {s:<12s} deviated devices {dry_run(subj, s)} | " f"in report {in_report(subj, s)}") print() print(f"{'template class':<14s} {'batch':>6s} {'stopping wave':>13s} " f"{'devices changed':>15s} {'to be rolled back':>18s} {'deviations prevented':>21s}") for s in CLASSES: for batch in (4, 10, 40): w, a, da = staged(subj, s, batch) print(f"{s:<14s} {batch:6d} {w if w else '-':>13} {a:15d} " f"{da:18d} {len(dry_run(subj, s)) - da:21d}") ``` ``` subjects 40 | special devices 9 | class distribution interactive 11 batch 20 standby 9 template class matched deviated visible in report hidden from report report coverage interactive 34 6 0 6 0.000 batch 34 6 3 3 0.500 standby 34 6 3 3 0.500 template class dry run finds report finds found only by dry run interactive 6 0 6 batch 6 3 3 standby 6 3 3 interactive deviated devices [7, 12, 20, 25, 26, 40] | in report [] batch deviated devices [4, 20, 25, 26, 31, 33] | in report [4, 31, 33] standby deviated devices [4, 7, 12, 31, 33, 40] | in report [4, 31, 33] template class batch stopping wave devices changed to be rolled back deviations prevented interactive 4 - 40 6 0 interactive 10 - 40 6 0 interactive 40 - 40 6 0 batch 4 1 4 1 5 batch 10 1 10 1 5 batch 40 1 40 6 0 standby 4 1 4 1 5 standby 10 1 10 2 4 standby 40 1 40 6 0 ``` #### Deviated Is Constant, Visible Is Variable The top table gives this topic's main result. Whichever class the template is written for, matched devices are **34**, deviated **6**. The same number in all three rows. The deviated count does not depend on which class the template targets, because the source of deviation is not the template being written wrong, it is **a single template claiming to represent forty different devices**. Writing a better template does not bring this six down to five; only narrowing the scope or writing a second template does, and that means giving up the lever's singularity. The three right-hand columns, though, do change. In the template written for the interactive class, deviations visible in the report are **0**, hidden **6**; report coverage **0.000**. In the other two templates, visible is **3**, hidden **3**, coverage **0.500**. The same six devices, three different visibilities. This is the course's third claim paid off a second time, and it stands here in its starkest form: **automation does not reduce deviation, it changes its visibility.** The six deviated devices are there in all three templates; the only thing that changes is how many fall into the report. In operations, only the second of these two numbers gets written to the dashboard, and as the dashboard improves, deviation is assumed to be falling. Device numbers confirm this. In the interactive template, deviations are `[7, 12, 20, 25, 26, 40]`, and the list visible in the report is empty. In the batch template, deviations are `[4, 20, 25, 26, 31, 33]`, in the report `[4, 31, 33]` — three visible, three not. In the standby template, the deviated devices are different, but the ones visible in the report are again `[4, 31, 33]`, because the report looks not at deviation but at **class**. In a set of forty subjects, the smallest measurable difference is 1/40 = 0.025. The gap between six and three is **0.075**, comfortably inside the band. #### What Dry Run Finds The middle table puts the two sources of verification side by side. Dry run finds **6** deviated devices in all three of the three templates. The report finds **0**, **3**, and **3**. The count found only by dry run is, respectively, **6**, **3**, and **3**. The reason for the gap is the method itself. Dry run does not look at an indicator; it asks every device in the inventory one by one and compares it against intent. Its coverage is not the indicator set's coverage, it is the **inventory's** — and the inventory contains all forty of forty devices. This is why dry run is independent of every blind spot in indicator design. The second difference is in **timing**, and it matters more. Dry run finds deviation **before any device changes**; the report finds it after the change is applied. In the first case, the count of devices to roll back is zero, because there is nothing to roll back. What dry run really buys is not the six devices it finds, it is the **moment** at which it finds those six. #### What Staged Rollout Costs The bottom table gives what rollout changes with batch size, and the first three rows carry the topic's harshest result. **Under the interactive template, rollout never stops at any batch size.** The stopping wave column reads empty in all three rows; devices changed are **40** in all three, to be rolled back **6**, deviations prevented **0**. The reason is clear: rollout stops on a deviation visible in the report, and under this template no deviation is visible in the report. Shrinking the batch to four changes nothing, because the problem is not batch size, it is that **the signal the stopping decision reads is empty**. In a blind spot with no signal, staged rollout is the same thing as unstaged rollout. In the other two templates, rollout stops in the first wave. Under the batch template, at batch size 4, devices changed are **4**, to be rolled back **1**, deviations prevented **5**; at batch size 10, changed **10**, to be rolled back again **1**, prevented **5**. Under the standby template, to be rolled back is **1** at batch size 4, **2** at batch size 10 — one more deviated device enters the first wave as the batch grows. The batch-40 rows stand for comparison: applied in a single step, all three templates read changed **40**, to be rolled back **6**, prevented **0**. Staged rollout's entire gain reads from here — with a working signal it prevents **5** deviations, with none, **0**. #### Rollback and a Destructive Change Rollback is not an operation, it is a decision made before the change. If the device's running configuration was not saved before applying, there is no target to roll back to, and the table's "to be rolled back" column stays a wish list. A bulk change is an **irreversible operation**: a template applied to forty devices erases the previous state on all forty the moment it is applied. A change that empties a device's configuration, shuts down its interface, cuts off management access, or splits the network in two is not written in runnable form in this course. What can be said is what it does and that it is irreversible: after a change that splits the network, reaching devices on the other half to fix them may not be possible, and those devices are exactly the ones where rollback needs to run. The counted surfaces and their narrowing follow. **First surface — the scope itself.** If the script can touch the entire inventory, a typo goes to forty devices at once. **Narrowing:** the script's scope is given explicitly on every run, and the default scope is empty; scope is bounded by a batch list; the apply step does not run until the dry run's output has been reviewed. **Second surface — the report's blind spot.** In the measurement, all **6** deviations under one template never fall into the report. **Narrowing:** the stopping decision is not tied to the report alone; at the end of every batch, dry run is re-run for the devices in that batch and compared against intent. This brings the three devices the report cannot see into the stopping decision's scope as well. **Third surface — the credential.** The script connects to forty devices and uses an identity to do it. Credentials, keys, and access tokens are not written in any example in this course; the script carries only a reference and gets the value from its runtime environment, not from inside the template or the configuration repository. **Fourth surface — the irreversible step itself.** **Narrowing:** a step that could be destructive is applied with a **rollback timer** — if the operator does not confirm within a set period, the device automatically reverts to the previous configuration. This is the only tool left in the hands of an operator who has lost access, and the timer has to be set up **before** applying. #### Summary - A bulk change is seven steps; dry run, staged rollout, and rollback are the same protection's before-, during-, and after-application forms. - Whichever class the template targets, matched devices are **34**, deviated **6**; the source of deviation is not the template being wrong, it is a single template claiming to represent forty different devices. - Deviations visible in the report are **0 / 3 / 3**, hidden **6 / 3 / 3**, report coverage **0.000 / 0.500 / 0.500**. The same six devices, three different visibilities: **automation does not reduce deviation, it changes its visibility.** - Dry run finds **6** deviations under all three templates, because its coverage is the inventory's, not an indicator's; and it finds them before any device changes. - Under the interactive template, staged rollout never stops at any batch size — changed **40**, prevented **0**. Under the other two, it stops in the first wave and prevents **5** deviations; in a blind spot with no signal, staged rollout is the same thing as unstaged rollout. #### Next Step Every deviated device counted in this topic was a device whose name was in the inventory, whose configuration we could read, and whose changes we could roll back when needed. Dry run being able to find the six deviations rested on this too: we could ask the device, and the answer it gave was ours. But what if the subject is an object **we do not own**? The next lesson changes the object under the lever, and asks the same measurement — one decision, forty subjects, the tail left over — again, on a configuration plane that is not ours. ### Cloud Networking Concepts Source: https://academia.sh/en/courses/network-operations/cloud-networking-concepts Virtual network, subnet, security group, and gateway all fall under the same lever-tail measure: when a single subnet prefix length is applied to forty subnets, /26 leaves seventeen short, while /25, which closes the tail, does not fit the virtual network's block. The previous lesson applied a single template to forty devices and counted the deviated minority. In every measurement in the course up to this point, the subject was the same kind: **a device we own**. It sat on a rack, had a serial number, could be reached through a shell, and got replaced when it broke. The lever touched this device. This lesson's question changes the subject: what if the subject is an object **we do not own**? In cloud networking, the virtual network, subnet, security group, and gateway are not devices; they are **configuration records**. There is no box to touch, no shell to open, no light to check at the moment of failure. The question is: when the subject turns into a record, does the lever-tail measure change? #### The Subject That Becomes a Record Cloud service models, the shared responsibility boundary, and provider selection were established in the Cloud Computing curriculum's Compute and Networking in the Cloud course; that narrative is not repeated here. This lesson takes up the cloud object from a single angle: **as the measurement's subject**. A subject that turns into a record gains three things. It is created and destroyed instantly; a single request can take its count from forty to four hundred. Its definition is text, so it enters version control and can be copied identically elsewhere. Third, its creation is always the same: the same definition produces the same object every time. What it loses is also three things, and this is the part that concerns the measurement. We cannot see the physical path underneath the object. We do not set the object's limits — how many subnets a virtual network can carry, how many rules a security group can hold, are numbers handed to us. And once an object is created, **some of its fields cannot be changed**: a virtual network's address block cannot be grown in place after it is set up. This third one directly creates the lever. An unchangeable field is a decision made once, for everyone. If all forty subnets sit inside the same virtual network, that virtual network's block is the common limit of all forty subjects. #### Four Objects Cloud networking's core comes down to four classes. The transcript below is not a call; it shows the shape of the objects' declarative definition and is not run. ```text # taught object transcript, not run virtual network: name: measurement-net block: 10.40.0.0/20 # cannot be changed after creation region: region-1 subnet: name: measurement-01 virtual network: measurement-net block: 10.40.0.0/26 # cut from the virtual network's block reserved addresses: 5 # network, broadcast, gateway, and two service addresses route table: measurement-default security group: name: measurement-device ingress: - source: security group measurement-collector port: 443/tcp - source: 10.40.0.0/20 port: 22/tcp egress: - destination: anywhere port: 443/tcp gateway: - type: egress-only # inside goes out, no inbound from outside - type: peering # a direct path to another virtual network peer: management-net ``` A **virtual network** is an address space and an isolation boundary. A **subnet** is a slice of that space and carries its own route table. A **security group** is a rule sequence attached not to a device but to a set of interfaces. A **gateway** is the object that establishes the virtual network's relationship with the outside; its type determines direction. In the security group's first rule, the source is not an address block, it is **another security group**. This is cloud networking's one genuinely original structural contribution: the rule binds to a **label** instead of an address, and rewriting the rule is not needed when the address changes. This lesson takes the security group only as a **configuration object**; the rule sequence's intent-versus-actual gap was measured in the Wireless Networks and Network Security course and is not repeated here. #### Public and Private Subnet The most commonly misunderstood thing about subnets is where their being **public** or **private** comes from. It does not come from the address block. Whatever a subnet's block is, the only thing that makes it reachable from outside is its **route table**: if the table has a default route to an outbound gateway, the subnet is public, if not, private. Two subnets inside the same virtual network can carry neighboring address blocks and have completely different access properties. This makes the route table a second lever, and it produces a different tail from the first. The table attaches not to a subnet but to a **set of subnets**; adding a route to the table adds it to every subnet sharing it at once. The subnets where the route is unnecessary form the tail, and this tail is entirely **silent**: an extra route produces no error, creates no latency, writes no log entry. It only silently widens the set of places that subnet can reach. Which table a subnet is attached to is therefore a decision independent of the address plan, and it has to be counted separately. If it is not known how many of the forty subnets share the same table, it is also not known how many subjects a single route row touches. **Narrowing:** the table is split by access property — outbound subnets share one table, non-outbound share another, and the default route is written only into the first. This lesson's measurement does not touch the second lever; the only thing measured is the address block. The two need to be kept separate, because even though they operate on the same subnet object, they leave different tails: one is failing to fit endpoints, the other is unnecessary access. #### The Lever: A Single Prefix Length Forty subnets are to be opened. How many endpoints each will carry differs from one to the next. The operator is handed a single number: **the subnets' prefix length**. This number is chosen once and applied to all forty of the forty subnets at once. Why a single number? Because choosing forty separate prefixes means forty separate decisions, and every one of those decisions has to be reviewed, documented, and remembered later. A single prefix gives a single decision: which block a subnet falls into is found by arithmetic, the plan fits in one table, route summarization is written in a single row — this is the gain established in the Switching and Routing course's Address and Naming Plan lesson. One detail here departs from the Network Models and Protocols course's subnet arithmetic. There, the count of addresses reserved per subnet was two — the network address and the broadcast address. In a cloud virtual network the count is larger and varies by provider; this measurement takes it as **five**: the gateway and two service addresses also sit in the block and can never be handed to an endpoint. Some providers reserve four instead, taking the first two and last two addresses of the block. In small blocks, this difference is decisive; **five** of the `/27` block's thirty-two addresses are gone from the start. The measurement probes two limits at once. A subnet's block must meet its endpoint demand. And the sum of the forty blocks must fit inside the virtual network's block. The two limits pull in opposite directions: shrinking the prefix solves the first and breaks the second. The assumptions the measurement rests on: - **AC61** — The forty subjects are the course's fixed set; each subject corresponds to a subnet, and the subject's capacity is read as that subnet's **endpoint count**. The set is not changed. - **AC62** — The virtual network's block is `/20`, that is **4096** addresses, and it cannot be grown after creation. - **AC63** — Every subnet has **5** unusable addresses: the network address, the broadcast address, the gateway, and two service addresses. Usable endpoint count is what remains after subtracting these five from the block size. - **AC64** — The oracle is the setup itself: we know every subnet's actual endpoint demand because we wrote it. The plan does not use this information, it only applies the single prefix. - **AC65** — The measurement is not a cloud call; no object is created, no request is sent to any provider. What is counted is address arithmetic. - **AC66** — The comparison row, the per-subject plan, is the absence of the lever: every subnet gets the smallest block that covers its own demand, and the decision count is forty. #### The Measurement ```python """Cloud networking: a single subnet prefix length applied to forty subnets at once. Lever - the single prefix length used across all of the virtual network's subnets. Tail - a subnet whose block does not meet its demand. Invisible - the address sitting idle inside subnets. """ SEED = 20260812 VPC_PREFIX = 20 # the virtual network's address block RESERVED = 5 # addresses unusable in every subnet def make_rng(seed): d = seed % 2147483646 + 1 def r(n): nonlocal d d = (d * 48271) % 2147483647 return d % n return r def subjects(count=40, seed=SEED): r, out = make_rng(seed), [] for i in range(count): out.append({ "no": i + 1, "capacity": 20 + r(81), "latency": 5 + r(45), "class": ("interactive", "batch", "standby")[r(3)], "special": r(9) == 0, }) return out def block_size(prefix): return 1 << (32 - prefix) def single_prefix(subj, prefix): """A single prefix is applied to forty subnets; demand is measured by endpoint count.""" usable = block_size(prefix) - RESERVED short = missing = idle = 0 for o in subj: gap = o["capacity"] - usable if gap > 0: short += 1 missing += gap else: idle -= gap return usable, short, missing, idle, block_size(prefix) * len(subj) def per_subject_prefix(o): """The prefix of the smallest block that covers one subject.""" prefix = 32 while block_size(prefix) - RESERVED < o["capacity"]: prefix -= 1 return prefix subj = subjects() vpc = block_size(VPC_PREFIX) print(f"virtual network /{VPC_PREFIX} = {vpc} addresses | subnets {len(subj)} | " f"endpoint demand {min(o['capacity'] for o in subj)}-{max(o['capacity'] for o in subj)} " f"| total demand {sum(o['capacity'] for o in subj)}") print() print(f"{'prefix':>7s} {'usable':>14s} {'short':>9s} {'missing':>9s} " f"{'idle addresses':>14s} {'addresses needed':>16s} {'fits vpc':>10s}") for prefix in (27, 26, 25, 24): u, s, m, i, g = single_prefix(subj, prefix) print(f" /{prefix:<4d} {u:14d} {s:9d} {m:9d} {i:14d} {g:16d} " f"{'fits' if g <= vpc else 'does not fit':>10s}") total = idle = 0 for o in subj: prefix = per_subject_prefix(o) total += block_size(prefix) idle += block_size(prefix) - RESERVED - o["capacity"] print(f" {'per subject':<12s} {'-':>9s} {0:9d} {0:9d} {idle:14d} {total:16d} " f"{'fits' if total <= vpc else 'does not fit':>10s}") print() spread = {} for o in subj: spread[per_subject_prefix(o)] = spread.get(per_subject_prefix(o), 0) + 1 print("distinct prefixes in the per-subject plan:", len(spread), "-", {f"/{k}": v for k, v in sorted(spread.items())}) ``` ``` virtual network /20 = 4096 addresses | subnets 40 | endpoint demand 22-99 | total demand 2336 prefix usable short missing idle addresses addresses needed fits vpc /27 27 36 1265 9 1280 fits /26 59 17 371 395 2560 fits /25 123 0 0 2584 5120 does not fit /24 251 0 0 7704 10240 does not fit per subject - 0 0 984 3520 fits distinct prefixes in the per-subject plan: 3 - {'/25': 17, '/26': 19, '/27': 4} ``` #### Where the Tail Moved The four rows show four settings of a single lever, and none of them is clean. The `/27` block leaves **36** of forty subnets short; missing endpoints total **1265**. Idle addresses, in contrast, are only **9** — the plan is tight, because it hands almost no subnet extra addresses. The cost of tightness is a tail of thirty-six out of forty. `/26` brings the tail down to **17** and missing endpoints to **371**. Idle addresses climb from **9** to **395**. Growing the lever one step melted the tail by more than half, but seventeen of forty subnets still cannot meet their demand. `/25` brings the tail to **0**. All forty of forty subnets meet their demand, no missing endpoints. And right on this row, the right-hand column takes over: this plan wants **5120** addresses, and the virtual network's block is **4096**. The plan **does not fit** the virtual network. `/24` does the same thing more heavily: **10240** addresses, **7704** idle. The tail did not disappear. **It moved from the subnet to the virtual network.** At `/26`, seventeen subnets fell short; at `/25`, no subnet falls short, but the virtual network itself falls short — and per **AC62**, the virtual network's block cannot be grown. The tail's new name is no longer "seventeen subjects," it is "the entire deployment." Growing the lever did not remove the tail, it changed **who is in it**. #### The Idle Address An indicator that counts only subnets that do not fit reads the `/25` row as **flawless**: zero tail. On that row, **2584** addresses sit unassigned to any endpoint — half of the **5120** addresses the plan wants. The same indicator reads the `/27` row as the worst, even though idle addresses there are only **9**. The two numbers exist at once, and a single indicator cannot see both. This is the fourth reading counted across the course: **an indicator that counts overflow does not see the gap.** In address planning, the gap is especially costly precisely because it is invisible — it produces no warning, writes no log entry, and is only noticed when a new subnet can no longer be added to the virtual network. The last row states what the lever itself costs. When every subnet is given its own prefix, the tail is **0**, idle addresses **984**, addresses needed **3520**, and the plan fits the virtual network. So a correct answer exists; it cannot be expressed with a single number. Its cost is written in the last row: the plan carries **three** separate prefixes, requires **forty** separate decisions, and breaks single-row route summarization. The lever's tail exists for exactly this reason: because the cost of forty decisions was judged greater than the cost of the tail. #### Security Group and Gateway The remaining two objects are two more faces of the same pattern, and they are not measured in this lesson; each gives rise to its own tail. **A security group** is a rule sequence, and a single group attaches to many interfaces. Widening one of the group's rules widens it on every interface it is attached to at once. **Narrowing:** binding the rule's source to another security group instead of an address restrains this widening: the rule is now bound to a **membership**, not an address range, and the membership list updates together with object lifecycle. How many interfaces the group is attached to still has to be counted separately; without that count, how many subjects a rule touches is unknown. **A gateway** is the object that establishes the virtual network's relationship with the outside, and its type determines direction: some gateways permit only inside-to-outside traffic, some open a two-way path, and some match only a specific peer network. Direction here is a **security** property, not a performance one. Because an egress-only gateway is one-directional, no inbound connection exists; creating a two-way gateway removes this restriction, and the moment it is removed, it is removed for every subnet in the virtual network. **Narrowing:** the gateway is written only into the route table of the subnets that need it — even if the gateway exists at the virtual-network level, a subnet with no route to it cannot pass through it. #### Summary - Cloud networking's objects are configuration records, not devices; they are created instantly, but some of their fields — like the virtual network's block — cannot be changed after creation, and that field becomes forty subjects' common limit. - A single subnet prefix length is a lever: `/27` leaves **36** of forty subnets below their demand, `/26` leaves **17**. - The `/25` plan that zeroes the tail wants **5120** addresses, and the virtual network's block is **4096**; the tail does not disappear, it moves from the subnet to the virtual network. - Idle addresses are a separate number: **2584** at `/25`, **9** at `/27`. An indicator that counts only what does not fit cannot see this difference. - Giving every subnet its own prefix brings the tail to **0** and idle addresses to **984**, but it brings three separate prefixes and forty separate decisions; the lever exists for exactly the purpose of not making these forty decisions. #### Next Step This entire lesson's measurement stayed inside a single virtual network: forty subnets cut from the same block, sharing the same gateways, subject to the same lever. But the reason cloud objects exist is most often that they work together with a network we own, and that network is somewhere else. The next lesson measures the options that connect two networks: what a site-to-site tunnel built over a shared path gives which subject, versus a dedicated link giving a separated path, and when the two are used together, which class the tail collects on. ### Hybrid Connectivity Source: https://academia.sh/en/courses/network-operations/hybrid-connectivity The choice between a site-to-site tunnel and a dedicated link is a single lever: dedicated alone leaves 23 of forty subjects without a path, tunnel alone throws 14 outside their budget, and the two together bring the tail down to 9 — but all nine come from the same class. The previous lesson's entire measurement stayed inside a single virtual network. Forty subnets were cut from the same block, shared the same gateways, and fit a single prefix. But the reason cloud objects exist is most often that they work together with a network we own — and that network is somewhere else, in a different address space, under a different team's responsibility. **Hybrid connectivity** is the connection layer that forces two networks to behave as a single addressing and access plane. Two basic options exist, and both do the same job: a **site-to-site tunnel** opens an encrypted channel over a shared path, and a **dedicated link** gives a separated path between two networks. This lesson's question is not which one is better. The question is: when the choice is applied to forty subjects as a single lever, **which subject gets what**, and who is left behind. #### Two Paths What separates the two options is not technology, it is **whose capacity is shared**. A site-to-site tunnel is built on top of a path that already exists. The path is **shared**: its capacity is unpredictable, its latency changes together with others' load, and the tunnel itself adds a fixed extra latency because of encryption and encapsulation. In return, setting it up is a configuration matter; opening a new tunnel means creating a new record. A dedicated link, on the other hand, is a separated path between two endpoints. Its capacity is known and not shared with anyone else, its latency is stable, and traffic over it never touches the shared path. In return, its capacity is **tied to the physical link**: the number chosen at setup grows through a configuration change only as far as that link has room left; once it has none, a change means a new installation. ```text # taught configuration transcript, not run site-to-site tunnel: peer: company-backbone tunnel selector: - 10.40.0.0/20 -> 172.20.0.0/16 key material: read from a secret store, not written here extra latency: depends on the shared path's condition dedicated link: peer: company-backbone reserved capacity: chosen at setup, cannot be grown later path: never touches the shared path routing priority: 1. dedicated link 2. site-to-site tunnel # if dedicated is full or down ``` Key material, credentials, and pre-shared secrets are not written in any example in this lesson; where they are read from is stated in a single line in the transcript above and passed over. The tunnel's routing side — which destinations the tunnel selector admits, how unselected flows never touch the rule sequence at all, how the two directions of an assigned zone pull in opposite directions — was measured in the Wireless Networks and Network Security course's VPN Technologies lesson and is not repeated here. That lesson counted which flow the tunnel admits. This lesson counts what the tunnel gives the flow that enters it. #### The Lever: Path Choice The operator is handed three options, and one is chosen. The choice applies to all forty of forty subjects at once: dedicated link alone, tunnel alone, or both together. The third option needs a priority rule, because the dedicated link's capacity is smaller than the forty subjects' total demand. The rule is taken in its most common operational form: **class priority**. The class most sensitive to latency is placed on the dedicated link first, and the rest fall to the tunnel. Every subject has a **latency budget**, and the budget comes from its class. If a subject exceeds its budget, it is in the tail. If it finds no path at all, it is in a heavier form of the tail: it gets no service. The assumptions the measurement rests on: - **AC67** — The forty subjects are the course's fixed set; the set is not changed. A subject's capacity is the units it wants from the dedicated link, its latency is the path's baseline latency without a tunnel. - **AC68** — The dedicated link's capacity is **900** units, and total demand is **2336**; the capacity was chosen at setup and cannot be grown throughout the measurement. - **AC69** — The tunnel adds a fixed **30 ms** on top of baseline latency. The tunnel's capacity is not limited; when the shared path becomes a bottleneck, it shows this through **latency**, not capacity. - **AC70** — Latency budgets are set by class: interactive **45 ms**, batch **60 ms**, standby **120 ms**. The budget is a service decision, not a finding of the measurement. - **AC71** — In the both-together choice, the dedicated link is filled by class priority; within a class, order follows subject number, and a subject whose capacity does not fit the remaining room passes to the tunnel. - **AC72** — A subject with less than **15 ms** left in its budget counts as **thin margin**. This subject does not appear in the tail on the indicator; only the path degrading is enough to put it in the tail. - **AC73** — The oracle is the setup itself: we know capacity, baseline latency, and class because we wrote them. The measurement is not a network measurement; no tunnel is set up, no request is sent to any peer. #### The Measurement ```python """Hybrid connectivity: a single path choice applied to forty subjects at once. Lever - which path the company's internal traffic is routed over. Tail - a subject exceeding its latency budget, or finding no path at all. Invisible - a subject with less than MARGIN_THRESHOLD ms left in its budget. """ SEED = 20260812 DEDICATED = 900 # the dedicated link's unit capacity TUNNEL_EXTRA = 30 # ms the tunnel adds over the shared path TUNNEL_DEGRADED = 80 # extra ms when the shared path degrades BUDGET = {"interactive": 45, "batch": 60, "standby": 120} PRIORITY = ("interactive", "batch", "standby") MARGIN_THRESHOLD = 15 # less than this much budget left counts as invisible tail def make_rng(seed): d = seed % 2147483646 + 1 def r(n): nonlocal d d = (d * 48271) % 2147483647 return d % n return r def subjects(count=40, seed=SEED): r, out = make_rng(seed), [] for i in range(count): out.append({ "no": i + 1, "capacity": 20 + r(81), "latency": 5 + r(45), "class": ("interactive", "batch", "standby")[r(3)], "special": r(9) == 0, }) return out def place(subj, choice): """Assigns each subject a path: 'dedicated', 'tunnel', or 'none'.""" if choice == "tunnel": return {o["no"]: "tunnel" for o in subj} path, remaining = {}, DEDICATED ordered = sorted(subj, key=lambda o: (PRIORITY.index(o["class"]), o["no"])) for o in ordered: if o["capacity"] <= remaining: path[o["no"]] = "dedicated" remaining -= o["capacity"] else: path[o["no"]] = "tunnel" if choice == "both" else "none" return path def measure(subj, path, extra): """Subjects exceeding budget, unserved, and with a narrowing margin.""" over = over_ms = unserved = thin = 0 for o in subj: p, budget = path[o["no"]], BUDGET[o["class"]] if p == "none": unserved += 1 continue latency = o["latency"] + (extra if p == "tunnel" else 0) if latency > budget: over += 1 over_ms += latency - budget elif budget - latency < MARGIN_THRESHOLD: thin += 1 return over, over_ms, unserved, thin subj = subjects() print(f"subjects {len(subj)} | total demand {sum(o['capacity'] for o in subj)} | " f"dedicated link {DEDICATED} units | tunnel extra latency {TUNNEL_EXTRA} ms") print("latency budget: " + ", ".join(f"{s} {BUDGET[s]} ms" for s in PRIORITY)) print() print(f"{'choice':<16s} {'on dedicated':>12s} {'on tunnel':>9s} {'unserved':>8s} " f"{'over budget':>12s} {'over ms':>8s} {'thin margin':>12s}") for choice in ("dedicated", "tunnel", "both"): path = place(subj, choice) a, ms, h, d = measure(subj, path, TUNNEL_EXTRA) print(f"{choice:<16s} {sum(1 for v in path.values() if v == 'dedicated'):12d} " f"{sum(1 for v in path.values() if v == 'tunnel'):9d} " f"{sum(1 for v in path.values() if v == 'none'):8d} " f"{a:12d} {ms:8d} {d:12d}") print() print(f"when the shared path degrades (extra latency {TUNNEL_EXTRA} -> {TUNNEL_DEGRADED} ms)") print(f"{'choice':<16s} {'over budget':>12s} {'over ms':>8s} {'thin margin':>12s}") for choice in ("dedicated", "tunnel", "both"): path = place(subj, choice) a, ms, h, d = measure(subj, path, TUNNEL_DEGRADED) print(f"{choice:<16s} {a:12d} {ms:8d} {d:12d}") print() path = place(subj, "both") for cls in PRIORITY: group = [o for o in subj if o["class"] == cls] ded = sum(1 for o in group if path[o["no"]] == "dedicated") over = sum(1 for o in group if path[o["no"]] == "tunnel" and o["latency"] + TUNNEL_EXTRA > BUDGET[cls]) print(f"both together | {cls:<11s} subjects {len(group):2d} | " f"dedicated {ded:2d} | tunnel {len(group) - ded:2d} | " f"over budget {over:2d}") ``` ``` subjects 40 | total demand 2336 | dedicated link 900 units | tunnel extra latency 30 ms latency budget: interactive 45 ms, batch 60 ms, standby 120 ms choice on dedicated on tunnel unserved over budget over ms thin margin dedicated 17 0 23 0 0 2 tunnel 0 40 0 14 144 10 both 17 23 0 9 76 5 when the shared path degrades (extra latency 30 -> 80 ms) choice over budget over ms thin margin dedicated 0 0 2 tunnel 32 1508 1 both 16 758 3 both together | interactive subjects 11 | dedicated 11 | tunnel 0 | over budget 0 both together | batch subjects 20 | dedicated 5 | tunnel 15 | over budget 9 both together | standby subjects 9 | dedicated 1 | tunnel 8 | over budget 0 ``` #### The Tail of Three Choices The first row is a trap, and it needs to be read carefully. In the dedicated-alone choice, subjects over budget are **0**, over ms **0**. An indicator that measures latency shows this deployment as flawless. The column next to it tells you what was paid: **23** of forty subjects find **no path at all**. The dedicated link has **900** units, total demand is **2336**; once capacity runs out, the next subject in line is left out. The tail did not disappear, it stepped **outside the indicator**. The latency indicator counts only served subjects, and an unserved subject has no latency. This is the sharpest form of what has been counted since the course's first lesson: a good-looking number's **denominator** can have been shrunk by the tail itself. In the tunnel-alone choice, the denominator is fixed — all forty of forty subjects are served, unserved is **0**. This time the tail is inside the indicator: **14** subjects exceed their budget, total overage **144 ms**. There is no capacity problem, there is a latency problem. The **30 ms** the tunnel adds is fixed and applied the same way to everyone; classes with a tight budget cannot absorb it. The both-together choice is best on both indicators: unserved **0**, over budget **9**, over ms **76**. The dedicated link takes the latency-sensitive subjects, the rest pass over the tunnel. The tail drops from **14** to **9**. #### Where the Tail Collected The bottom transcript gives the both-together choice's real result, and this result matters more than the number itself. All **11** of the interactive class's **11** subjects are on the dedicated link; over budget **0**. Because the standby class's budget is **120 ms**, its eight subjects on the tunnel are comfortable too; over **0**. All **nine** of the nine in the tail come from the **batch** class. The priority rule did this. The rule put the interactive class first, that class took two-thirds of the dedicated link with **580** units, and the remaining **320** units met the demand of only five of the batch class's twenty subjects. Fifteen batch subjects fell to the tunnel, and nine of them exceeded their budget. This is the course's second claim in this lesson's form. None of the three choices removed the tail. Dedicated alone placed the tail on the unserved, tunnel alone on everyone with a tight budget, both together on a **single class**. The choice giving the lowest number is the choice that concentrates the pain into the narrowest place. Which one is correct is not a measurement decision, it is a **policy** decision — and whether the batch class's nine subjects accept this cannot be read from the table. #### Failover and Two Side Effects of Two Paths The real rationale for a both-together setup is most often not capacity, it is **failover**: when the dedicated link fails, traffic falls to the tunnel and service is not interrupted. This cost is already written into the measurement. The moment the dedicated link goes down, all forty of forty subjects move to the tunnel, meaning the deployment drops to the tunnel row: over budget **14**, over ms **144**. Failover keeps the service running, but it raises the tail from nine to fourteen, and it does this at the moment of a single failure. An operation planning failover has to write the row that comes **after** the failover too. Two paths existing at the same time has two more side effects, and both fall outside this measurement. The first is **address overlap**. Hybrid connectivity forces two separate address spaces to behave as a single plane; if the two spaces use the same block, no path can be established. This is why the previous lesson's virtual network block is a decision chosen once and irreversibly: the way to resolve an overlap later is to renumber the subnets or add a translating layer. The second option only works by breaking the address the rule sequence reads. The second is **path asymmetry**. The outbound leg can go over the dedicated link and the return over the tunnel; the two directions taking separate paths is not, by itself, a fault, but it breaks every object that keeps state. A security group that sees one direction does not see the other, and recognizes the flow only by half. **Narrowing:** pin the **flow**, not the direction, to a path — both directions of traffic between two endpoints are pinned to the same path, and the pinning is written the same way on both ends through routing priority. #### The Invisible Tail: Thin Margin The last column counts subjects that have not entered the tail but stand at its edge. A subject with less than **15 ms** left in its budget is not in the tail; it produces no warning, appears in no report. The second table shows what happens when the shared path degrades. When the tunnel's extra latency climbs from **30 ms** to **80 ms**, the tail in the tunnel-alone choice jumps from **14** to **32**, over ms from **144** to **1508**. In the both-together choice, the tail climbs from **9** to **16**, overage from **76** to **758**. The dedicated row does not move at all: **0** and **0**. This is exactly what a separated path is worth, and it shows precisely here. The movement in the thin-margin column reads backwards at first glance. In the tunnel-alone choice, this number falls from **10** to **1**, in both-together from **5** to **3**. This is not an improvement. The thin-margin subjects **crossed into the tail** along with the degradation; the number falling is because the subjects standing at the edge ran out. An indicator improving can come from the set it counts **emptying out**. The operational rule that follows is singular: thin margin has to be counted as a **separate indicator**. The indicator counting the tail speaks **after** degradation; the indicator counting thin margin speaks **before** degradation, and the two are not the same number. #### Summary - A site-to-site tunnel is built over a shared path and adds a fixed extra latency; a dedicated link gives a separated path, but its capacity is chosen at setup and grows only as far as the physical link has room. - In the dedicated-alone choice, subjects over budget are **0**, because **23** of forty subjects find no path at all and never enter the indicator's denominator. - Tunnel alone gives a path to all forty of forty subjects but throws **14** outside their budget; both together bring the tail down to **9**. - All nine of the nine in the tail are from the batch class: once the priority rule puts the interactive class's eleven on the dedicated link, the remaining **320** units meet the demand of only five batch subjects. - Subjects with less than **15 ms** of margin left do not appear in the tail; when the shared path degrades from **30 ms** to **80 ms**, they cross into the tail, and the thin-margin count falling is not an improvement, it is the edge set emptying out. #### Next Step Every number in this lesson gave a choice's **result**: the path was chosen, applied to forty subjects, the tail was counted. In operations, this order is meant to run in reverse — what the tail will be should be known **before** touching the lever. The next lesson measures the place built for exactly this: the **lab environment** where a copy of the real network is produced. What is measured is not the lab itself, it is **how much it represents the real thing**: how many subjects can be modeled, how many cannot, and how many subjects a change validated in the lab behaves differently on in the field. ### Network Simulation Environments Source: https://academia.sh/en/courses/network-operations/network-simulation-environments The lab environment's scale is a lever: the seventeen subjects on which the change fails in the field stay constant regardless of scale; the only thing that changes is how many the lab foresees, and even at full scale, nine subjects cannot be represented. The previous lesson counted a path choice's result: the choice was made, applied to forty subjects, the tail was measured. Every measurement in the course followed this order. In operations, this order is meant to run in reverse — what the tail will be should be known before touching the lever. The **lab environment** is built for exactly this: a copy of the real network is produced, the change is applied there first, and the result is read there. This lesson's question is not how the lab is set up. The question is: how much does the lab represent reality, and on how many subjects does a change validated there behave differently in the field. #### What the Lab Promises The lab delivers three things in full. Topology can be copied exactly: which device connects to which, which subnet sits where, which path carries which cost. Configuration is text, so it transfers identically. And the lab is **disposable**: a broken setup is deleted and recreated, which is why things that cannot be tried in the field are tried there. What it does not deliver is also three things, and this is the part that concerns the measurement. **Timing is not real**: virtual links' latency is fabricated, and queue behavior does not come out the way it does under real load. **Load is not real**: traffic generated in the lab imitates the field's scale, not its shape. And **hardware is not real**: a device's own resource limits, chip-level table capacity, and vendor-specific behavior are represented by a **generic stand-in**. This third one gives rise to the lever. A lab does not model every device separately; a modeling **scale** is chosen, and that scale applies to all forty subjects. ```text # taught lab definition, not run topology: node: - name: core-1 model: generic router interface: [ge-0, ge-1] - name: edge-07 model: generic switch interface: [ge-0, ge-1] link: - core-1:ge-0 <-> edge-07:ge-0 latency: 5ms # fabricated, not measured capacity: 1g # fabricated validation: - reachability: edge-07 -> core-1 - configuration diff: field vs. lab - rollback rehearsal: change applied and rolled back ``` The `model: generic` lines in the transcript carry this lesson's entire question. The object sitting in the lab is not the field device itself, it is its **class**. Class behavior is correct; device behavior is approximate. #### The Lever: The Lab's Scale Scale is a cost decision. Modeling forty subjects one-to-one means forty nodes, forty configurations, and forty times the resources; modeling four subjects is a tenth of that. The operator chooses a single number: how many subjects will be represented in the lab. The change measured is something concrete: a configuration that reserves a fixed amount of resource on every device. If a device's capacity is below this amount, the change fails on it. Alongside this, a second source of failure exists, and it has nothing to do with scale: because special devices are represented by a generic stand-in in the lab, the lab's verdict about them is **not evidence**. The assumptions the measurement rests on: - **AC74** — The forty subjects are the course's fixed set and are not changed. A subject's capacity is the upper limit of the resource the change can consume. - **AC75** — The change reserves **45** units on every device; a device whose capacity is below this fails in the field. - **AC76** — A special device is represented by a generic stand-in in the lab. Even if its capacity is sufficient, it behaves differently in the field, and the lab cannot see this at **any** scale. - **AC77** — Scale is choosing modeled subjects at even intervals by subject number: at scale four, every tenth is modeled, at forty, all of them. The choice does not use the oracle. - **AC78** — The oracle is the setup itself: we know capacity and which device is special because we wrote them. The lab has no access to this information. - **AC79** — The measurement is not a lab setup; no node is started, no device command runs. What is counted is the coverage of the representation. #### The Measurement ```python """Lab environment: a single scale decision determines the representation of forty subjects. Lever - the number of subjects modeled one-to-one in the lab. Tail - a subject where the change fails in the field. Invisible - a subject that fails in the field and never shows up in the lab at all. """ SEED = 20260812 ALLOCATED = 45 # units the change allocates on every device def make_rng(seed): d = seed % 2147483646 + 1 def r(n): nonlocal d d = (d * 48271) % 2147483647 return d % n return r def subjects(count=40, seed=SEED): r, out = make_rng(seed), [] for i in range(count): out.append({ "no": i + 1, "capacity": 20 + r(81), "latency": 5 + r(45), "class": ("interactive", "batch", "standby")[r(3)], "special": r(9) == 0, }) return out def in_field(o): """Does the change fail on this subject?""" return o["capacity"] < ALLOCATED or o["special"] def in_lab(o, modeled): """Does the lab foresee this failure in advance? A special device is represented by a generic stand-in; its behavior cannot be reproduced. """ return o["no"] in modeled and o["capacity"] < ALLOCATED and not o["special"] subj = subjects() print(f"subjects {len(subj)} | units the change allocates {ALLOCATED} | " f"insufficient capacity {sum(o['capacity'] < ALLOCATED for o in subj)} | " f"special devices {sum(o['special'] for o in subj)} | " f"fails in the field {sum(in_field(o) for o in subj)}") print() print(f"{'scale':>6s} {'modeled':>11s} {'unmodeled':>14s} " f"{'fails in field':>17s} {'foreseen in lab':>22s} {'missed':>6s}") for step in (10, 5, 2, 1): modeled = {o["no"] for o in subj if o["no"] % step == 0} foreseen = sum(in_lab(o, modeled) for o in subj) failed = sum(in_field(o) for o in subj) print(f"{len(modeled):6d} {len(modeled):11d} " f"{len(subj) - len(modeled):14d} {failed:17d} " f"{foreseen:22d} {failed - foreseen:6d}") print() full = {o["no"] for o in subj} missed = [o for o in subj if in_field(o) and not in_lab(o, full)] print(f"at full scale, {len(missed)} subjects missed:", [(o["no"], o["capacity"], "special" if o["special"] else "generic") for o in missed]) ``` ``` subjects 40 | units the change allocates 45 | insufficient capacity 12 | special devices 9 | fails in the field 17 scale modeled unmodeled fails in field foreseen in lab missed 4 4 36 17 0 17 8 8 32 17 1 16 20 20 20 17 3 14 40 40 0 17 8 9 at full scale, 9 subjects missed: [(4, 30, 'special'), (7, 78, 'special'), (12, 55, 'special'), (20, 29, 'special'), (25, 31, 'special'), (26, 56, 'special'), (31, 57, 'special'), (33, 24, 'special'), (40, 70, 'special')] ``` #### What Scale Changed The middle column does not move from scale to scale at all: subjects failing in the field are **17** at every scale. The lab does not change the field, because the lab does not touch the field. The only thing the lever changes is the two right-hand columns. At scale **4**, the lab foresees **none** of the failures. Validation completes, the report comes back clean, the change is approved — and it fails on seventeen devices in the field. At scale **8**, foreseen is **1**, at scale **20**, **3**. A lab modeling half of the forty subjects catches three of the seventeen failures. At scale **40**, no subject is left unmodeled, and foreseen climbs to **8**. Missed is still **9**. The last row states who these nine are: all nine of the nine are **special devices**. Five have capacity above **45** — their failure comes not from capacity but from behavior the generic stand-in cannot represent. The remaining four already have insufficient capacity, and the lab would see this if it were in a position to measure it; it cannot, because its verdict about those four devices **is not evidence in either direction**. The result that follows is the course's third claim in its final form. **The lab's scale did not reduce failure; it changed how many of them were foreseen.** And growing scale all the way to the end still does not close the tail, because part of the tail comes not from scale but from **representation**. A forty-node lab wants ten times the nodes of a four-node one, and in return, missed drops from **17** to **9**. This does not make the lab unnecessary; it changes what it is used for. Where the lab is reliable is **topology** — reachability, configuration diff, rollback rehearsal. Where it is unreliable is **capacity and device-specific behavior**. This is why, for every bulk change applied in the field, the lab alone is not enough: dry run, staged rollout, and a written rollback step stand alongside it. If nine subjects do not show up in the lab, they need to be seen in the **last** wave, not the first. #### Summary - The lab reliably delivers topology, configuration, and rollback rehearsal; it does not deliver timing, real load, or device-specific behavior. - The lab's scale is a single lever and does not change the field's failure: **17** of forty subjects fail at every scale. - At scale 4, the lab foresees **none** of these seventeen, at scale 20, **3**, at full scale, **8**. - Even at full scale, **9** subjects are missed, and all nine are special devices; this part of the tail comes not from scale but from being represented by a generic stand-in. - The lab is therefore not approval itself, it is one part of it; dry run, staged rollout, and written rollback have to stand alongside it. #### Course Wrap-Up A single question was asked throughout the course: when the operator touches a lever, how many of the forty subjects does it misserve. Sixteen lessons asked this question on sixteen separate objects, and every time it produced two numbers — the tail, and the part of the tail invisible on the indicator. | Lesson | Lever | Tail | Invisible | |---|---|---|---| | Load Balancers | the balancer's layer | layer 4: **17**, layer 7: **16** subjects over | the connection-count indicator says **10** for everyone; the 17 over-subjects are not there | | Load Balancing Algorithms | the distribution rule | at load 1200: **6 / 0 / 0 / 7**, at load 2600: **23 / 40 / 40 / 22** | **1160** idle units at load 1200 | | Reverse Proxy and Forward Proxy | a single limit number | unprotected subjects on reverse proxy **6** to **34**, throttled subjects on forward proxy **27** to **4** | **9** subjects and **430** units the forward proxy never sees | | Content Delivery Networks | which content sits at the edge | misses on pull climb from **33** to **245**, bounded push gives **379** | objects the edge never sees, **7** to **25**; **402** units idle under push | | Quality of Service | the scheduling rule | unmet demand of **433** units is the same across all three rules; queued subjects **40 / 9 / 29** | overall average wait falls from **1.19** to **0.40** while the longest wait climbs from **3** to **13** | | Network Telemetry | the sampling rate | missed heavy events **0 / 13 / 15 / 20 / 20** | subjects never seen at all **0 / 1 / 8 / 17 / 26** | | Device Monitoring Protocols | polling timeout and notification threshold | timed-out devices **30** to **0**; missed heavy events **18** to **4** | never polled **11**, never reported climbs as high as **35** | | Packet Analysis Tools | capture filter narrowness and buffer size | the narrowest filter leaves **91** flows outside and misses **21** heavy events; unfiltered, **32** flows do not fit the buffer | **33** subjects never seen at all when the filter is narrow, **5** even unfiltered | | Reachability and Latency Measurement | the probe list's coverage | **36** subjects never probed across four targets; the indicator oscillates between **37.5** and **62.5** while the real value holds at **57.5%** | invisible unreachable subjects **15 / 12 / 11 / 3 / 0**, missed heavy events as high as **20** | | The Case for Network Automation | a single template instead of manual touches | manual: **12** errors across **122** touches, **10** faulty devices | of the template's **6** deviated devices, hidden from the report **6 / 3 / 3** | | Configuration Models | the configuration model's coverage | fields outside the model **42 / 17 / 9** | **9** field instances no model can represent | | Programmable Interfaces | the request form | the append form is not idempotent and changes state on **40** devices on a second application; on retry it breaks **5** devices | the out-of-model field full replace erases on **9** devices without reporting it | | Managing Devices with Scripts | a single template applied to forty devices | matched **34**, deviated **6** — whichever class the template targets | deviations hidden from the report **6 / 3 / 3**, report coverage **0.000 / 0.500 / 0.500** | | Cloud Networking Concepts | a single subnet prefix length | **17** subnets that do not fit their block at `/26` | idle addresses: **2584** at `/25`, **9** at `/27` | | Hybrid Connectivity | the internal traffic's path choice | both together: **9** subjects, all nine from the batch class | **5** subjects with less than 15 ms of margin left | | Network Simulation Environments | the lab's scale | **17** subjects fail in the field, at every scale | **9** special devices invisible even at full scale | Every one of the sixteen rows was read from its own lesson's measurement; no number was copied from a shared definition. The table repeats a single pattern. The lever has no "good" setting; it has settings that **move where the tail sits**. And every row's right-hand column says the same thing: a lever's most expensive effect falls not on what it measures, but on **what it cannot measure**. With this course, the M04 curriculum closes. Six courses passed through the same network with six separate measures. | Course | Measurement Axis | |---|---| | How the Internet Works | a single request's end-to-end trace, and the single thing each stage hands off to the next | | Network Models and Protocols | the promise a layer makes against the promise it explicitly does not make | | Application Layer Protocols | the decision an intermediary can take from a message without a round trip | | Switching and Routing | the packet's fate for as long as the tables disagree | | Wireless Networks and Network Security | the gap between the boundary's intended state and what actually passes | | Network Operations and Automation | the tail a single lever leaves across forty subjects | The order is not arbitrary. The first course traced a request from start to finish and showed that every stage forgets what came before it. The second split this forgetting into layers and separated what each layer promises from what it explicitly does not promise. The third counted how the promise is written into a message and what the reader can extract from it. The fourth showed that the path carrying the message has its own settling time. The fifth measured where the boundary placed on that path diverges between intent and outcome. The sixth counted the single hand applying all of this to forty subjects at once. Something changes at the last step. Every measurement in this course had a single operator: they touched the lever, they read the tail, they wrote the rollback. In reality, it is not like this. One person writes the template, another reviews it, a third applies it, and only a fourth knows the ninth special device in the field. **Running a network is a team's job** — and the right-hand column of these tables is most often something one person knows but the team does not, because it was never written down. Team work has its own discipline, and that discipline is established not in this curriculum but in the Software Development Practice curriculum: keeping a change's history, recording it in a revertible form, having it read by someone else before it is applied, and writing down the reasoning behind a decision. Each of the three steps written in this course as "dry run, staged rollout, rollback" corresponds to a practice there. The hand touching the lever is one person's hand; whether the team that owns the forty subjects knows what that hand did is the subject of a separate discipline.