--- title: 'Reachability and Latency Measurement' source: 'https://academia.sh/en/courses/network-operations/reachability-and-latency-measurement' course: 'Network Operations and Automation' language: en updated: '2026-08-17T18:07:15+00:00' license: 'CC BY-SA 4.0' --- # Reachability and Latency Measurement The end-to-end health indicator is a sample: with a four-target probe list, 36 of the forty subjects are never probed, availability swings between 37.5 and 62.5 while the true value sits at 57.5, and the indicator's resolution is set by the target count, not the subject count. The previous three lessons all looked from inside the network: the record a device produces, the reply a device gives, the packet crossing the wire. All three had individual devices as their subject, and in all three, what was measured was how many devices stayed invisible. The real question asked in operations, though, is asked from outside, and it is one sentence: **can one end reach the other, and how long does it take?** The tests that answer this question are collapsed into a single number, and that number is written at the very top of the dashboard. This lesson's question is who that single number **speaks on behalf of**. ## Two Separate Words In this lesson, two concepts that get confused with each other are called by separate names, and the distinction is binding. **Reachability** is whether one end can get to a target or not. What measures it is a **reachability test**: an echo request is sent, a reply is awaited, the result comes out binary. The test itself, and how many separate causes a non-response can fit, were established in the **Linux Network Administration and Troubleshooting** course; here the test is used not as a procedure but as a **data source**. **Availability**, by contrast, is a ratio: how many of the tests succeeded. It is derived from a binary result and written as a percentage. This is the number we see on the dashboard. The difference sits at the center of the measure. Reachability is **about a subject**; availability is **about a set**, and the probe list determines who that set is. The list is a lever: a single configuration file chooses which of the forty subjects gets spoken for. Indicator design, alert thresholds, and error budgets are the subject of the **Observability and Reliability** course and are not repeated here. What is measured here is not how to design an indicator, but which subject an already-designed indicator **fails to represent.** ## How a Test Is Set Up ```text # taught probe definition and dashboard format , not executed probe definition name end-to-end-reach target list device-03, device-11, device-27, device-38 procedure reachability test, 5 attempts, 1 s interval success rule at least 4 attempts got a reply latency value median of the attempts window 24 h dashboard availability ....% average latency .... ms slowest target .... ms target count 4 ``` The definition's most important line is `target list`, and it never appears on the dashboard. Even though the dashboard has a `target count` field, it does not say **who** the targets are or **how many subjects** the inventory contains. The indicator reports the numerator, not the denominator. ## The Indicator Is a Sample The probe list selects a subset of the forty subjects, and the indicator measures only that subset. This has the same shape as the sampling rate from the first lesson — there, events were being selected; here, subjects are. The lever's tail is doubled in the same way: a subject not on the list never enters the indicator, and if a subject not on the list is broken, **the indicator stays green.** The second, less noticed, consequence is **resolution**. In a test with four targets, the ratio can only take the values 0, 25, 50, 75, and 100; the smallest measurable difference is **100/4 = 25 points**. Putting a one-percent threshold on such an indicator asks it to report a difference below its own measurement floor. The measurement's assumptions: - **OB22** — The forty subjects are the course's fixed population and are not changed in this lesson. - **OB23** — A subject counts as unreachable if it produced at least one heavy event in the window. The oracle is the setup; the test does not use this information, it is only compared against the result. - **OB24** — Every target is probed once, and the probe gives a binary result. Retrying attempts and the success rule do not change the result, they only reduce noise; the measurement therefore takes a single attempt. - **OB25** — The measured latency is the target's own latency. Contributions from devices along the path are not modeled in this lesson; if they were, the measured values would grow, but the trend below would not change. - **OB26** — The probe list's order is a fixed sequence coming from the setup, and the lists are nested: the eight-target list contains the whole of the four-target list. If the order changes, which subjects are invisible changes, but the counts stay similar. - **OB27** — `Availability` is computed only over the probed targets; an unprobed subject enters neither the numerator nor the denominator. The calculation in the field is the same. - **OB28** — The smallest measurable difference across forty subjects is 1/40 = 0.025; in a four-target list, it is 1/4 = 0.25. The indicator's resolution depends on the list, not the inventory. - **OB29** — The probes are taken as independent: each target's result comes only from its own state, path sharing is not modeled. If it were, the indicator's swing would grow. ## Measurement ```python """End-to-end health: the probe list is a single lever, what is measured is what the indicator fails to represent. Part 1 - forty subjects; a subject producing a heavy event counts as unreachable. Part 2 - five probe lists: availability, latency indicator, and the tail. """ SEED = 20260812 def rng(seed): state = seed % 2147483646 + 1 def draw(n): nonlocal state state = (state * 48271) % 2147483647 return state % n return draw def subjects(count=40, seed=SEED): draw, items = rng(seed), [] for i in range(count): items.append({"id": i + 1, "capacity": 20 + draw(81), "latency": 5 + draw(45), "class": ("interactive", "batch", "standby")[draw(3)], "special": draw(9) == 0}) return items def events(subs, count=400, seed=SEED + 3): draw, items = rng(seed), [] for i in range(count): s = subs[draw(len(subs))] items.append({"id": i + 1, "subject": s["id"], "heavy": draw(25) == 0}) return items def probe_order(subs, seed=SEED + 11): """The probe list's order; its prefixes give nested target sets.""" draw, remaining, order = rng(seed), [s["id"] for s in subs], [] while remaining: order.append(remaining.pop(draw(len(remaining)))) return order subs = subjects() evts = events(subs) lat = {s["id"]: s["latency"] for s in subs} heavy = {s["id"]: 0 for s in subs} for e in evts: if e["heavy"]: heavy[e["subject"]] += 1 unreachable = {n for n, c in heavy.items() if c >= 1} order = probe_order(subs) print(f"subject {len(subs)} | heavy event {sum(heavy.values())} | unreachable subject " f"{len(unreachable)} | true availability " f"{100 * (len(subs) - len(unreachable)) / len(subs):.1f}%") print(f"true latency: average {sum(lat.values()) / len(subs):.1f} ms, " f"slowest {max(lat.values())} ms, fastest {min(lat.values())} ms") print() print(f"{'targets':>7s} {'unprobed':>8s} {'availability':>12s} {'resolution':>10s} " f"{'avg ms':>7s} {'slowest':>8s} {'invisible':>9s} {'heavy missed':>13s}") for n in (4, 8, 16, 32, 40): h = order[:n] ok = [x for x in h if x not in unreachable] print(f"{n:7d} {len(subs) - n:8d} {100 * len(ok) / n:11.1f}% " f"{100 / n:9.1f}p {sum(lat[x] for x in h) / n:7.1f} " f"{max(lat[x] for x in h):8d} {len(unreachable - set(h)):9d} " f"{sum(c for x, c in heavy.items() if x not in h):13d}") ``` ``` subject 40 | heavy event 22 | unreachable subject 17 | true availability 57.5% true latency: average 23.4 ms, slowest 48 ms, fastest 6 ms targets unprobed availability resolution avg ms slowest invisible heavy missed 4 36 50.0% 25.0p 22.0 40 15 20 8 32 37.5% 12.5p 21.6 40 12 15 16 24 62.5% 6.2p 21.1 40 11 14 32 8 56.2% 3.1p 26.0 48 3 3 40 0 57.5% 2.5p 23.4 48 0 0 ``` ## The Subject Not Represented With the four-target list, the indicator reads **50.0%**. Behind this number are two successful and two failed tests — four observations in total. At the same time, **36** of the forty subjects were never probed at all, **15** unreachable subjects never appear in a single test, and **20** heavy events never enter the indicator at all. The structure here is the same as the first lesson's result, read backwards. There, an unreported subject left a gap on the dashboard; here it does not even leave a gap, because the indicator is a single percentage, and a gap does not show inside a percentage. **An unprobed subject neither raises nor lowers the indicator; it sits outside the indicator altogether.** At sixteen targets, unprobed subjects fall to **24**; at thirty-two targets, to **8**. Invisible unreachable subjects do not fall at the same rate: 15, 12, 11, 3. On the third row, doubling the list's target count only drops the invisible unreachable subjects one at a time — because the targets added to the list are mostly subjects that are already working. Growing the coverage does not shrink the tail proportionally. ## The Indicator's Own Noise As important as the columns on the right is how the percentage on the left behaves. As the probe list grows, the indicator reads **50.0**, **37.5**, **62.5**, **56.2**, and **57.5** in turn. The true value is **57.5** across the entire window and never changed. With nothing happening on the network, the indicator swings **between 37.5 and 62.5**, a band of twenty-five points. The reason is written in the resolution column. In a four-target list, a single test's result moves the indicator by **25 points**; at eight targets, **12.5**; at forty targets, **2.5** points. If an alert threshold is set below this resolution, the alert measures the list, not the network. This has a direct operational consequence: **an availability ratio computed from a small target list cannot be read as a trend.** Seeing 50 and then 75 in two consecutive windows is not an improvement, it is a single subject changing state. The measure's band is the divisor itself. ## What the Average Hides The latency columns show a third tail. At four targets, the measured average is **22.0 ms**; the forty subjects' true average is **23.4 ms**. The gap is small, and the indicator therefore looks correct. The misleading column is not the average, it is the **slowest**. Across the four-, eight-, and sixteen-target lists, the measured slowest value is **40 ms** in all three. The population's true slowest subject is **48 ms**, and it only shows up in the thirty-two-target list. So up to sixteen targets, the indicator underreports the tail by eight milliseconds — and it does so without giving any sign of it. The relationship between the average and the slowest also reverses: at thirty-two targets, the average **rises** to **26.0** ms, because slow subjects have entered the list. The indicator looks like it has gotten worse, but the network has not changed; only the field of view has widened. **An indicator getting worse does not mean the thing it measures has gotten worse.** ## The Probes Are Not Independent The measurement counts every target as a separate observation and computes the percentage accordingly. In the field, this assumption only partly holds, because the probes all start **from the same point** and share their paths. Every target leaving a single vantage point shares the first few devices. If a shared device breaks, every target fails at once, and the indicator drops to **0%**. Even if thirty-nine of the forty subjects are sound, the table reports a total outage. The reverse case is subtler: when a device far from the vantage point breaks, only one or two targets go through it, so the indicator drops by only a few points and stays under the threshold. This is why a percentage measured from a single point does not give information about a **location**. A drop in the percentage does not mean "this much of the network is broken," it means "from this point, this much is not visible." The way to tell locations apart is not to tweak the divisor, but to **multiply the vantage points**: when the same target list is run from two separate points, targets that fail from only one point show a problem specific to that point, and targets that fail from both show a problem in the target itself. This distinction was not modeled in the measurement; the probes were taken as independent, and every target's result came only from its own state. If it had been modeled, the swing in the availability column would have **grown**, because a shared device's state would push multiple targets in the same direction at once. The independence assumption makes the indicator look steadier than it is. ## Active and Passive Measurement The measurement above is **active measurement**: the side doing the measuring produces the probe traffic. This topic's first three lessons, by contrast, were **passive measurement** — reading from traffic that already existed, without adding any packets. The two have their tails in opposite places, and this is the reason for reading all four together. Passive measurement only sees a subject that is **talking**. A device with no traffic produces no flow record, never enters a capture, and sends no notification; this is why the twenty-six subjects lost in the first lesson were lost. Passive measurement, in exchange, measures real user traffic: the latency it sees is a latency someone actually experienced. Active measurement does the reverse. Because it produces the probe traffic itself, it can also measure a silent subject; a device placed on the list is probed even if it is never used at all. Its cost is twofold. The latency it measures is a latency nobody experienced — probe packets can differ from the real load's path, the real load's size, and the real load's class. Second, the measurement is **bound by the list**: active measurement has to know in advance what to measure, while passive measurement can see even what it does not know about. The two methods have no shared gap, and this is not a coincidence. The subject passive measurement fails to see is the silent one; the subject active measurement fails to see is the one not on the list. A device being both silent and off the list is a separate deficiency, and it comes from the inventory itself. The same result came out of the comparison between the monitoring models: **when independent blind spots are laid on top of each other, the tail shrinks, but a subject the inventory does not see appears in neither.** ## Narrowing the Tail Three narrowings make it visible who the indicator speaks on behalf of. **First narrowing — write the denominator.** A `subjects in inventory` field is placed on the dashboard alongside `target count`. With a four-target list, the table says "50%, 4/40 subjects," and the reader sees what was not measured. This is a correction made without adding any measurement. **Second narrowing — tie the resolution to the threshold.** The alert threshold is chosen larger than the list's resolution. For a four-target list, a threshold smaller than 25 points cannot be defined; if it needs to be defined, the list must be grown. The threshold and the divisor are decided together. **Third narrowing — feed the target list from the previous lessons' output.** A subject never visible in telemetry, a subject never reported in monitoring, and a subject never captured — these are the probe list's **first candidates**. All three lessons were counting who is invisible; this lesson says where that list gets written. A list fed this way closes more of the tail while staying smaller than a randomly chosen list. ## Summary - Reachability is about a subject and gives a binary result; availability is a ratio, and the probe list determines its set — the list is a lever. - With a four-target list the indicator reads **50.0%**, while **36** of the forty subjects are never probed, **15** unreachable subjects stay invisible, and **20** heavy events never enter the indicator at all. - As the list grows, unprobed subjects fall from 36 to 0, but invisible unreachable subjects do not fall at the same rate (15, 12, 11, 3): growing the coverage does not shrink the tail proportionally. - While the true value is fixed at **57.5%**, the indicator swings between **37.5** and **62.5**; resolution is **25** points at four targets and **2.5** at forty, and the alert threshold cannot be set below this band. - The measured slowest value stays at **40 ms** through sixteen targets; the true slowest is **48 ms**; at thirty-two targets the average rises because the field of view changed, not the network. ## Next Step Every lever touched throughout this topic was a lever of **observation**. The sampling rate, the timeout threshold, the capture filter, and the probe list — all four decided what would be seen, but none of them touched the network itself. A badly set sampling rate drops a device out of the report; it does not drop a connection. This is where the whole weight of the measurement came from: a subject left in the tail was not harmed, only unseen. The next topic removes this distinction. What happens when the lever is no longer observation's but **configuration itself**? A single rule again applies to all forty subjects at once, but this time the moment it is applied, the devices' behavior changes, and a subject left in the tail falls out not of the report, but of the service.