Reachability and Latency Measurement

The previous three lessons all looked from inside the network: the record a device produces, the reply a device gives, the packet crossing the wire. All three had individual devices as their subject, and in all three, what was measured was how many devices stayed invisible.

The real question asked in operations, though, is asked from outside, and it is one sentence: can one end reach the other, and how long does it take? The tests that answer this question are collapsed into a single number, and that number is written at the very top of the dashboard. This lesson’s question is who that single number speaks on behalf of.

Two Separate Words

In this lesson, two concepts that get confused with each other are called by separate names, and the distinction is binding.

Reachability is whether one end can get to a target or not. What measures it is a reachability test: an echo request is sent, a reply is awaited, the result comes out binary. The test itself, and how many separate causes a non-response can fit, were established in the Linux Network Administration and Troubleshooting course; here the test is used not as a procedure but as a data source.

Availability, by contrast, is a ratio: how many of the tests succeeded. It is derived from a binary result and written as a percentage. This is the number we see on the dashboard.

The difference sits at the center of the measure. Reachability is about a subject; availability is about a set, and the probe list determines who that set is. The list is a lever: a single configuration file chooses which of the forty subjects gets spoken for.

Indicator design, alert thresholds, and error budgets are the subject of the Observability and Reliability course and are not repeated here. What is measured here is not how to design an indicator, but which subject an already-designed indicator fails to represent.

How a Test Is Set Up

# taught probe definition and dashboard format , not executed

probe definition
  name             end-to-end-reach
  target list      device-03, device-11, device-27, device-38
  procedure        reachability test, 5 attempts, 1 s interval
  success rule     at least 4 attempts got a reply
  latency value    median of the attempts
  window           24 h

dashboard
  availability      ....%
  average latency    .... ms
  slowest target      .... ms
  target count        4

The definition’s most important line is target list, and it never appears on the dashboard. Even though the dashboard has a target count field, it does not say who the targets are or how many subjects the inventory contains. The indicator reports the numerator, not the denominator.

The Indicator Is a Sample

The probe list selects a subset of the forty subjects, and the indicator measures only that subset. This has the same shape as the sampling rate from the first lesson — there, events were being selected; here, subjects are. The lever’s tail is doubled in the same way: a subject not on the list never enters the indicator, and if a subject not on the list is broken, the indicator stays green.

The second, less noticed, consequence is resolution. In a test with four targets, the ratio can only take the values 0, 25, 50, 75, and 100; the smallest measurable difference is 100/4 = 25 points. Putting a one-percent threshold on such an indicator asks it to report a difference below its own measurement floor.

The measurement’s assumptions:

• OB22 — The forty subjects are the course’s fixed population and are not changed in this lesson.
• OB23 — A subject counts as unreachable if it produced at least one heavy event in the window. The oracle is the setup; the test does not use this information, it is only compared against the result.
• OB24 — Every target is probed once, and the probe gives a binary result. Retrying attempts and the success rule do not change the result, they only reduce noise; the measurement therefore takes a single attempt.
• OB25 — The measured latency is the target’s own latency. Contributions from devices along the path are not modeled in this lesson; if they were, the measured values would grow, but the trend below would not change.
• OB26 — The probe list’s order is a fixed sequence coming from the setup, and the lists are nested: the eight-target list contains the whole of the four-target list. If the order changes, which subjects are invisible changes, but the counts stay similar.
• OB27 — Availability is computed only over the probed targets; an unprobed subject enters neither the numerator nor the denominator. The calculation in the field is the same.
• OB28 — The smallest measurable difference across forty subjects is 1/40 = 0.025; in a four-target list, it is 1/4 = 0.25. The indicator’s resolution depends on the list, not the inventory.
• OB29 — The probes are taken as independent: each target’s result comes only from its own state, path sharing is not modeled. If it were, the indicator’s swing would grow.

Measurement

"""End-to-end health: the probe list is a single lever, what is measured is what the indicator fails to represent.

Part 1 - forty subjects; a subject producing a heavy event counts as unreachable.
Part 2 - five probe lists: availability, latency indicator, and the tail.
"""
SEED = 20260812


def rng(seed):
    state = seed % 2147483646 + 1

    def draw(n):
        nonlocal state
        state = (state * 48271) % 2147483647
        return state % n
    return draw


def subjects(count=40, seed=SEED):
    draw, items = rng(seed), []
    for i in range(count):
        items.append({"id": i + 1, "capacity": 20 + draw(81), "latency": 5 + draw(45),
                      "class": ("interactive", "batch", "standby")[draw(3)],
                      "special": draw(9) == 0})
    return items


def events(subs, count=400, seed=SEED + 3):
    draw, items = rng(seed), []
    for i in range(count):
        s = subs[draw(len(subs))]
        items.append({"id": i + 1, "subject": s["id"], "heavy": draw(25) == 0})
    return items


def probe_order(subs, seed=SEED + 11):
    """The probe list's order; its prefixes give nested target sets."""
    draw, remaining, order = rng(seed), [s["id"] for s in subs], []
    while remaining:
        order.append(remaining.pop(draw(len(remaining))))
    return order


subs = subjects()
evts = events(subs)
lat = {s["id"]: s["latency"] for s in subs}
heavy = {s["id"]: 0 for s in subs}
for e in evts:
    if e["heavy"]:
        heavy[e["subject"]] += 1
unreachable = {n for n, c in heavy.items() if c >= 1}
order = probe_order(subs)

print(f"subject {len(subs)} | heavy event {sum(heavy.values())} | unreachable subject "
      f"{len(unreachable)} | true availability "
      f"{100 * (len(subs) - len(unreachable)) / len(subs):.1f}%")
print(f"true latency: average {sum(lat.values()) / len(subs):.1f} ms, "
      f"slowest {max(lat.values())} ms, fastest {min(lat.values())} ms")
print()
print(f"{'targets':>7s} {'unprobed':>8s} {'availability':>12s} {'resolution':>10s} "
      f"{'avg ms':>7s} {'slowest':>8s} {'invisible':>9s} {'heavy missed':>13s}")
for n in (4, 8, 16, 32, 40):
    h = order[:n]
    ok = [x for x in h if x not in unreachable]
    print(f"{n:7d} {len(subs) - n:8d} {100 * len(ok) / n:11.1f}% "
          f"{100 / n:9.1f}p {sum(lat[x] for x in h) / n:7.1f} "
          f"{max(lat[x] for x in h):8d} {len(unreachable - set(h)):9d} "
          f"{sum(c for x, c in heavy.items() if x not in h):13d}")

subject 40 | heavy event 22 | unreachable subject 17 | true availability 57.5%
true latency: average 23.4 ms, slowest 48 ms, fastest 6 ms

targets unprobed availability resolution  avg ms  slowest invisible  heavy missed
      4       36        50.0%      25.0p    22.0       40        15            20
      8       32        37.5%      12.5p    21.6       40        12            15
     16       24        62.5%       6.2p    21.1       40        11            14
     32        8        56.2%       3.1p    26.0       48         3             3
     40        0        57.5%       2.5p    23.4       48         0             0

The Subject Not Represented

With the four-target list, the indicator reads 50.0%. Behind this number are two successful and two failed tests — four observations in total. At the same time, 36 of the forty subjects were never probed at all, 15 unreachable subjects never appear in a single test, and 20 heavy events never enter the indicator at all.

The structure here is the same as the first lesson’s result, read backwards. There, an unreported subject left a gap on the dashboard; here it does not even leave a gap, because the indicator is a single percentage, and a gap does not show inside a percentage. An unprobed subject neither raises nor lowers the indicator; it sits outside the indicator altogether.

At sixteen targets, unprobed subjects fall to 24; at thirty-two targets, to 8. Invisible unreachable subjects do not fall at the same rate: 15, 12, 11, 3. On the third row, doubling the list’s target count only drops the invisible unreachable subjects one at a time — because the targets added to the list are mostly subjects that are already working. Growing the coverage does not shrink the tail proportionally.

The Indicator’s Own Noise

As important as the columns on the right is how the percentage on the left behaves. As the probe list grows, the indicator reads 50.0, 37.5, 62.5, 56.2, and 57.5 in turn. The true value is 57.5 across the entire window and never changed. With nothing happening on the network, the indicator swings between 37.5 and 62.5, a band of twenty-five points.

The reason is written in the resolution column. In a four-target list, a single test’s result moves the indicator by 25 points; at eight targets, 12.5; at forty targets, 2.5 points. If an alert threshold is set below this resolution, the alert measures the list, not the network.

This has a direct operational consequence: an availability ratio computed from a small target list cannot be read as a trend. Seeing 50 and then 75 in two consecutive windows is not an improvement, it is a single subject changing state. The measure’s band is the divisor itself.

What the Average Hides

The latency columns show a third tail. At four targets, the measured average is 22.0 ms; the forty subjects’ true average is 23.4 ms. The gap is small, and the indicator therefore looks correct.

The misleading column is not the average, it is the slowest. Across the four-, eight-, and sixteen-target lists, the measured slowest value is 40 ms in all three. The population’s true slowest subject is 48 ms, and it only shows up in the thirty-two-target list. So up to sixteen targets, the indicator underreports the tail by eight milliseconds — and it does so without giving any sign of it.

The relationship between the average and the slowest also reverses: at thirty-two targets, the average rises to 26.0 ms, because slow subjects have entered the list. The indicator looks like it has gotten worse, but the network has not changed; only the field of view has widened. An indicator getting worse does not mean the thing it measures has gotten worse.

The Probes Are Not Independent

The measurement counts every target as a separate observation and computes the percentage accordingly. In the field, this assumption only partly holds, because the probes all start from the same point and share their paths.

Every target leaving a single vantage point shares the first few devices. If a shared device breaks, every target fails at once, and the indicator drops to 0%. Even if thirty-nine of the forty subjects are sound, the table reports a total outage. The reverse case is subtler: when a device far from the vantage point breaks, only one or two targets go through it, so the indicator drops by only a few points and stays under the threshold.

This is why a percentage measured from a single point does not give information about a location. A drop in the percentage does not mean “this much of the network is broken,” it means “from this point, this much is not visible.” The way to tell locations apart is not to tweak the divisor, but to multiply the vantage points: when the same target list is run from two separate points, targets that fail from only one point show a problem specific to that point, and targets that fail from both show a problem in the target itself.

This distinction was not modeled in the measurement; the probes were taken as independent, and every target’s result came only from its own state. If it had been modeled, the swing in the availability column would have grown, because a shared device’s state would push multiple targets in the same direction at once. The independence assumption makes the indicator look steadier than it is.

Active and Passive Measurement

The measurement above is active measurement: the side doing the measuring produces the probe traffic. This topic’s first three lessons, by contrast, were passive measurement — reading from traffic that already existed, without adding any packets. The two have their tails in opposite places, and this is the reason for reading all four together.

Passive measurement only sees a subject that is talking. A device with no traffic produces no flow record, never enters a capture, and sends no notification; this is why the twenty-six subjects lost in the first lesson were lost. Passive measurement, in exchange, measures real user traffic: the latency it sees is a latency someone actually experienced.

Active measurement does the reverse. Because it produces the probe traffic itself, it can also measure a silent subject; a device placed on the list is probed even if it is never used at all. Its cost is twofold. The latency it measures is a latency nobody experienced — probe packets can differ from the real load’s path, the real load’s size, and the real load’s class. Second, the measurement is bound by the list: active measurement has to know in advance what to measure, while passive measurement can see even what it does not know about.

The two methods have no shared gap, and this is not a coincidence. The subject passive measurement fails to see is the silent one; the subject active measurement fails to see is the one not on the list. A device being both silent and off the list is a separate deficiency, and it comes from the inventory itself. The same result came out of the comparison between the monitoring models: when independent blind spots are laid on top of each other, the tail shrinks, but a subject the inventory does not see appears in neither.

Narrowing the Tail

Three narrowings make it visible who the indicator speaks on behalf of.

First narrowing — write the denominator. A subjects in inventory field is placed on the dashboard alongside target count. With a four-target list, the table says “50%, 4/40 subjects,” and the reader sees what was not measured. This is a correction made without adding any measurement.

Second narrowing — tie the resolution to the threshold. The alert threshold is chosen larger than the list’s resolution. For a four-target list, a threshold smaller than 25 points cannot be defined; if it needs to be defined, the list must be grown. The threshold and the divisor are decided together.

Third narrowing — feed the target list from the previous lessons’ output. A subject never visible in telemetry, a subject never reported in monitoring, and a subject never captured — these are the probe list’s first candidates. All three lessons were counting who is invisible; this lesson says where that list gets written. A list fed this way closes more of the tail while staying smaller than a randomly chosen list.

Summary

• Reachability is about a subject and gives a binary result; availability is a ratio, and the probe list determines its set — the list is a lever.
• With a four-target list the indicator reads 50.0%, while 36 of the forty subjects are never probed, 15 unreachable subjects stay invisible, and 20 heavy events never enter the indicator at all.
• As the list grows, unprobed subjects fall from 36 to 0, but invisible unreachable subjects do not fall at the same rate (15, 12, 11, 3): growing the coverage does not shrink the tail proportionally.
• While the true value is fixed at 57.5%, the indicator swings between 37.5 and 62.5; resolution is 25 points at four targets and 2.5 at forty, and the alert threshold cannot be set below this band.
• The measured slowest value stays at 40 ms through sixteen targets; the true slowest is 48 ms; at thirty-two targets the average rises because the field of view changed, not the network.

Next Step

Every lever touched throughout this topic was a lever of observation. The sampling rate, the timeout threshold, the capture filter, and the probe list — all four decided what would be seen, but none of them touched the network itself. A badly set sampling rate drops a device out of the report; it does not drop a connection. This is where the whole weight of the measurement came from: a subject left in the tail was not harmed, only unseen.

The next topic removes this distinction. What happens when the lever is no longer observation’s but configuration itself? A single rule again applies to all forty subjects at once, but this time the moment it is applied, the devices’ behavior changes, and a subject left in the tail falls out not of the report, but of the service.