Network Telemetry

The previous lesson built Quality of Service. Classification, prioritization, and shaping were a single lever; the delay one class gained was another class’s tail, and the two numbers were written together.

This lesson asks where those two numbers come from. The lever was managing traffic — but how do we know what the lever is doing? The operator does not have the network itself in hand; what they have are the records the network produces about itself. Telemetry is the collection of these records. The lesson’s question is not what telemetry says, but what it does not say — because the setting that governs collection is itself a lever, and it has a tail of its own.

What Telemetry Collects

The data a network device produces splits into two separate classes, and each answers a different question.

A flow record is a summary. The device counts packets that share the same address–port–protocol five-tuple as a single logical flow and writes one line when the flow ends or a duration elapses: who talked to whom, how long it lasted, how many packets and bytes passed. A flow record names the subject.

A counter is an accumulation. The device keeps values like incoming bytes, outgoing bytes, and dropped packets on each interface by incrementing them; the reader takes the difference between two readings. A counter does not name — it says how much traffic passed, not whose it was. The gauge, which carries an instantaneous value, and the histogram, which carries a distribution, belong to the same class.

The distinction between metric types was established in the Observability and Reliability course, and time-series querying and aggregation in the Observability and Operations course; neither is repeated here. In this lesson the subject is not a service but a network device, and what is measured is not the result of aggregation but the number of subjects that never entered aggregation at all.

The real tension is this: a counter is cheap and misses no event, but it names no event either. A flow record names, but it is expensive — for every flow, the device’s processor, memory, and reporting link do work. Because it is expensive it gets reduced, and the name for that reduction is sampling.

The Shape of a Flow Record

The dump below shows the fields carried by a flow record and an interface counter. This is a format example; it is not a measurement, and it has not been executed.

# taught dump , not executed

flow record
  observing device  device-07
  source            station.example
  destination       example.test
  protocol          TCP
  packets / bytes   143 / 20416
  start             00:00:12.400
  duration          840 ms
  sampling rate     1-in-10

interface counter (cumulative)
  bytes in          918233100
  bytes out         640112940
  packets dropped   1204

The sampling rate field in the record matters: the record itself reports that it is a sample. If the reader misses this field, they take the record for a full count. There is no such field on the counter side, and none is needed — a counter is not sampled.

The Record’s Journey

There are two distances between a record’s production and its reading, and both affect the measurement.

The first is time. A flow record is written when the flow ends; a transfer that runs for minutes produces no line until the moment it finishes. Devices loosen this with an active timeout: once a set duration elapses, an intermediate record is written even if the flow is still running, and the flow continues under the same identity. If no intermediate record is produced, a long flow stays invisible exactly while it is causing trouble; if it is produced, the same flow is represented by multiple lines in the report, and a dashboard that counts flows mistakes it for several flows.

The second is the path. The record travels from the device to the collector over the network — meaning telemetry uses the very network it measures. This has two consequences. When the network degrades, record loss happens at the same moment as the event that was meant to be measured; the minute that needs the most information delivers the fewest records. The second consequence is quieter: if transport is unreliable, a dropped record leaves no trace, because the collector does not know how many records it should expect. The counter diverges again here: a counter is cumulative, and however many readings are missed between two reads, the difference stays correct.

Sampling Is a Lever

The sampling rate is a single number. The operator sets it once, and the setting applies to forty subjects at once. The lever’s tail — the minority it serves wrongly — shows up here in two separate forms. The first is a missed heavy event: a problem that is never known because it was never recorded. The second is subtler: an invisible subject. If a device does not appear in even one line of the report, there is neither a high value nor a low value for it on the dashboard; there is no value at all, and the gap looks the same color as health.

The measurement’s assumptions:

• OB1 — The forty subjects are the course’s fixed population and are not changed in this lesson; subjects differ from one another in capacity, latency, and class.
• OB2 — Four hundred events pass in the observation window. Every event belongs to a subject, and a portion of the events is marked heavy; the heaviness comes from the setup itself.
• OB3 — The oracle is the setup: we know which event is heavy and which subject it belongs to because we wrote them. The measurement does not use this information, it only compares the result against it.
• OB4 — Sampling is deterministic: an event whose sequence number is divisible by the one-in-N value is recorded. Random sampling would give a different record set, but it would not change the record count or the trend below.
• OB5 — Every flow record is a fixed 64 bytes. The bytes column shows the relative size of the record volume; it is not a network measurement, it is a multiple of the record count.
• OB6 — The estimate is made with this rule: the number of observed heavy events is multiplied by the sampling rate. This is the usual path taken when generalizing from a sample to the total.

Measurement

"""Network telemetry: the sampling rate is a single lever, what is measured is the tail.

Part 1 - forty subjects and four hundred events.
Part 2 - five sampling rates: records, observed heavy events, missed, invisible subjects.
"""
SEED = 20260812
RECORD_BYTES = 64


def rng(seed):
    state = seed % 2147483646 + 1

    def draw(n):
        nonlocal state
        state = (state * 48271) % 2147483647
        return state % n
    return draw


def subjects(count=40, seed=SEED):
    draw, items = rng(seed), []
    for i in range(count):
        items.append({"id": i + 1, "capacity": 20 + draw(81), "latency": 5 + draw(45),
                      "class": ("interactive", "batch", "standby")[draw(3)],
                      "special": draw(9) == 0})
    return items


def events(subs, count=400, seed=SEED + 3):
    draw, items = rng(seed), []
    for i in range(count):
        s = subs[draw(len(subs))]
        items.append({"id": i + 1, "subject": s["id"], "heavy": draw(25) == 0})
    return items


def sample(evts, one_in):
    """One in every `one_in` events is recorded."""
    return [e for e in evts if e["id"] % one_in == 0]


def visible(evts, one_in):
    """Record count, observed heavy events, missed heavy events, observed subject set."""
    k = sample(evts, one_in)
    heavy = sum(1 for e in k if e["heavy"])
    return len(k), heavy, sum(1 for e in evts if e["heavy"]) - heavy, {e["subject"] for e in k}


subs = subjects()
evts = events(subs)
print(f"subject {len(subs)} | event {len(evts)} | heavy event {sum(e['heavy'] for e in evts)} | "
      f"event-producing subject {len({e['subject'] for e in evts})}")
print()
print(f"{'1-in-N':>7s} {'records':>7s} {'bytes':>6s} {'heavy seen':>10s} "
      f"{'heavy missed':>12s} {'heavy estimate':>14s} {'invisible subjects':>19s}")
for one_in in (1, 2, 5, 10, 20):
    k, heavy, missed, seen = visible(evts, one_in)
    print(f"{one_in:7d} {k:7d} {k * RECORD_BYTES:6d} {heavy:10d} {missed:12d} "
          f"{heavy * one_in:14d} {len(subs) - len(seen):19d}")

seen10 = {e["subject"] for e in sample(evts, 10)}
counts = {s["id"]: sum(1 for e in evts if e["subject"] == s["id"]) for s in subs}
missing = [n for n in counts if n not in seen10]
print()
print(f"1-in-10 | invisible subjects {len(missing)}"
      f" | average events: invisible "
      f"{sum(counts[n] for n in missing) / len(missing):.2f}, "
      f"visible {sum(counts[n] for n in counts if n in seen10) / len(seen10):.2f}")
print(f"invisible subject's highest event count {max(counts[n] for n in missing)}"
      f" | lowest-event subject {min(counts.values())} events"
      f" | highest-event subject {max(counts.values())} events")

subject 40 | event 400 | heavy event 22 | event-producing subject 40

 1-in-N records  bytes heavy seen heavy missed heavy estimate  invisible subjects
      1     400  25600         22            0             22                   0
      2     200  12800          9           13             18                   1
      5      80   5120          7           15             35                   8
     10      40   2560          2           20             20                  17
     20      20   1280          2           20             40                  26

1-in-10 | invisible subjects 17 | average events: invisible 8.18, visible 11.35
invisible subject's highest event count 13 | lowest-event subject 5 events | highest-event subject 18 events

Reading the Numbers

The first line gives the measurement’s baseline. At full count, records are 400, observed heavy events 22, missed 0, invisible subjects 0. All forty of the forty subjects produced at least one event; invisibility, then, is not a property of the population but a result the lever produces.

Missed heavy events. At one-in-two, records fall by half and observed heavy events drop from 22 to 9: 13 heavy events are missed. The disproportion here is notable — while records fall by half, heavy-event visibility falls by more than half. The reason is that heavy events are few: how twenty-two of them land against a fixed-stride filter is left to chance. At one-in-ten, observed heavy events fall to 2 and 20 heavy events are missed. At one-in-twenty, records fall by half again, but observed heavy events stay at 2; the decline is not smooth, because what is being counted is no longer statistics but coincidence.

The estimate misses. The usual path from sample to total is multiplying the observed count by the sampling rate. The column gives this estimate: while the true value is 22, the estimate comes out to 22, 18, 35, 20, and 40 in turn. At one-in-five the estimate is one and a half times the truth; at one-in-twenty, nearly double. If the same multiplication were applied to the record count, it would give exactly 400 on every line — because the total is large and evenly distributed. The rule is this: sampling preserves the total, not the minority. A heavy event is a minority, and the minority is exactly what the measurement is after.

Invisible subjects. The last column is this lesson’s main number. At one-in-two, 1 subject is absent from every line of the report; at one-in-five, 8; at one-in-ten, 17; at one-in-twenty, 26. Twenty-six of the forty subjects — more than two-thirds — are entirely absent from the dashboard.

The last two lines say who this vanishing touches. At one-in-ten, the 17 invisible subjects’ average event count is 8.18; the visible twenty-three’s is 11.35. The loss is not random: a subject that talks less falls out first. But the filtering is not a sharp threshold either — one of the invisible subjects has 13 events, meaning a device that produced more than twice the traffic of the population’s least active subject (5 events) could still drop out of the report. The loss is both biased and coincidental, and because both are at play, the operator cannot estimate who is invisible just by looking at the report.

The result is the course’s third claim: the lever renders the tail invisible. The sampling rate is set as a performance dial, and it genuinely does bring the record volume down from 25,600 bytes to 1,280 bytes. But the same setting also decides which subject gets reported, and nobody made that decision explicitly. The smallest measurable difference across forty subjects is 1/40 = 0.025; 26 subjects is twenty-six times that, and well inside the measurement band.

The Gap in the Dashboard

The most dangerous thing about an invisible subject is that the dashboard does not show it as missing. The table format below makes this concrete; it is not a measurement, and it has not been executed.

# taught dashboard format , not executed

top traffic-producing devices    (window: last 5 min, source: flow record)
  rank  device      flows  bytes       heavy event
     1  device-31     14   1.2 GB              1
     2  device-07     12   0.9 GB              0
     3  device-22     11   0.8 GB              1
   ...
    23  device-16      1   0.1 GB              0

  total devices listed : 23

The table lists twenty-three devices and is correct — every listed line rests on a record that was genuinely produced. Nothing in it is wrong. What is missing is that nowhere does it say the table was produced from a population of forty devices. The total devices listed field says 23; there is no field called devices in inventory. A ranking list can only rank what produced data, because the input to ranking is the record, and a device with no record never enters the ranking at all.

The distinction is this: a zero-valued row is a piece of information, a row that does not exist is not. The first says “no traffic passed from this device”; the second says nothing at all, and the reader confuses it with the first. Until the inventory’s total count is written onto the dashboard, this distinction stays invisible.

Narrowing the Invisibility

The sampling lever’s tail cannot be removed — if records are being reduced, something will go unseen. What the tail consists of can, however, be chosen.

First narrowing — take heavy events outside sampling. An event marked heavy is recorded regardless of the sampling rate. This brings missed heavy events down to 0 and raises the record volume by only as many as there are heavy events; at one-in-twenty, that is 42 records instead of 20. Its cost is that heaviness must be determined on the device — the device has to know what counts as heavy.

Second narrowing — at least one record per subject. The sampling rate is kept, but at least one record is made mandatory for every subject in the window. Invisible subjects become 0, and record volume grows by at most forty records. This buys visibility from coverage, not volume.

Third narrowing — keep the counter alongside the flow record. A counter is not sampled. Even if the flow record never shows a subject, that subject’s interface counter keeps being read and reports the presence of traffic. A counter does not say who spoke, but it says someone spoke; reading the two sources together lets silence and invisibility be told apart.

What the three share is that they take the decision out of being implicit. Left alone, the sampling rate applies a visibility policy anyway; the difference is that the policy is unwritten.

Summary

• Telemetry comes from two sources: the flow record names the subject but is sampled because it is expensive, the counter is not sampled but does not name the subject.
• The sampling rate is a single lever applied to forty subjects at once; dropping from one-in-one to one-in-twenty takes records from 400 to 20, and record volume from 25,600 bytes to 1,280 bytes.
• Observed heavy events fall from 22 to 2 and 20 heavy events are missed; the decline is not smooth, because a heavy event is a minority and how it lands against the filter is left to chance.
• An estimate made from a sample preserves the total but not the minority: while the truth is 22, the estimate comes out to 22 / 18 / 35 / 20 / 40.
• Invisible subjects rise from 0 to 26; the lever is set as a performance dial, but it also chooses who gets reported, and the gap is the same color as health on the dashboard.

Next Step

In this lesson, the device produced the records and the collector received them; who spoke when was never asked. Yet collection can be set up in two separate ways: the collector asks devices, or the device reports on its own. The two models’ round-trip cost was measured in the Application Layer Protocols course and will not be repeated here. The next lesson takes up the same distinction with this course’s measure: in poll-based monitoring and event notification, how many subjects never get reported, and are the subjects the two models miss the same subjects?