Device Monitoring Protocols

The previous lesson read telemetry from the record itself: flow records and counters were produced on the device and went to a collector. Who initiated the record was never asked. Yet collection can be set up in two separate ways, and the choice is itself a lever.

In the first form the manager asks: it walks the devices in order, requests values from each, and writes down the reply. In the second form the device reports: it sends a message to the manager on its own when a threshold is crossed or a state changes. The two models’ round-trip cost and capture latency were measured in the Application Layer Protocols course and will not be repeated here. This lesson’s question is different: on the same forty devices, how many subjects does each model never report — and are the subjects they fail to report the same subjects?

The Two Models’ Message Formats

In poll-based monitoring, the device presents an object tree. Every measurable value has an object identifier, and the manager requests the value by giving the identifier. The tree is ordered, so the manager can also walk a branch it does not know by saying “give me the next one.” In event notification the direction is reversed: the device sends the same object identifier and value without being asked. Notification has two forms, acknowledged and unacknowledged; the difference between them was measured in the previous course and is only mentioned here. A third form sets up a subscription: the manager declares a path and an interval, and the device streams that path on a regular schedule or on every change.

# taught message format , not executed

poll-based monitoring
  manager -> device   query    object identifier 1.3.6.1.2.1.2.2.1.10.3
  device -> manager   reply    value 918233100, type counter32
  manager -> device   next     next branch in the tree

event notification
  device -> manager   notification             object identifier, value, timestamp
  device -> manager   acknowledged notification manager acknowledges, device retries if none arrives

subscription
  manager -> device   subscribe  path /interfaces/interface/counters, interval 10 s
  device -> manager   stream     on every interval or on every change

The format itself does not enter the measurement. What enters the measurement is the setting that decides what each model reports.

Polling’s Budget

In the polling model, the manager walks the list from the start and asks each device in order. The scan has to finish within a period: when the next period begins, the scan starts over from the beginning. This scan amounts to a time budget.

Two things spend the budget. A device that responds takes up time equal to its own latency. A device that does not respond takes up time equal to the timeout — and gives no data at all. The setting in the manager’s hands is this timeout threshold, and it pulls in two directions at once.

If the threshold is kept low, a slow device is cut off before it can answer and goes unreported; in exchange, every cutoff is cheap, and the scan reaches the end of the list. If the threshold is raised, slow devices can answer too, but every answer takes up more time and the budget runs out before the scan reaches the end of the list. In that case, devices at the tail of the list never even get asked. The two causes must be counted separately: a device that times out was asked but did not answer, while a device that is never asked is left out without the manager even knowing about it.

Notification’s Threshold

The notifying model has no scan, and therefore no time budget. In its place, a notification threshold sits on the device: the device does not speak on every event, only when it sees activity crossing the threshold. Without the threshold, the manager would receive every event that falls in the window as a separate message, and that would be a far bigger load than the scan budget.

The silence the threshold produces has two meanings, and the message does not tell them apart. If a device is quiet, either it genuinely did not cross the threshold, or it is in no condition to speak. In polling, silence is at least a silence that was asked for; in notification, it was not even asked.

The measurement’s assumptions:

• OB7 — The forty subjects are the course’s fixed population; each subject has a response latency, and this latency is the query’s time cost.
• OB8 — The scan period budget is 700 ms. The sum of the forty devices’ latencies is above this budget, meaning the scan cannot cover the whole list even in the best case.
• OB9 — The scan walks the list in subject order and starts over from the beginning every period; devices at the tail never enter the round at all. If the order is shuffled, which subjects are invisible changes, but not how many.
• OB10 — A query that times out takes up time equal to the timeout from the budget and returns no value.
• OB11 — The notification threshold is defined over the number of events produced in the window: a device sends a notification only if it produced as many events as the threshold or more, and when it sends, it sends all of its events.
• OB12 — All of a reported device’s heavy events are visible; all of an unreported device’s heavy events are missed. The oracle is the setup, and which event is heavy is known because we wrote it.
• OB13 — In the row comparing the two models, a timeout of 30 ms was chosen for polling and a threshold of 9 events for notification; these two settings produce blind spots of a similar size.

Measurement

"""Device monitoring: polling vs. notification, what is measured is what neither reports.

Part 1 - polling: fixed time budget, lever is the timeout threshold.
Part 2 - notification: lever is the notification threshold (events needed in the window).
Part 3 - intersection of the two models' invisible-subject sets.
"""
SEED = 20260812
SCAN_BUDGET = 700  # ms, one scan period


def rng(seed):
    state = seed % 2147483646 + 1

    def draw(n):
        nonlocal state
        state = (state * 48271) % 2147483647
        return state % n
    return draw


def subjects(count=40, seed=SEED):
    draw, items = rng(seed), []
    for i in range(count):
        items.append({"id": i + 1, "capacity": 20 + draw(81), "latency": 5 + draw(45),
                      "class": ("interactive", "batch", "standby")[draw(3)],
                      "special": draw(9) == 0})
    return items


def events(subs, count=400, seed=SEED + 3):
    draw, items = rng(seed), []
    for i in range(count):
        s = subs[draw(len(subs))]
        items.append({"id": i + 1, "subject": s["id"], "heavy": draw(25) == 0})
    return items


def poll(subs, timeout, budget=SCAN_BUDGET):
    """The scanner walks the list from the start; each query costs as much time as the
    device's latency, an unresponsive device costs as much as the timeout. The scan
    stops when the budget runs out."""
    spent = asked = 0
    seen = set()
    for s in subs:
        cost = min(s["latency"], timeout)
        if spent + cost > budget:
            break
        spent += cost
        asked += 1
        if s["latency"] <= timeout:
            seen.add(s["id"])
    return asked, spent, seen


def notify(subs, evts, threshold):
    """A device sends a notification only if the event count in the window reaches the threshold."""
    counts = {s["id"]: 0 for s in subs}
    for e in evts:
        counts[e["subject"]] += 1
    speaking = {n for n, c in counts.items() if c >= threshold}
    return sum(counts[n] for n in speaking), speaking


def missed(evts, seen):
    return sum(1 for e in evts if e["heavy"] and e["subject"] not in seen)


subs = subjects()
evts = events(subs)
print(f"subject {len(subs)} | event {len(evts)} | heavy event {sum(e['heavy'] for e in evts)}"
      f" | latency {min(s['latency'] for s in subs)}–{max(s['latency'] for s in subs)} ms"
      f" | full scan time {sum(s['latency'] for s in subs)} ms")
print()
print(f"{'timeout':>9s} {'asked':>7s} {'spent ms':>9s} "
      f"{'responded':>10s} {'timed out':>10s} {'never asked':>12s} "
      f"{'heavy missed':>13s}")
poll_seen = {}
for za in (10, 20, 30, 40, 50):
    s, g, seen = poll(subs, za)
    poll_seen[za] = seen
    print(f"{za:9d} {s:7d} {g:9d} {len(seen):10d} {s - len(seen):10d} "
          f"{len(subs) - s:12d} {missed(evts, seen):13d}")
print()
print(f"{'threshold':>10s} {'speaking':>9s} {'messages':>9s} "
      f"{'never reported':>15s} {'heavy missed':>13s}")
notify_seen = {}
for esik in (1, 6, 9, 12, 15):
    msgs, spk = notify(subs, evts, esik)
    notify_seen[esik] = spk
    print(f"{esik:10d} {len(spk):9d} {msgs:9d} {len(subs) - len(spk):15d} "
          f"{missed(evts, spk):13d}")
print()
a, b = poll_seen[30], notify_seen[9]
ya, yb = set(s["id"] for s in subs) - a, set(s["id"] for s in subs) - b
print(f"polling (timeout 30) missed {len(ya)} | notification (threshold 9) missed {len(yb)}")
print(f"missed by both {len(ya & yb)} | missed by only one {len(ya ^ yb)}"
      f" | seen by either {len(subs) - len(ya & yb)}")
print(f"heavy events missed by both together {missed(evts, a | b)}")

subject 40 | event 400 | heavy event 22 | latency 6–48 ms | full scan time 935 ms

  timeout   asked  spent ms  responded  timed out  never asked  heavy missed
       10      40       385         10         30            0            18
       20      40       634         18         22            0             9
       30      36       698         24         12            4             8
       40      29       664         26          3           11             5
       50      29       678         29          0           11             4

 threshold  speaking  messages  never reported  heavy missed
         1        40       400               0             0
         6        37       385               3             1
         9        25       301              15             3
        12        13       184              27            15
        15         5        83              35            16

polling (timeout 30) missed 16 | notification (threshold 9) missed 15
missed by both 6 | missed by only one 19 | seen by either 34
heavy events missed by both together 1

Reading the Polling Table

The sum of the forty devices’ latencies is 935 ms, and the budget is 700 ms. Even with no timeouts at all, the scan cannot cover the whole list; this means the lever’s range of motion is limited from the very start.

At a 10 ms timeout, the scan is cheap: all forty of the forty devices get asked, and a total of 385 ms is spent. But only 10 devices respond — thirty could not fit under the threshold. The report shows a quarter of the forty devices, and 18 heavy events are missed.

When the threshold is raised to 50 ms, timed-out devices fall to 0; now every device that gets asked responds. In exchange, the scan stops at 29 devices and 11 devices never get asked. The number of respondents has risen from 10 to 29, but it never reaches forty.

The 30 ms row sitting in between is where both causes act at once: 12 devices time out, 4 devices never get asked, leaving 24 respondents. The same report carries two separate reasons for invisibility, and on the dashboard both sit as the same gap.

The heavy-events-missed column follows this trend but never falls to zero on any row: 18, 9, 8, 5, 4. Even at the most generous setting, four of the twenty-two heavy events never enter a single query.

Reading the Notification Table

At a notification threshold of 1, no device stays quiet: all forty speak, 0 devices stay invisible, and no heavy event is missed. The cost sits in the column — 400 messages. Polling, at its most expensive row, was firing forty queries; notification here produces ten times that many messages. Notification load depends not on the number of subjects but on the number of events, and this is the two models’ most fundamental structural difference.

When the threshold is pulled down to 9, messages fall to 301 and 25 devices speak; 15 devices are never reported. At threshold 15, the load drops as low as 83 messages, but now only 5 devices are speaking and 35 devices are silent. 16 of the twenty-two heavy events are missed.

What stands out is that the threshold drops the load faster than linearly: as it falls from 400 to 83, visible devices fall from 40 to 5. The threshold keeps the minority that produces many events and drops the rest. The device polling lost was the slow device; the device notification loses is the quiet device.

The Blind Spots Are Not the Same

The last three lines are this lesson’s real conclusion. At a 30 ms timeout, polling never reports 16 devices; at a threshold of 9, notification never reports 15. The numbers are close to each other; but the devices neither one sees number only 6. That is, twenty-five of thirty-one blind spots belong to a single model alone.

This has a direct operational consequence. When the two models run together, the number of visible devices rises to 34, and the heavy events missed by both together fall from 22 to 1. The best polling setting alone was missing 4 heavy events, and the best notification setting only worked at the cost of making all forty devices speak; laid on top of each other, everything becomes visible except a single heavy event.

The reason for this is not the quality of the models but the fact that their blind spots are independent. Polling filters by latency, notification filters by activity; a device being both slow and quiet is a separate coincidence, and it happens in only six of the forty devices. The tail does not disappear — six devices are still absent from every report — but who the tail consists of becomes measurable.

Inventory and Liveness

There is a place where the measurement knows all forty subjects from the start, and this is a convenience the setup grants the operator. In the field, both models need an inventory, but their needs point in opposite directions.

Polling uses the inventory as input: a device not on the list is never asked, so a gap in the inventory turns directly into an invisible subject. There is an upside to this — because the list is known, who did not respond is also known. In the table above, the “timed out” column is exactly this information, and it is the name of a problem.

Notification produces the inventory as output: the manager comes to know the devices that send it messages. If a device not in the inventory speaks, the manager learns about it — something polling cannot do. But the reverse is also true: a device that never speaks never appears in the inventory at all, and its absence does not show up as a problem. As with the ranking list in the previous lesson, a row that does not exist carries no information.

The common mechanism that closes this asymmetry is a heartbeat: the device sends a short message at regular intervals even when it has nothing to say. The manager can now measure silence, because silence has become a missing message. Its cost is a calculable load — forty devices sending a notification once a minute means 2,400 messages an hour, and this load is independent of the event count. Its gain, though, is directly visible in the measurement: at a notification threshold of 15, 35 devices are silent; once a heartbeat is added, all thirty-five of them become visible, carrying not the information of what they did, but that they exist.

Summary

• In poll-based monitoring the manager asks, in event notification the device speaks; the two models’ round-trip cost was measured in the previous course, what is measured here is how many subjects never get reported.
• Polling’s lever is the timeout threshold, and it cuts from two directions at once: when the threshold is low, 30 devices time out; when it is high, 11 devices never get asked; the respondent count is at most 29 within a 700 ms budget.
• Notification’s lever is the threshold, and it ties the load to the event count: at threshold 1, 400 messages with 0 invisible devices; at threshold 15, 83 messages with 35 invisible devices.
• Polling loses the slow device, notification loses the quiet device; because the two models’ blind spots are independent, the intersection of two sets of 16 and 15 is only 6 devices.
• When the two models are used together, visible devices rise to 34 and missed heavy events fall to 1; the tail does not disappear, but who is in it is now known.

Next Step

Both of these models relied on the device’s own account: the device produced both the reply and the notification, and the device decided their content. If a device has a behavior it never reports, there is no way to learn it by asking the device. The next lesson therefore changes the source and looks at the packet itself: once a filter is set up to sift the bytes crossing the wire, how much does the filter fail to see — how many flows does a narrow filter miss, and how many bytes does a wide one?