Managing Devices with Scripts

The previous lesson measured what a single request leaves on a single device. In operations, a request is never singular: a script walks the inventory, connects to forty devices in sequence, applies the same template to each, and writes a report at the end. Once written, the script carries the same decision to all forty of forty devices — this is exactly what the lever is.

This lesson’s question is not how to write the script, it is what the script cannot see. When the template is applied, how many devices fall outside it, how many of these devices appear in the report that gets written, and what decision opens the gap between the two numbers.

What the Script Does

A bulk change is seven steps, and the order of the steps determines the measurement’s result.

# taught bulk-change procedure, not run

1  read the inventory           -> 40 devices
2  dry run the template         -> per-device diff list
3  review the diff              -> flag what falls outside the template
4  apply batch 1                -> first 10 devices
5  read the report               -> any deviation visible
6  if not, batch 2, 3, 4        -> rollout continues
7  if so, stop and roll back    -> only what was applied

The second step is dry run, the loop from four through seven is staged rollout, the second half of seven is rollback. The three are not three separate protections; they are the same protection in three different moments — before applying, while applying, and after applying.

The infrastructure-as-code procedure and tool discussion is the subject of the Infrastructure as Code course and are not repeated here. Here the object is a network device, and what is measured is not the procedure’s efficiency, it is the minority that falls outside the template.

Two Sources of Verification

There are two ways for a script to know a change is going right, and the two do not measure the same thing.

Dry run compares intent against the device’s state. It computes what the template would write, reads the current configuration from the device, takes the difference between the two, and writes to no device. Its coverage is the entire inventory: it looks at all forty of forty devices, because looking requires no indicator.

The report comes from indicators. Metrics collected after applying are read, and devices deviating from expectation are listed. Its coverage is the indicators’ coverage — and the indicator set never sees all forty of forty devices at equal detail. In this setup, the report is built on the interactive class’s indicators; a deviation in another class never appears in it.

This is not a flaw in whoever wrote the report. An indicator set is always a coverage decision: which class’s which metric, collected at what frequency. The question asked when making that decision is “which devices matter”; the question not asked is “what deviation are we becoming unable to see.”

The assumptions the measurement rests on:

• AC19 — The forty-device subject set is the course’s constant and is not changed in this lesson.
• AC20 — The template is written for a device class. A special device whose class does not match deviates; the deviation arises from the template’s scope decision, not from a device fault.
• AC21 — Dry run compares intent against the device’s actual state and writes to no device. It does not use the oracle; it reads deviation from the device’s own state.
• AC22 — The report is built on the interactive class’s indicators. A deviation in another class never appears in it.
• AC23 — Staged rollout batches devices in number order. The report is read at the end of every batch; if a deviation visible in the report exists, rollout stops.
• AC24 — When rollout stops, deviated devices applied up to that point are rolled back; deviated devices in later batches were never touched at all.
• AC25 — When batch size is 40, rollout is a single step, and there is no interval at which a stopping decision could be made.

The Measurement

"""Bulk change: deviated devices constant, visible deviation variable.

Part 1 - the template is applied: matched, deviated, visible and hidden in the report.
Part 2 - dry run against the report's coverage, side by side.
Part 3 - staged rollout: changed and prevented deviations until it stops.
"""
SEED = 20260812
CLASSES = ("interactive", "batch", "standby")


def make_rng(seed):
    d = seed % 2147483646 + 1

    def r(n):
        nonlocal d
        d = (d * 48271) % 2147483647
        return d % n
    return r


def subjects(count=40, seed=SEED):
    r, out = make_rng(seed), []
    for i in range(count):
        out.append({"no": i + 1, "capacity": 20 + r(81),
                    "latency": 5 + r(45),
                    "class": ("interactive", "batch", "standby")[r(3)],
                    "special": r(9) == 0})
    return out


def apply_template(subj, template_class):
    """A single template is applied to forty devices; special devices deviate."""
    matched = deviated = hidden = 0
    for o in subj:
        if o["class"] == template_class or not o["special"]:
            matched += 1
        else:
            deviated += 1
            if o["class"] != "interactive":
                hidden += 1
    return matched, deviated, hidden


def dry_run(subj, template_class):
    """Before applying, intent is compared against the device's state."""
    return [o["no"] for o in subj
            if not (o["class"] == template_class or not o["special"])]


def in_report(subj, template_class):
    """The report is built on the interactive class's indicators."""
    return [no for no in dry_run(subj, template_class)
            if subj[no - 1]["class"] == "interactive"]


def staged(subj, template_class, batch):
    """Applied batch by batch; stops at the first deviation visible in the report."""
    applied = deviated_applied = wave = 0
    for start in range(0, len(subj), batch):
        wave += 1
        visible = 0
        for o in subj[start:start + batch]:
            applied += 1
            if not (o["class"] == template_class or not o["special"]):
                deviated_applied += 1
                if o["class"] == "interactive":
                    visible += 1
        if visible:
            return wave, applied, deviated_applied
    return 0, applied, deviated_applied


subj = subjects()
print(f"subjects {len(subj)} | special devices {sum(o['special'] for o in subj)} | "
      f"class distribution " +
      " ".join(f"{s} {sum(1 for o in subj if o['class'] == s)}" for s in CLASSES))
print()
print(f"{'template class':<14s} {'matched':>5s} {'deviated':>6s} {'visible in report':>16s} "
      f"{'hidden from report':>19s} {'report coverage':>16s}")
for s in CLASSES:
    m, d, h = apply_template(subj, s)
    print(f"{s:<14s} {m:5d} {d:6d} {d - h:16d} {h:19d} "
          f"{(d - h) / d:16.3f}")

print()
print(f"{'template class':<14s} {'dry run finds':>19s} {'report finds':>12s} "
      f"{'found only by dry run':>31s}")
for s in CLASSES:
    k, r = dry_run(subj, s), in_report(subj, s)
    print(f"{s:<14s} {len(k):19d} {len(r):12d} {len(k) - len(r):31d}")
for s in CLASSES:
    print(f"  {s:<12s} deviated devices {dry_run(subj, s)} | "
          f"in report {in_report(subj, s)}")

print()
print(f"{'template class':<14s} {'batch':>6s} {'stopping wave':>13s} "
      f"{'devices changed':>15s} {'to be rolled back':>18s} {'deviations prevented':>21s}")
for s in CLASSES:
    for batch in (4, 10, 40):
        w, a, da = staged(subj, s, batch)
        print(f"{s:<14s} {batch:6d} {w if w else '-':>13} {a:15d} "
              f"{da:18d} {len(dry_run(subj, s)) - da:21d}")

subjects 40 | special devices 9 | class distribution interactive 11 batch 20 standby 9

template class matched deviated visible in report  hidden from report  report coverage
interactive       34      6                0                   6            0.000
batch             34      6                3                   3            0.500
standby           34      6                3                   3            0.500

template class       dry run finds report finds           found only by dry run
interactive                      6            0                               6
batch                            6            3                               3
standby                          6            3                               3
  interactive  deviated devices [7, 12, 20, 25, 26, 40] | in report []
  batch        deviated devices [4, 20, 25, 26, 31, 33] | in report [4, 31, 33]
  standby      deviated devices [4, 7, 12, 31, 33, 40] | in report [4, 31, 33]

template class  batch stopping wave devices changed  to be rolled back  deviations prevented
interactive         4             -              40                  6                     0
interactive        10             -              40                  6                     0
interactive        40             -              40                  6                     0
batch               4             1               4                  1                     5
batch              10             1              10                  1                     5
batch              40             1              40                  6                     0
standby             4             1               4                  1                     5
standby            10             1              10                  2                     4
standby            40             1              40                  6                     0

Deviated Is Constant, Visible Is Variable

The top table gives this topic’s main result. Whichever class the template is written for, matched devices are 34, deviated 6. The same number in all three rows. The deviated count does not depend on which class the template targets, because the source of deviation is not the template being written wrong, it is a single template claiming to represent forty different devices. Writing a better template does not bring this six down to five; only narrowing the scope or writing a second template does, and that means giving up the lever’s singularity.

The three right-hand columns, though, do change. In the template written for the interactive class, deviations visible in the report are 0, hidden 6; report coverage 0.000. In the other two templates, visible is 3, hidden 3, coverage 0.500. The same six devices, three different visibilities.

This is the course’s third claim paid off a second time, and it stands here in its starkest form: automation does not reduce deviation, it changes its visibility. The six deviated devices are there in all three templates; the only thing that changes is how many fall into the report. In operations, only the second of these two numbers gets written to the dashboard, and as the dashboard improves, deviation is assumed to be falling.

Device numbers confirm this. In the interactive template, deviations are [7, 12, 20, 25, 26, 40], and the list visible in the report is empty. In the batch template, deviations are [4, 20, 25, 26, 31, 33], in the report [4, 31, 33] — three visible, three not. In the standby template, the deviated devices are different, but the ones visible in the report are again [4, 31, 33], because the report looks not at deviation but at class.

In a set of forty subjects, the smallest measurable difference is 1/40 = 0.025. The gap between six and three is 0.075, comfortably inside the band.

What Dry Run Finds

The middle table puts the two sources of verification side by side. Dry run finds 6 deviated devices in all three of the three templates. The report finds 0, 3, and 3. The count found only by dry run is, respectively, 6, 3, and 3.

The reason for the gap is the method itself. Dry run does not look at an indicator; it asks every device in the inventory one by one and compares it against intent. Its coverage is not the indicator set’s coverage, it is the inventory’s — and the inventory contains all forty of forty devices. This is why dry run is independent of every blind spot in indicator design.

The second difference is in timing, and it matters more. Dry run finds deviation before any device changes; the report finds it after the change is applied. In the first case, the count of devices to roll back is zero, because there is nothing to roll back. What dry run really buys is not the six devices it finds, it is the moment at which it finds those six.

What Staged Rollout Costs

The bottom table gives what rollout changes with batch size, and the first three rows carry the topic’s harshest result.

Under the interactive template, rollout never stops at any batch size. The stopping wave column reads empty in all three rows; devices changed are 40 in all three, to be rolled back 6, deviations prevented 0. The reason is clear: rollout stops on a deviation visible in the report, and under this template no deviation is visible in the report. Shrinking the batch to four changes nothing, because the problem is not batch size, it is that the signal the stopping decision reads is empty. In a blind spot with no signal, staged rollout is the same thing as unstaged rollout.

In the other two templates, rollout stops in the first wave. Under the batch template, at batch size 4, devices changed are 4, to be rolled back 1, deviations prevented 5; at batch size 10, changed 10, to be rolled back again 1, prevented 5. Under the standby template, to be rolled back is 1 at batch size 4, 2 at batch size 10 — one more deviated device enters the first wave as the batch grows.

The batch-40 rows stand for comparison: applied in a single step, all three templates read changed 40, to be rolled back 6, prevented 0. Staged rollout’s entire gain reads from here — with a working signal it prevents 5 deviations, with none, 0.

Rollback and a Destructive Change

Rollback is not an operation, it is a decision made before the change. If the device’s running configuration was not saved before applying, there is no target to roll back to, and the table’s “to be rolled back” column stays a wish list.

A bulk change is an irreversible operation: a template applied to forty devices erases the previous state on all forty the moment it is applied. A change that empties a device’s configuration, shuts down its interface, cuts off management access, or splits the network in two is not written in runnable form in this course. What can be said is what it does and that it is irreversible: after a change that splits the network, reaching devices on the other half to fix them may not be possible, and those devices are exactly the ones where rollback needs to run.

The counted surfaces and their narrowing follow.

First surface — the scope itself. If the script can touch the entire inventory, a typo goes to forty devices at once. Narrowing: the script’s scope is given explicitly on every run, and the default scope is empty; scope is bounded by a batch list; the apply step does not run until the dry run’s output has been reviewed.

Second surface — the report’s blind spot. In the measurement, all 6 deviations under one template never fall into the report. Narrowing: the stopping decision is not tied to the report alone; at the end of every batch, dry run is re-run for the devices in that batch and compared against intent. This brings the three devices the report cannot see into the stopping decision’s scope as well.

Third surface — the credential. The script connects to forty devices and uses an identity to do it. Credentials, keys, and access tokens are not written in any example in this course; the script carries only a reference and gets the value from its runtime environment, not from inside the template or the configuration repository.

Fourth surface — the irreversible step itself. Narrowing: a step that could be destructive is applied with a rollback timer — if the operator does not confirm within a set period, the device automatically reverts to the previous configuration. This is the only tool left in the hands of an operator who has lost access, and the timer has to be set up before applying.

Summary

• A bulk change is seven steps; dry run, staged rollout, and rollback are the same protection’s before-, during-, and after-application forms.
• Whichever class the template targets, matched devices are 34, deviated 6; the source of deviation is not the template being wrong, it is a single template claiming to represent forty different devices.
• Deviations visible in the report are 0 / 3 / 3, hidden 6 / 3 / 3, report coverage 0.000 / 0.500 / 0.500. The same six devices, three different visibilities: automation does not reduce deviation, it changes its visibility.
• Dry run finds 6 deviations under all three templates, because its coverage is the inventory’s, not an indicator’s; and it finds them before any device changes.
• Under the interactive template, staged rollout never stops at any batch size — changed 40, prevented 0. Under the other two, it stops in the first wave and prevents 5 deviations; in a blind spot with no signal, staged rollout is the same thing as unstaged rollout.

Next Step

Every deviated device counted in this topic was a device whose name was in the inventory, whose configuration we could read, and whose changes we could roll back when needed. Dry run being able to find the six deviations rested on this too: we could ask the device, and the answer it gave was ours. But what if the subject is an object we do not own? The next lesson changes the object under the lever, and asks the same measurement — one decision, forty subjects, the tail left over — again, on a configuration plane that is not ours.