Adding and Expanding a Disk

Throughout this topic, storage was always something that already existed: a device, partitions on it, file systems inside them, mount points above them. The course’s last lesson returns to the start of the chain and follows a device joining the system from start to finish.

What is followed is not the steps themselves but the gap between them. At every step a tool says “done,” and a layer above is not yet aware of it. The number to be measured is how many readings of the chain give a wrong “the work is done” diagnosis.

The Chain’s Links

When a new device is physically attached, the system does not use it by itself. There are nine steps in between, and every step depends on the one before it.

The device must first be recognized: the kernel scans the bus and adds the device to its tree. Then it is partitioned; the partition table gets written to the device, but the kernel can keep holding the old table in memory, and a reread is needed. After that, a file system is built on the partition. The built file system is not visible in the tree; it needs to be mounted. Because a manual mount is lost on restart, a persistent mount entry is written. Finally, the unit that will produce the data must start writing to the new path; the path changing does not by itself change the unit’s configuration.

• ST38 — The chain is nine steps, and the order of steps is fixed in this measurement; the persistent mount entry is written before the manual mount, because this is common practice.
• ST39 — The oracle is the state “the new space is persistently in use”: the file system is mounted, the entry is persistent, and the unit is writing to the new path. Until all three hold at once, the work is not done.
• ST40 — Five tools are read, and each sees a single link in the chain: lsblk the device, blkid the file system, findmnt and df the mount, entry checking the persistence.
• ST41 — In the verification boot, the discovery order is drawn from the shared definition’s generator; 40 boots are observed, and the entry is written in two forms.

"""Chain for joining a new device to the system: what each tool says at every step."""
SEED = 20260218
BOOTS = 40
FIELDS = ("visible", "partition", "kernel_read", "fs", "mounted", "volume_writes",
          "persistent")
STEP = (
    ("0 device attached",          0, 0, 0, 0, 0, 0, 0),
    ("1 rescanned",                1, 0, 0, 0, 0, 0, 0),
    ("2 partition created",        1, 1, 0, 0, 0, 0, 0),
    ("3 kernel read table",        1, 1, 1, 0, 0, 0, 0),
    ("4 file system built",        1, 1, 1, 1, 0, 0, 0),
    ("5 persistent entry written", 1, 1, 1, 1, 0, 0, 1),
    ("6 manually mounted",         1, 1, 1, 1, 1, 0, 1),
    ("7 unit writes to new path",  1, 1, 1, 1, 1, 1, 1),
    ("8 restarted",                1, 1, 1, 1, 1, 1, 1),
)
TOOL = {"lsblk": "visible", "blkid": "fs", "findmnt": "mounted",
        "df": "mounted", "entry check": "persistent"}


def state(a):
    d = dict(zip(FIELDS, a[1:]))
    d["oracle"] = d["mounted"] and d["persistent"] and d["volume_writes"]
    return d


print("step                       ", "  ".join(f"{k:>14s}" for k in TOOL), " oracle")
for a in STEP:
    d = state(a)
    print(f"{a[0]:28s}", "  ".join(f"{str(bool(d[v])):>14s}" for v in TOOL.values()),
          f" {str(bool(d['oracle'])):>6s}")
print()
print("tool             correct reads  wrong reads")
total = 0
for tool, field in TOOL.items():
    y = sum(bool(state(a)[field]) != bool(state(a)["oracle"]) for a in STEP)
    total += y
    print(f"  {tool:14s} {len(STEP) - y:11d}  {y:12d}")
print("wrong diagnosis:", total, "/", len(STEP) * len(TOOL))
print()


def generator(seed):
    d = seed

    def next_value(n):
        nonlocal d
        d = (d * 1103515245 + 12345) % 2147483648
        return d % n
    return next_value


def verify(seed, entry_format, device_count=5, count=BOOTS):
    """Persistent entry written to a DEVICE NAME shifts when discovery order changes."""
    r = generator(seed)
    correct = 0
    for _ in range(count):
        remaining, order = list(range(device_count)), []
        while remaining:
            order.append(remaining.pop(r(len(remaining))))
        correct += 1 if entry_format == "id" else order[1] == 1
    return correct


for seed in (SEED, 20260219):
    print(f"seed {seed}: {BOOTS} boots |"
          f" entry written to name correct {verify(seed, 'name'):3d}"
          f" | entry written to id correct {verify(seed, 'id'):3d}")

step                                 lsblk           blkid         findmnt              df     entry check  oracle
0 device attached                     False           False           False           False           False   False
1 rescanned                            True           False           False           False           False   False
2 partition created                    True           False           False           False           False   False
3 kernel read table                    True           False           False           False           False   False
4 file system built                    True            True           False           False           False   False
5 persistent entry written             True            True           False           False            True   False
6 manually mounted                     True            True            True            True            True   False
7 unit writes to new path              True            True            True            True            True    True
8 restarted                            True            True            True            True            True    True

tool             correct reads  wrong reads
  lsblk                    3             6
  blkid                    6             3
  findmnt                  8             1
  df                       8             1
  entry check              7             2
wrong diagnosis: 13 / 45

seed 20260218: 40 boots | entry written to name correct  10 | entry written to id correct  40
seed 20260219: 40 boots | entry written to name correct   6 | entry written to id correct  40

Three numbers side by side. The oracle: the work finishes at the seventh step; in none of the states before that is the new space persistently in use. The tool’s output: lsblk shows the device from the first step onward, and what it shows is correct — just not the answer to the question being asked. Wrong diagnosis: thirteen of forty-five readings say the work is done.

The distribution shows the chain’s shape. The lowest-level tool is wrong the most: lsblk six times, blkid three times. The topmost tools are wrong the least, but not zero — findmnt and df once each, because at the sixth step the file system is mounted and the unit is still writing to the old path. Entry checking is wrong twice, because the persistent entry is written before mounting and the file is correct on paper. No tool sees the whole chain, and each tells the truth about its own link.

This distribution’s practical consequence is an ordering rule: verification starts not from the bottom of the chain but from the top. First it is asked whether the unit is really writing to the new path, then whether the mount is persistent, and last whether the device is visible; asked in reverse order, the first three answers come back positive and the questioning stops there.

The last two lines measure the chain’s eighth step. In the verification boot, if the entry is written to the device name, the correct file system mounts in only ten of forty boots; in the second seed, this number drops to six. With the entry written to the identity, forty of forty are correct. A chain not verified by a restart does not count as complete.

Irreversible Steps

Three of the chain’s nine steps are irreversible, and all three gather in the same place: the steps that write to the device. Writing the partition table erases the old layout, building a file system erases everything on the partition, and formatting as swap does the same thing. The common risk of all three is choosing the wrong target, and none of them asks “are you sure.” This lesson does not give these commands in a runnable, complete form.

The safe path is four steps, and all four are measurable. The first is verifying the target by its identity: in lsblk and blkid‘s output, it is read that the target device is empty, has no mounted file system on it, and is the expected size. The second is a dry run: the option that shows partitioning tools’ result without applying it, and the table’s text dump, print out beforehand what the change will be. The third is testing on a separate device: mounting a file as a loop device makes it possible to run the whole chain away from real data. The fourth is a snapshot: if the change is on a logical volume, a snapshot taken beforehand makes recovery possible.

The expansion direction follows the same rule too, and it has an added ordering. When a device is grown, the kernel does not see the new size by itself; the logical volume cannot grow before the partition grows, the file system cannot grow before the logical volume grows. The order runs upward, and it reverses on shrinking. What comes out when it is not reversed is not an error message but a silent data loss.

Summary

• Joining a new device to the system is a nine-step chain: recognition, partitioning, rereading the table, building the file system, the persistent entry, mounting, and directing the unit to the new path.
• Five tools see five separate links of the chain, and none sees the whole; thirteen of forty-five readings say the work is done when it is not. The lowest-level tool is wrong the most: lsblk six, blkid three times.
• The chain does not count as complete until it is verified by a restart: an entry written to the device name mounts the correct file system in ten of forty boots, in six in the second seed; with the entry written to the identity, forty of forty are correct.
• Three of the chain’s steps are irreversible, and none of them asks for confirmation; the safe path is verifying the target by its identity, reading the dry-run output, testing on a separate device, and taking a snapshot.
• Expansion is done bottom to top, shrinking top to bottom; reversing the order produces not an error message but silent data loss.

Course Wrap-Up

This course asked one question twenty-two times: how big is the gap between what the output says and the system’s reality. In every lesson, three numbers stood side by side — the oracle, what the tool showed, and how many times the diagnosis drawn from that showing was wrong. The course’s rule was this: a command’s number is not the row it shows, but how many times the diagnosis drawn from those rows is wrong.

The table below collects the measure of the twenty-two lessons in one place. The rows left blank are not made up: every row is written only from its own source lesson, and those lessons were not produced in this batch.

Lesson	Oracle	Tool’s Output	Wrong Diagnosis
Listing and Searching Processes	14 of 24 processes are heavy	interval 1/5/15/30/60 → 14, 14, 11, 9, 9	0, 0, 3, 5, 5; lifetime-average column 7
Foreground and Background Jobs	of 13 jobs, & keeps 0 alive, detaching keeps 4 alive	jobs 13 lines	& 13, detaching 9, the path that severs the hangup signal 0
Signals	2 accept the request, 5 resist; with the forcing signal 7/7	exit code 0 on every attempt	wait 0/10/30 seconds → 7, 6, 5
Process Priorities	total work 237,906 → 238,244, load steady at 3.97	NI column changed 6/6	6/6; 94 percent of the freed share goes to the other heavy processes
Resource Limits	8 failure causes	3 messages + 1 silence, each message maps to two causes	message alone 21, limit alone 29, combined 13
The Service Manager Model	6 units, 7 transitive dependencies	flat 0 / direct 4 / transitive 7 / ordered 9 lines	3 / 2 / 0 / 2 — the reading that prints the most lines does not give the best diagnosis
Service Status and Control	6 of 6 units failed, 60 restarts	4 units read “active,” hidden time 600	4/6; polling 1/5/15/30/60 → 480/99/33/18/10
Defining a New Service	60 crashes, 9 of 9 combinations failed	5 “active,” 2 “exited,” 2 “failed”	7/9; start limit none/20/10/5/3 → hidden time 600/600/106/62/41
The Boot Process	ingest → 4 affected, 5 total impacted	“1 unit”; 4 units never tried	3/6
Bootloaders	11 of 24 changes work	generator output 24, command line reading 21, behavior test 11	13 / 10 / 0; remaining log lines are 0 in both
Log Architecture	86 errors in 4000 lines, real source 31 with process	six unit files: 28, 0, 31, 0, 27, 0	5/6; three files completely silent
Authentication Logs	41 attack lines in 601 lines	four filters: 153, 67, 51, 43 lines	114 / 28 / 12 / 2
Log Filtering and Rotation	86 errors, the first at the 6.1th second	kept after rotation: 86, 42, 19, 12, 5	of six questions 0/3/4/4/4; ratio 0.2209 and 0.0581
System Health Metrics	instantaneous load crosses the threshold for 54 seconds	the averaged band 6.18–6.25 never crosses the threshold	54; false alarm 0
Tracing Performance Problems	18 events (4/4/10)	five snapshots: 432, 1728, 432, 54, 54 lines	12 / 12 / 8 / 2 / 0 — the diagnosis does not change when lines quadruple
Block Devices and Partitions	free space in four pieces, largest contiguous 2,881,536 sectors	partition list 3,702,784 free sectors	6 / 12 decisions
File Systems	54 reads touching a corrupted block	0 errors in a design with no checksum	54 silent reads
Inodes and Metadata	write fails at 32,768 files	block usage 0.0125	2 / 12 warnings
Mounting and Persistent Mounts	717 files really stale	1504 files look stale	787 wrong deletions
Logical Volume Management	writable space 296 blocks	lvs 4392, vgs 8488 blocks	7 / 21 readings
Swap Space	memory pressure 84 seconds	used swap sits at 533	272 false alarms
Adding and Expanding a Disk	work finishes at the seventh step	lsblk shows the device from the first step	13 / 45 readings

The Storage topic’s seven lessons showed one pattern over and over. Tools do not lie; each tells the truth about its own layer, and the question asked is not that layer’s question. A partition list is not a gap map, a usage percentage is not writability, a logical volume’s size is not a file system’s size, used swap is not memory pressure. The wrong diagnosis is born in these gaps, and its count does not shrink by adding a column: in the inode measurement, all three separate warning rules gave two wrong readings each in twelve cases, only the type of the error changed.

A second pattern concerned time. The snapshot filled at the four-hundred-thirteenth second, and monitoring that looked once every sixty seconds saw a number that had not budged until then; when the swap rate fell between two looks, the missed seconds rose from 2 to 49. Evidence disappears on its own, and in storage this happens through a filled area or a dropped snapshot leaving no trace behind. One who looks late sees little.

Throughout the course, destructive commands were not written in complete form, and this was not an omission but part of the measurement. What the commands that write the partition table, build a file system, delete a logical volume, and format swap have in common is that they produce no warning at all when the target is chosen wrongly. The safe path was the same four steps in every lesson: verifying the target by its identity, reading the dry-run output, doing the trial on a separate device, and leaving a way back.

The next course, Linux Network Administration and Troubleshooting, carries the same question outside a single machine. There too is an interface list, and an address appearing in the list does not mean a connection can be established; there too is a routing table, and a row in the table does not mean a packet reaches its destination. The layered-diagnosis method is this course’s storage-chain layering carried over onto the network: a sequence of queries narrowing from the physical link toward the application, and at every layer a tool saying “correct here, not above.” And there, evidence disappears even faster, because a packet passes through and leaves, sitting nowhere unless it is captured.