Structure and Record Examination

The previous lesson took events as ready-made records: each one had a type, a source, and a time. A significant part of the trace collected in an examination does not arrive ready-made like that. The system writes to structures it keeps for its own operation — a configuration store entry, a file system metadata record, a shortcut, a cache entry. None of these was written to serve as a log, but each leaves a trace that can be ordered and read.

This lesson treats those traces in terms of evidentiary value, and the unit of measure is a ratio: the number of clean machines carrying the same trace. The rarer a trace is, the more it distinguishes a claim; if a trace is already present on most machines, it does not distinguish the claim at all. Most sentences that go into a report sit somewhere between these two, and where they sit cannot be read unless it is written down. The fleet, the base numbers, and the claim list are fictional.

• FE25. A system structure is a data structure that is not kept as an event record but leaves a trace because the system’s operation requires it.
• FE26. A structure is in the direct class only if it comes into being solely because the claimed event occurred. If it can also come into being for other reasons, it is in the inference class.
• FE27. The base is the number of clean machines carrying the same trace. The fictional fleet is 268 machines; the base values are hand-assigned, and no real fleet was scanned.
• FE28. A claim’s distinctiveness is the expected number of clean machines carrying the same combination of traces. Below one is distinctive, between one and ten is weak, above ten is noise.
• FE29. The expected number is calculated under the assumption that the traces are independent of one another. This assumption does not actually hold; the calculation is therefore a lower bound.
• FE30. That the structures are present on the event machine is a given. This lesson measures not the finding, but the value of what is found.

Structure, Question, and Base

// forensics/structure.mjs — traces from system structures and the fleet base MODEL.
// Base numbers are hand-assigned fictional values; no real fleet was scanned.

export const FLEET = 268;                    // number of machines counted as clean

// structure -> [class, machines carrying the same trace in the clean fleet, question it answers]
export const STRUCTURE = {
  "deleted-file-name": ["direct", 4, "the file existed on this machine"],
  "recycle-bin-metadata": ["direct", 47, "the file was deleted"],
  "application-cache": ["direct", 9, "the binary was executed"],
  "scheduled-task-definition": ["direct", 6, "the item runs automatically"],
  "configuration-store-entry": ["direct", 12, "the item runs at startup"],
  "removable-media-record": ["direct", 22, "the media was connected to this machine"],
  "installation-record": ["direct", 63, "the software was installed"],
  "shortcut-record": ["inference", 31, "the file was accessed"],
  "recent-items-list": ["inference", 58, "the file was listed"],
  "file-timestamp": ["inference", 96, "the file changed within the window"],
  "share-directory-entry": ["inference", 74, "a network share was connected"],
  "user-profile-stamp": ["inference", 187, "the user logged in"],
};

// Claim -> structures it rests on. Claims are sentences intended for the report.
export const CLAIM = {
  "the file existed on the machine and was deleted": ["deleted-file-name", "recycle-bin-metadata"],
  "the item runs on every startup": ["configuration-store-entry", "scheduled-task-definition"],
  "the binary was executed on this machine": ["application-cache", "shortcut-record"],
  "data was copied to removable media": ["removable-media-record", "file-timestamp",
    "recent-items-list"],
  "the software was installed on the machine": ["installation-record", "application-cache"],
  "a user opened the file": ["recent-items-list", "shortcut-record",
    "user-profile-stamp"],
  "this user was logged in at that time": ["user-profile-stamp"],
};

// Expected clean-machine count carrying the same trace combination (traces assumed independent).
export const expected = (basis) =>
  FLEET * basis.reduce((a, s) => a * STRUCTURE[s][1] / FLEET, 1);
export const directCount = (basis) =>
  basis.filter((s) => STRUCTURE[s][0] === "direct").length;
export const classify = (b) => (b < 1 ? "distinctive" : b < 10 ? "weak" : "noise");

The third field in the table carries the question each structure answers, and this is the field most often confused when writing a claim. The deleted-file-name record answers “the file existed on this machine,” not “who brought the file here.” Recycle-bin metadata says “the file was deleted,” not “the file was hidden.” The application cache says “the binary was executed,” not “the binary did such-and-such.” Any sentence that goes beyond the question a structure answers is an extension with no basis, and it cannot be written into any column of the table.

The difference between the two classes looks at when the structure comes into being. An application cache entry does not come into being unless the binary was executed; the entry’s presence directly shows the execution. A file timestamp, on the other hand, changes when the file is modified, when it is copied, and when it is restored from backup; the timestamp falling within the event window is an inference. The same distinction applies to the shortcut record: a shortcut can come into being both when a user opens the file and when a script touches it.

A Claim’s Distinctiveness

// forensics/value.mjs — base per structure, distinctiveness per claim, and dependence on a single structure.
import { FLEET, STRUCTURE, CLAIM, expected, directCount, classify } from "./structure.mjs";

const y = (g, ...s) => console.log(s.map((v, i) =>
  (g[i] < 0 ? String(v).padEnd(-g[i]) : String(v).padStart(g[i]))).join("").trimEnd());
const structures = Object.keys(STRUCTURE), claims = Object.keys(CLAIM);

console.log(`model: ${structures.length} structures, ${claims.length} claims, a ${FLEET}-machine ` +
  `clean fleet; ${structures.filter((k) => STRUCTURE[k][0] === "direct").length} structures direct, ` +
  `${structures.filter((k) => STRUCTURE[k][0] === "inference").length} structures inference`);

const A = [-28, -10, 8, 7, 16];
console.log("");
y(A, "structure", "class", "base", "share", "claims");
for (const k of structures)
  y(A, k, STRUCTURE[k][0], STRUCTURE[k][1], `${Math.round(STRUCTURE[k][1] * 100 / FLEET)}%`,
    claims.filter((i) => CLAIM[i].includes(k)).length);

const B = [-49, 8, 10, 15, 14];
console.log("");
y(B, "claim", "basis", "direct", "clean machine", "result");
for (const i of claims) {
  const b = expected(CLAIM[i]);
  y(B, i, CLAIM[i].length, directCount(CLAIM[i]), b.toFixed(2), classify(b));
}

const unsupported = claims.filter((i) => directCount(CLAIM[i]) === 0);
console.log(`\nclaims without a direct basis: ${unsupported.length} (${unsupported.join("; ")})`);
console.log(`distinctive ${claims.filter((i) => expected(CLAIM[i]) < 1).length}, ` +
  `weak ${claims.filter((i) => classify(expected(CLAIM[i])) === "weak").length}, ` +
  `noise ${claims.filter((i) => classify(expected(CLAIM[i])) === "noise").length}`);

// Dependence on a single structure: which claim changes class if a structure is removed.
console.log("");
const C = [-28, 12, 19, 14];
y(C, "if structure removed", "affected", "left unsupported", "class dropped");
for (const k of structures) {
  const affected = claims.filter((i) => CLAIM[i].includes(k));
  if (!affected.length) continue;
  const orphaned = affected.filter((i) => CLAIM[i].length === 1);
  const dropped = affected.filter((i) => CLAIM[i].length > 1 &&
    classify(expected(CLAIM[i].filter((z) => z !== k))) !== classify(expected(CLAIM[i])));
  y(C, k, affected.length, orphaned.length, dropped.length);
}

model: 12 structures, 7 claims, a 268-machine clean fleet; 7 structures direct, 5 structures inference

structure                   class         base  share          claims
deleted-file-name           direct           4     1%               1
recycle-bin-metadata        direct          47    18%               1
application-cache           direct           9     3%               2
scheduled-task-definition   direct           6     2%               1
configuration-store-entry   direct          12     4%               1
removable-media-record      direct          22     8%               1
installation-record         direct          63    24%               1
shortcut-record             inference       31    12%               2
recent-items-list           inference       58    22%               2
file-timestamp              inference       96    36%               1
share-directory-entry       inference       74    28%               0
user-profile-stamp          inference      187    70%               2

claim                                               basis    direct  clean machine        result
the file existed on the machine and was deleted         2         2           0.70   distinctive
the item runs on every startup                          2         2           0.27   distinctive
the binary was executed on this machine                 2         1           1.04          weak
data was copied to removable media                      3         1           1.71          weak
the software was installed on the machine               2         2           2.12          weak
a user opened the file                                  3         0           4.68          weak
this user was logged in at that time                    1         0         187.00         noise

claims without a direct basis: 2 (a user opened the file; this user was logged in at that time)
distinctive 2, weak 4, noise 1

if structure removed            affected   left unsupported class dropped
deleted-file-name                      1                  0             1
recycle-bin-metadata                   1                  0             1
application-cache                      2                  0             2
scheduled-task-definition              1                  0             1
configuration-store-entry              1                  0             1
removable-media-record                 1                  0             1
installation-record                    1                  0             0
shortcut-record                        2                  0             1
recent-items-list                      2                  0             1
file-timestamp                         1                  0             0
user-profile-stamp                     2                  1             0

The last column ranges from 0.27 to 187, and this range is the lesson’s result. The sentence at the bottom — “this user was logged in at that time” — rests on a single structure, and that structure’s base is 187 of the 268 machines. The sentence is very probably true and distinguishes nothing: seven out of every ten machines in the fleet carry the same trace. A claim being true and a claim showing something are separate qualities, and in a report only the second one does any work.

The fourth and sixth rows should be read together. “Data was copied to removable media” rests on three bases and its expected number is 1.71; “a user opened the file” also rests on three bases and its expected number is 4.68. What makes the difference is not the number of bases but their class: in the first, the removable-media record is in the direct class and its base is 8 percent; in the second, all three bases are inference, and the weakest one has a base of 70 percent. Adding a basis does not increase distinctiveness; adding a rare basis does.

The independence assumption must also be stated here. The product accepts that the traces come into being independently of each other. In reality, the shortcut record and the recent-items list are most often born of the same action; if one is present, the probability that the other is also present is higher than the product gives. That is, the real number is larger than the calculated one, and the 4.68 read from the table is a lower bound. The independence assumption always makes a claim look stronger than it is; an analysis that does not state this is biased in its own favor.

The third table counts the cost of losing a structure. The application cache is a basis for two claims, and its loss downgrades the class of both claims; it is the single most expensive structure. The user profile stamp is also a basis for two claims, but its loss downgrades neither class — one is already noise, and the other stays in the same class without it. Two structures can be a basis for the same number of claims, and the weight they carry differs by exactly the difference in their bases.

The quietest row in the table is the share directory entry: it has been found, it has been classified, and it is a basis for no claim at all. The set of traces collected and the set of traces used are not the same set, and that difference is the source of the gap between a report’s length and its value.

A Structure’s Lifespan and the Cost of Delay

// forensics/lifespan.mjs — the time before a structure is overwritten, and the effect of delay on claims.
import { FLEET, STRUCTURE, CLAIM, expected, classify } from "./structure.mjs";

const y = (g, ...s) => console.log(s.map((v, i) =>
  (g[i] < 0 ? String(v).padEnd(-g[i]) : String(v).padStart(g[i]))).join("").trimEnd());

// structure -> [number of entries it holds, entries added per day under ordinary use]  0 = never overwritten
const CYCLE = {
  "deleted-file-name": [2400, 180], "recycle-bin-metadata": [500, 12],
  "application-cache": [1024, 96], "scheduled-task-definition": [1, 0],
  "configuration-store-entry": [1, 0], "removable-media-record": [64, 3],
  "installation-record": [1, 0], "shortcut-record": [149, 22],
  "recent-items-list": [20, 14], "file-timestamp": [1, 0.35],
  "share-directory-entry": [32, 9], "user-profile-stamp": [1, 1],
};
const lifespan = (k) => (CYCLE[k][1] ? CYCLE[k][0] / CYCLE[k][1] : Infinity);
const readable = (k, day) => day <= lifespan(k);

const A = [-28, 10, 10, 12, 10];
y(A, "structure", "entries", "daily", "life days", "base");
for (const k of Object.keys(STRUCTURE))
  y(A, k, CYCLE[k][0], CYCLE[k][1], CYCLE[k][1] ? lifespan(k).toFixed(1) : "continuous", STRUCTURE[k][1]);

const B = [-10, 14, 14, 8, 10, 14];
console.log("");
y(B, "delay", "readable", "distinctive", "weak", "noise", "unsupported");
for (const day of [1, 3, 7, 14, 30, 90]) {
  const remaining = Object.fromEntries(Object.keys(CLAIM).map((i) =>
    [i, CLAIM[i].filter((k) => readable(k, day))]));
  const count = (s) => Object.values(remaining).filter((d) => d.length && classify(expected(d)) === s).length;
  y(B, `${day} day${day === 1 ? "" : "s"}`, Object.keys(STRUCTURE).filter((k) => readable(k, day)).length,
    count("distinctive"), count("weak"), count("noise"),
    Object.values(remaining).filter((d) => !d.length).length);
}
const permanent = Object.keys(STRUCTURE).filter((k) => lifespan(k) === Infinity);
console.log(`\n${permanent.length} of the ${Object.keys(STRUCTURE).length} structures are never overwritten; ` +
  `fleet ${FLEET} machines, base values do not change`);

structure                      entries     daily   life days      base
deleted-file-name                 2400       180        13.3         4
recycle-bin-metadata               500        12        41.7        47
application-cache                 1024        96        10.7         9
scheduled-task-definition            1         0  continuous         6
configuration-store-entry            1         0  continuous        12
removable-media-record              64         3        21.3        22
installation-record                  1         0  continuous        63
shortcut-record                    149        22         6.8        31
recent-items-list                   20        14         1.4        58
file-timestamp                       1      0.35         2.9        96
share-directory-entry               32         9         3.6        74
user-profile-stamp                   1         1         1.0       187

delay           readable   distinctive    weak     noise   unsupported
1 day                 12             2       4         1             0
3 days                 9             2       2         2             1
7 days                 7             2       2         1             2
14 days                5             1       0         3             3
30 days                4             1       0         2             4
90 days                3             1       0         1             5

3 of the 12 structures are never overwritten; fleet 268 machines, base values do not change

Most structures are cyclical: they hold a fixed number of entries and drop the oldest one when a new one arrives. Lifespan is the ratio of these two numbers, and in the table it ranges from one day to continuous. Three structures are never overwritten at all; these are definitional records, and they stay until deleted.

When two columns of the first table are read together, an inverse relationship emerges. The structure with the shortest lifespan — the user profile stamp — is also the structure with the highest base: its lifespan is one day, its base is 187. The more often a structure is written to, the sooner it is overwritten and the more clean machines it is also found on. A trace that disappears quickly is the least distinguishing trace even when it has not yet disappeared.

The second table carries the course’s measurement axis into this lesson. As delay rises from one day to ninety days, the number of readable structures goes from 12 to 3, and the number of claims left without a basis goes from 0 to 5. The number of claims never changes; the only thing that changes is whether the structures carrying those claims are still there.

The fourteen-day row also deserves attention: the number of claims in the weak class is zero. The claims have not improved; because the structures carrying the bases in between have been erased by the cycle, the claims have been thrown to the two extremes — they have either stayed distinctive by resting on permanent structures, or they have fallen to noise, left with only a high-base structure. Delay does not just reduce evidence, it also distorts its distribution.

Summary

• A structure is in the direct class if it comes into being only because the claimed event occurred, and in the inference class if it can also come into being for other reasons; 7 of the 12 structures are direct, and their bases range from 4 to 187 in the 268-machine fleet, that is, from 1 percent to 70 percent.
• Of seven claims, 2 are distinctive, 4 are weak, and 1 is noise; the noise claim rests on a single inference structure and is also true on 187 clean machines.
• The number of bases does not determine distinctiveness: a claim with three inference bases (4.68) turns out weaker than a claim with one direct and two inference bases (1.71).
• The independence assumption makes the calculation a lower bound and always biases it in the claim’s favor; traces born of the same action are found together.
• The loss of a structure is measured as a number: losing the application cache downgrades the class of two claims, while losing the user profile stamp downgrades neither.
• Nine of the structures are cyclical and their lifespans range from 1 to 41.7 days; when delay rises from one day to ninety days, readable structures go from 12 to 3, and claims left without a basis go from 0 to 5.

Next Step

This topic’s five lessons produced five separate numbers: how many items the chain breaks at, how many items scope brings in, how many items order costs, how many events skew reverses, and how many clean machines a trace is also found on. None of these is a report sentence yet. A report consists of claims built on top of these numbers, and the numbers do not appear in the report unless the link between each claim and each item is written down one by one.

The final lesson measures this mapping and closes the course. Every claim intended for the report is linked to the evidence item it rests on; three columns are counted separately — how many claims are supported directly by an item, how many rest on an inference, and how many have no evidence at all. What the technical finding means in a legal context depends exactly on how these three numbers are written.