Cloud Differences in Incident Response

The previous lesson counted the patterns and put the closing controls in order. One question remained: what can be collected when one of these patterns actually occurs. The incident response lifecycle, evidence integrity, chain of custody, and write-protected copying were measured in the Security Operations and Incident Response course; the response steps are not repeated here. The only thing this lesson measures is the difference: how many items collectible on a host you manage yourself are not collectible on rented infrastructure, from which plane isolation is done, and what evidentiary value a snapshot has.

The measurement is on the regional measurement network, is fictional, and is a model written with node.

CD31. Twenty evidence items are marked in four columns: the host you manage yourself, the managed instance, the managed container service, the function execution environment. CD32. The mark has three values: the tenant collects it directly, it comes only by request to the provider, it cannot be collected. CD33. For every item, who can perform verification is written down: the tenant, the provider’s attestation, or no one. CD34. Five items are volatile: they disappear when the workload ends. The snapshot does not carry the volatile class or deleted-file remnants. CD35. Isolation can be done from four planes; each plane’s application time, the edge class it cuts, and the number of legitimate tasks it stops are written into the model (fictional). The moment the alert drops is minute 9, taken from the first lesson.

Collected on the Host, Not Collectible in the Cloud

// evidence.mjs — collectibility in the cloud of evidence items collectible on a host. IT IS A MODEL.
// The evidence items themselves and the chain of custody were measured in the Security Operations
// and Incident Response course; here only the difference between HOST and CLOUD is counted.
const split = (s) => s.trim().split("\n").map((l) => l.trim().split("|"));

// Item: name|host|managed instance|container service|function environment|who can verify
// 1 = tenant collects directly, 2 = only comes by request to the provider, 0 = cannot be collected
export const ITEM = split(`
  full-disk-image|1|2|0|0|provider
  memory-dump|1|0|0|0|-
  deleted-file-remnants|1|0|0|0|-
  hardware-layer-log|1|0|0|0|-
  hypervisor-log|1|0|0|0|-
  filesystem-timestamps|1|2|0|0|provider
  process-list|1|1|1|0|tenant
  open-network-connections|1|1|1|0|tenant
  loaded-kernel-modules|1|1|0|0|tenant
  shell-history|1|1|0|0|tenant
  scheduled-tasks|1|1|0|0|tenant
  local-user-accounts|1|1|0|0|tenant
  cache-and-temp-files|1|1|0|0|tenant
  system-logs|1|1|1|0|tenant
  application-logs|1|1|1|1|tenant
  network-flow-log|1|1|1|1|provider
  management-plane-log|0|1|1|1|provider
  call-metadata|0|1|1|1|provider
  snapshot-metadata|0|1|1|0|provider
  image-authentication|0|1|1|1|provider`)
  .map(([name, k, o, c, i, d]) => ({ name, host: +k, instance: +o, container: +c, func: +i, verify: d }));

const FORM = [["managed instance", "instance"], ["container service", "container"],
  ["function environment", "func"]];

const gen = (W) => (row) => row.map((v, j) => j ? String(v).padStart(W[j]) : String(v).padEnd(W[0]))
  .join("");

// Volatile items: disappear when the workload ends. The snapshot carries the disk class.
export const VOLATILE = ["memory-dump", "process-list", "open-network-connections",
  "loaded-kernel-modules", "cache-and-temp-files"];
export const SNAPSHOT_EXCLUDED = [...VOLATILE, "deleted-file-remnants"];

const onHost = ITEM.filter((k) => k.host === 1);

if (import.meta.url === `file://${process.argv[1]}`) {
  const tbl = gen([22, 8, 15, 17, 17, 16]);
  console.log(`${ITEM.length} evidence items; collectible on a host you manage yourself ` +
    `${onHost.length}, not present on the host at all ${ITEM.length - onHost.length}`);
  console.log("\n" + tbl(["deployment form", "direct", "from provider", "not collectible",
    "missing on host", "new from cloud"]));
  for (const [name, field] of FORM) {
    const direct = ITEM.filter((k) => k[field] === 1).length;
    const provider = ITEM.filter((k) => k[field] === 2).length;
    const missing = onHost.filter((k) => k[field] === 0).length;
    const fresh = ITEM.filter((k) => k.host === 0 && k[field] >= 1).length;
    console.log(tbl([name, direct, provider, ITEM.length - direct - provider, missing, fresh]));
  }

  const D = ["tenant", "provider", "-"];
  console.log("\n" + tbl(["verified by", "items", "on host", "on instance", "in container", "in function"]));
  for (const d of D) {
    const g = ITEM.filter((k) => k.verify === d);
    console.log(tbl([d === "-" ? "none" : d, g.length, g.filter((k) => k.host === 1).length,
      g.filter((k) => k.instance >= 1).length, g.filter((k) => k.container >= 1).length,
      g.filter((k) => k.func >= 1).length]));
  }

  const tenantVerify = ITEM.filter((k) => k.verify === "tenant");
  console.log(`\nof the ${onHost.length} items collectible on the host, ` +
    `${onHost.filter((k) => k.instance === 0).length} are not collectible on a managed instance, ` +
    `${onHost.filter((k) => k.container === 0).length} on a container service, ` +
    `${onHost.filter((k) => k.func === 0).length} in a function environment`);
  console.log(`in return, the cloud gives ${ITEM.filter((k) => k.host === 0).length} items: ` +
    `verification for all of them rests on the provider's attestation`);
  console.log(`of the ${tenantVerify.length} items the tenant can verify itself, ` +
    `${tenantVerify.filter((k) => k.func >= 1).length} remains in the function environment; for ` +
    `everything else, verification is the provider's attestation`);
}

20 evidence items; collectible on a host you manage yourself 16, not present on the host at all 4

deployment form         direct  from provider  not collectible  missing on host  new from cloud
managed instance            14              2                4                4               4
container service            9              0               11               11               4
function environment         5              0               15               14               3

verified by              items        on host      on instance     in container     in function
tenant                       9              9                9                4               1
provider                     7              3                7                5               4
none                         4              4                0                0               0

of the 16 items collectible on the host, 4 are not collectible on a managed instance, 11 on a container service, 14 in a function environment
in return, the cloud gives 4 items: verification for all of them rests on the provider's attestation
of the 9 items the tenant can verify itself, 1 remains in the function environment; for everything else, verification is the provider's attestation

Sixteen of the twenty items are collectible on a host you manage yourself. On a managed instance, four of that number drop away, and the ones that drop are not random: memory dump, deleted file remnants, hardware-layer log, hypervisor log. All four come from the layer beneath the host, and that layer does not belong to the tenant. On a container service, eleven items become uncollectible; in a function execution environment, fourteen do. Of the five items that remain in the function environment, four are the provider’s own logs.

The bill is not one-sided. The cloud gives four items that do not exist on a host at all: management-plane log, call metadata, snapshot metadata, and image authentication. These are real evidence items with no counterpart on a host you manage yourself. But verification for all four rests on the provider’s attestation: the tenant cannot reach the plane where these records are produced, and therefore cannot independently recompute their contents.

The table below turns this distinction into a number. There are nine items the tenant can verify itself, and all nine exist on the host; on a managed instance all nine remain, on a container service that drops to four, and in the function environment it drops to one. The decline in evidence and the decline in verifiability move in the same direction, and in the function environment both bottom out together.

Which Plane Isolation Is Done From

// response.mjs — from which plane isolation is done determines where the radius stops and how
// many volatile evidence items are lost. IT IS A MODEL. The response lifecycle was measured in
// the earlier course; here only the plane choice is counted.
import { ITEM, VOLATILE, SNAPSHOT_EXCLUDED } from "./evidence.mjs";

const split = (s) => s.trim().split("\n").map((l) => l.trim().split("|"));
const EDGE = split(`
  portal-service-account|portal-instance|network|3
  portal-service-account|reader-role|management|4
  portal-instance|portal-instance-identity|in-instance|2
  portal-instance-identity|subscriber-database|network|6
  portal-instance-identity|secret-store|management|4
  secret-store|collection-workload-identity|management|3
  collection-workload-identity|raw-meter-bucket|network|5
  collection-workload-identity|measurement-queue|network|4
  collection-workload-identity|monitoring-role|management|4
  monitoring-role|measurement-dashboard|management|3
  portal-instance-identity|billing-batch-role|management|3
  billing-batch-role|billing-archive-bucket|network|6
  billing-batch-role|key-management|management|4
  key-management|customer-key|management|2
  billing-batch-role|backup-role|management|3
  backup-role|backup-bucket|network|7
  backup-role|deployment-role|management|4
  deployment-role|template-store|network|5
  deployment-role|configuration-bucket|network|4
  deployment-role|workload-instance|in-instance|9
  reader-role|report-bucket|network|4
  reader-role|log-bucket|network|5
  reader-role|configuration-bucket|network|6`).map(([k, h, d, s]) => ({ src: k, dst: h, plane: d, dur: +s }));

const ALERT = 9;   // the first visible log moment measured in the first lesson (min)

// Isolation plane: name|application time (min)|edge planes it cuts|is volatile evidence preserved|
// legitimate tasks stopped (fictional unit)
const PLANE = split(`
  no isolation|0|-|1|0
  network plane|6|network|1|4
  identity plane|2|management in-instance|1|3
  instance plane|3|network management in-instance|0|2
  account plane|45|network management in-instance|1|8`)
  .map(([name, t, cuts, vol, cost]) => ({ name, time: +t, cuts: cuts.split(" "), volatile: +vol, cost: +cost }));

// At the moment of the cut, edges in the planes cut can no longer be walked.
function walk(p) {
  const cut = p.time === 0 ? Infinity : ALERT + p.time;
  const arrival = { "portal-service-account": 0 };
  for (;;) {
    let best = null;
    for (const e of EDGE) {
      if (!(e.src in arrival) || e.dst in arrival) continue;
      const t = arrival[e.src] + e.dur;
      if (t > cut && p.cuts.includes(e.plane)) continue;
      if (best === null || t < best.t) best = { e, t };
    }
    if (best === null) return { items: Object.keys(arrival).length, cut };
    arrival[best.e.dst] = best.t;
  }
}

const gen = (W) => (row) => row.map((v, j) => j ? String(v).padStart(W[j]) : String(v).padEnd(W[0]))
  .join("");
const tbl = gen([16, 9, 9, 8, 15, 15]);

console.log(tbl(["plane", "applies", "cuts at", "radius", "volatile lost", "tasks stopped"]));
for (const p of PLANE) {
  const r = walk(p);
  console.log(tbl([p.name, p.time ? `${p.time} min` : "-", r.cut === Infinity ? "-" : `min ${r.cut}`,
    `${r.items}/28`, p.volatile ? `0/${VOLATILE.length}` : `${VOLATILE.length}/${VOLATILE.length}`, p.cost]));
}

const identity = walk(PLANE[2]), instance = walk(PLANE[3]), account = walk(PLANE[4]);
console.log(`\nthe fastest plane is identity: it cuts at minute ${identity.cut} and stops the ` +
  `radius at ${identity.items}/28; all ${VOLATILE.length} volatile evidence items stay in place`);
console.log(`terminating the instance brings the radius to ${instance.items}/28 but erases all ` +
  `${VOLATILE.length} volatile items; account freeze arrives at minute ${account.cut} and the ` +
  `radius becomes ${account.items}/28`);

// The snapshot's evidentiary value: which items it carries, how many the tenant can verify.
const carried = ITEM.filter((k) => k.host === 1 && !SNAPSHOT_EXCLUDED.includes(k.name));
console.log(`\nthe snapshot carries ${carried.length} of the ${ITEM.filter((k) => k.host === 1).length} ` +
  `items collectible on the host, and does not carry ${SNAPSHOT_EXCLUDED.length}; the content of ` +
  `${carried.filter((k) => k.verify === "tenant").length} of what it carries can be read by the tenant`);
console.log(`the source's hash cannot be recomputed: there is no offline access to the raw device, ` +
  `so for all ${carried.length} items where the snapshot matches the source, verification rests ` +
  `on the provider's attestation`);

plane             applies  cuts at  radius  volatile lost  tasks stopped
no isolation            -        -   23/28            0/5              0
network plane       6 min   min 15   19/28            0/5              4
identity plane      2 min   min 11   13/28            0/5              3
instance plane      3 min   min 12   13/28            5/5              2
account plane      45 min   min 54   23/28            0/5              8

the fastest plane is identity: it cuts at minute 11 and stops the radius at 13/28; all 5 volatile evidence items stay in place
terminating the instance brings the radius to 13/28 but erases all 5 volatile items; account freeze arrives at minute 54 and the radius becomes 23/28

the snapshot carries 10 of the 16 items collectible on the host, and does not carry 6; the content of 5 of what it carries can be read by the tenant
the source's hash cannot be recomputed: there is no offline access to the raw device, so for all 10 items where the snapshot matches the source, verification rests on the provider's attestation

The table runs all four planes on the same timeline. The alert drops at minute 9 and the walk ends at minute 24; the fifteen minutes in between are the entire window in which the decision can be made. The network plane is applied in six minutes, cuts at minute 15, and stops the radius at 19 — because management-plane calls do not travel over the network, the identity is still valid. The identity plane is applied in two minutes, cuts at minute 11, and stops the radius at 13: the best outcome, at the second-lowest cost.

The instance plane gives the same radius, 13. The difference is in the evidence column: terminating the instance erases all five at once of the five volatile items. Same radius, five fewer items of evidence. This is the trade-off at the center of the lesson, and it is what simplifies the decision — if two options stop the same thing, the one that preserves the evidence is chosen.

The account plane is the last row, and it arrives at minute 54 because of its forty-five-minute approval time. Since the walk ends at minute 24, it cuts nothing at all: the radius stays at 23, and all eight legitimate tasks stop. The broadest authority, the highest cost, zero effect.

The Snapshot’s Evidentiary Value

The snapshot carries ten of the sixteen items collectible on the host and does not carry six: the five volatile items plus deleted file remnants. Of the ten it carries, the content of five can be read by the tenant.

The real limit is on the verification side. A copy’s evidentiary value rests on the source’s hash being recomputable. On rented infrastructure, the tenant has no offline access to the raw device; therefore, for all ten items, the fact that the image matches the source is the provider’s attestation. A snapshot does not produce evidence, it carries evidence; the trust it produces comes from the provider’s signature. This does not make the image worthless — it is the one mechanism that preserves most of the items collectible on the host, and if taken before the instance is terminated, it also leaves the volatile class in place. All it has to do is state its limit: at which plane the record is held and whom the verification rests on are written in the investigation report, because the other side will ask.

Summary

• Sixteen of the twenty evidence items are collectible on a host you manage yourself; four are not collectible on a managed instance, eleven on a container service, fourteen in a function environment.
• The four items that drop away on a managed instance come from the layer beneath the host: memory dump, deleted file remnants, hardware-layer log, hypervisor log.
• The cloud gives four items that do not exist on a host, but verification for all four rests on the provider’s attestation; the nine items the tenant can verify itself drop to one in the function environment.
• Isolation from the identity plane cuts at minute 11, stops the radius at 13, and preserves all five volatile items; terminating the instance gives the same radius and erases all five at once.
• The account plane arrives at minute 54 with a forty-five-minute approval; since the walk ends at minute 24, the radius stays at 23 and all eight legitimate tasks stop.
• The snapshot carries ten of the sixteen items and does not carry six; because the source’s hash cannot be recomputed, verification for everything it carries is the provider’s attestation.

Course Wrap-Up

The table makes the course’s rule readable in a single line: the measure of security in the cloud is not how many controls are open, but how many items are reachable from a single misconfigured item. If that number does not drop, the control added has only provided visibility. Three examples of this came up over the course: enabling logging raised the visible step count from eleven to nineteen and left the radius at 23; the admission gate closed two more items and left the radius at eight; guest-system isolation raised the item count needed to cross the boundary from one to three and left the radius at eighteen. All three are useful, all three have a cost, and all three are zero on this measure.

Across nine courses, the path each defense closed and that closure’s cost to legitimate work were counted. One question was never asked: which of these should be done first. The last two lessons of this course touched that question for the first time — one of thirteen controls removes seventeen items for a cost of two, ten remove zero for a cost of fifteen — but the metric that produced the ordering came from a single graph. In a real organization, budget is not the only scarce thing: attention, the approval window, and change capacity are scarce too. M24/K10 Governance, Risk and Compliance takes up these three questions: which defense to do first, how to allocate a scarce budget, and how to prove to someone that it was done.