Network Flow Data

The previous lesson’s k4 item appeared in no identity or management record; its third step existed only as an outbound connection, in network flow data. That source is structurally different from the other two. A flow record does not write the connection’s content; it writes who talked to whom, on which port, for how long, and how many bytes.

This lesson turns that constraint into a measurement. The question is not “is flow data useful” but which question can be answered without content — and, in return, how many legitimate connections get flagged. The record set below is a model; the external addresses are drawn from blocks reserved for documentation and do not correspond to a real endpoint.

• DI18. A flow record has seven fields: time, host, destination, port, duration, bytes sent, bytes received. There is no content field, and this is not a configuration gap — it is the source’s definition.
• DI19. The fictional network has four legitimate traffic patterns: personnel browsing, meter collectors’ fifteen-minute transfers, nightly backup, and long admin sessions.
• DI20. Four fictional items are given only as records: a regularly spaced connection sequence, a high-exit-ratio transfer, a destination absent from the baseline, and a single long session.
• DI21. A metric operates on a (host, destination, port) triple and looks only at metadata. The label is the model’s ground truth and is not written to the record.
• DI22. The baseline is the set of (destination, port) pairs seen on previous days, and it is given as model data.

Four Metadata-Only Metrics

// flowdata.mjs — fictional network flow records and four metadata-only metrics. THIS IS A MODEL.
// External addresses come from blocks reserved for documentation; there is no real endpoint or organization.
export const SEED = 20260404;
const generator = (t) => { let s = t >>> 0;
  return () => (s = (s * 1664525 + 1013904223) >>> 0) / 4294967296; };
const r = generator(SEED);
const randInt = (n) => Math.floor(r() * n);

const ENDPOINT = Array.from({ length: 40 }, (_, i) => `uc-${101 + i}`);
const COLLECTOR = Array.from({ length: 6 }, (_, i) => `tp-${11 + i}`);
const INTERNAL = Array.from({ length: 8 }, (_, i) => `10.90.1.${20 + i}`);
const EXTERNAL = Array.from({ length: 30 }, (_, i) => `198.51.100.${10 + i}`);

// Flow record: the connection has no CONTENT. Only who, with whom, how much, how long.
const raw = [];
const add = (t, host, dest, port, duration, sent, received, label) =>
  raw.push({ t: Math.round(t), host, dest, port, duration, sent, received, label });

// Personnel browsing: bytes received are much larger than bytes sent, destination varies.
for (const u of ENDPOINT)
  for (let k = 0, n = 30 + randInt(60); k < n; k++)
    add(randInt(1440), u, EXTERNAL[randInt(30)], 443, 1 + randInt(90), 800 + randInt(9000),
      20000 + randInt(900000), "browsing");
// Meter collectors: a small and REGULAR transfer to an internal destination every fifteen minutes.
for (const [j, tp] of COLLECTOR.entries())
  for (let t = j * 2; t < 1440; t += 15)
    add(t, tp, INTERNAL[j % 8], 5671, 2 + randInt(3), 40000 + randInt(4000), 900 + randInt(400), "measurement-transfer");
// Nightly backup: large sent, small received, once a day.
for (const u of ENDPOINT.slice(0, 22))
  add(120 + randInt(120), u, INTERNAL[6], 8443, 40 + randInt(40), 3e8 + randInt(2e8), 5000 + randInt(4000), "backup");
// Long admin sessions.
for (const u of ENDPOINT.slice(0, 5)) add(540 + randInt(120), u, INTERNAL[7], 22, 150 + randInt(180),
  90000 + randInt(50000), 300000 + randInt(200000), "admin-session");

// Four fictional items. Only the RECORD is given; the method is not described.
const ITEM = ["d1-regular-interval", "d2-exit-ratio", "d3-new-destination", "d4-long-session"];
for (let t = 30, k = 0; k < 96; k++, t += 15)                       // d1
  add(t + (r() < 0.5 ? 0 : 1), ENDPOINT[7], "203.0.113.24", 443, 1 + randInt(2), 1200 + randInt(400),
    600 + randInt(300), ITEM[0]);
for (let k = 0, t = 800; k < 12; k++)                               // d2
  add(t += 4 + randInt(9), ENDPOINT[19], "203.0.113.31", 443, 20 + randInt(40), 4e7 + randInt(2e7),
    3000 + randInt(2000), ITEM[1]);
for (let k = 0; k < 3; k++)                                         // d3
  add(1010 + k * 7, ENDPOINT[31], "203.0.113.9", 9443, 3 + randInt(6), 9000 + randInt(4000),
    12000 + randInt(9000), ITEM[2]);
add(300, ENDPOINT[24], INTERNAL[3], 8443, 415, 2200000, 6100000, ITEM[3]);      // d4
// New work item: three legitimate external destinations seen for the first time during the day.
for (const [j, u] of ENDPOINT.slice(30, 36).entries())
  for (let k = 0, n = 2 + randInt(5); k < n; k++)
    add(600 + randInt(500), u, `192.0.2.${40 + (j % 3)}`, 443, 2 + randInt(30), 3000 + randInt(9000),
      40000 + randInt(300000), "new-work-item");
raw.sort((a, b) => a.t - b.t);
export const FLOW = raw.map(({ label, ...a }, i) => ({ ...a, i }));
export const LABEL = raw.map((a) => a.label);   // ground truth of the model; metrics Do not see it

// Baseline: (destination, port) pairs seen on previous days (model data).
export const BASELINE = new Set([...INTERNAL.flatMap((h) => [`${h}|5671`, `${h}|8443`, `${h}|22`]),
  ...EXTERNAL.map((h) => `${h}|443`), "203.0.113.24|443", "203.0.113.31|443"]);

// Four metrics: they look only at metadata, not at content.
const pairs = () => {
  const m = new Map();
  for (const a of FLOW) {
    const k = `${a.host}|${a.dest}|${a.port}`;
    (m.get(k) || m.set(k, []).get(k)).push(a);
  }
  return m;
};
export const METRIC = {
  "regular interval": () => [...pairs()].filter(([, l]) => {
    if (l.length < 12) return false;
    const d = l.slice(1).map((a, i) => a.t - l[i].t);
    const o = d.reduce((x, y) => x + y, 0) / d.length;
    return o > 0 && Math.sqrt(d.reduce((x, y) => x + (y - o) ** 2, 0) / d.length) / o < 0.15;
  }),
  "exit ratio": () => [...pairs()].filter(([, l]) => {
    const s = l.reduce((x, a) => x + a.sent, 0), rc = l.reduce((x, b) => x + b.received, 0);
    return s > 5e6 && s / rc > 20;
  }),
  "new destination": () => [...pairs()].filter(([k]) => {
    const [, h, p] = k.split("|");
    return !BASELINE.has(`${h}|${p}`);
  }),
  "long session": () => [...pairs()].filter(([, l]) => l.some((a) => a.duration > 120)),
};

if (import.meta.url === `file://${process.argv[1]}`) {
  const fmt = (B) => (h) => h.map((v, j) => (j ? String(v).padStart(B[j])
    : String(v).padEnd(B[0]))).join("");
  const line = fmt([16, 12, 10, 13, 34]);
  console.log(`model (seed ${SEED}): ${FLOW.length} flow records, ` +
    `${new Set(FLOW.map((a) => a.host)).size} hosts; no content field in the record`);
  console.log(`example: ${JSON.stringify(FLOW[LABEL.indexOf(ITEM[0])])}`);
  console.log();
  console.log(line(["metric", "flagged", "item", "false pos.", "source of false positives"]));
  for (const [name, f] of Object.entries(METRIC)) {
    const l = f();
    const lab = ([, x]) => LABEL[x[0].i];
    const correct = l.filter((c) => ITEM.includes(lab(c)));
    const fp = l.filter((c) => !ITEM.includes(lab(c)));
    console.log(line([name, l.length, correct.map((c) => lab(c).slice(0, 2)).join(",") || "-",
      fp.length, [...new Set(fp.map(lab))].join(", ") || "-"]));
  }
  const all = new Set(Object.values(METRIC).flatMap((f) => f().map(([, x]) => LABEL[x[0].i])
    .filter((e) => ITEM.includes(e))));
  const total = Object.values(METRIC).reduce((a, f) => a + f().length, 0);
  console.log(`\n${all.size} of the four fictional items are flagged by at least one metric; ` +
    `the four metrics together flag ${total} pairs`);
}

model (seed 20260404): 2979 flow records, 46 hosts; no content field in the record
example: {"t":31,"host":"uc-108","dest":"203.0.113.24","port":443,"duration":1,"sent":1402,"received":676,"i":70}

metric               flagged      item   false pos.         source of false positives
regular interval           7        d1            6              measurement-transfer
exit ratio                23        d2           22                            backup
new destination            7        d3            6                     new-work-item
long session               6        d4            5                     admin-session

4 of the four fictional items are flagged by at least one metric; the four metrics together flag 43 pairs

The example record shows the lesson’s limit: there are seven fields and none of them says what was carried. Against that, the four metrics flag all four of the fictional items. A pattern can be seen without seeing the content; what is measured is not the content itself but the connection’s shape.

The shape’s discriminative power, though, is low, and the table’s last two columns record that. The regular-interval metric flags seven pairs, six of which are the meter collectors’ fifteen-minute transfer. That same regularity — a fixed interval, small and equal-sized packets — is this network’s normal work; the fictional item has exactly the shape of normal work. The exit-ratio metric is even worse: twenty-two of twenty-three pairs are nightly backup.

The new-destination metric flags six legitimate items, because all three new work connections are also seen for the first time during the day. One detail matters here: d1’s destination is in the baseline, so the new-destination metric never sees it at all. No single metric catches all four items at once; coverage comes from the union of the four metrics, and that union flags 43 pairs.

The Question Answered Without Content

// question.mjs — which questions a flow record can and cannot answer, then the cost of cutting it off.
import { FLOW, LABEL, BASELINE, METRIC } from "./flowdata.mjs";

const pair = FLOW.filter((a) => LABEL[a.i] === "d1-regular-interval");
const bytes = (n) => (n > 1e9 ? `${(n / 1e9).toFixed(1)} GB` : `${(n / 1e6).toFixed(1)} MB`);
const median = (v) => v.slice().sort((a, b) => a - b)[Math.floor(v.length / 2)];
// Each question is a function: it returns the answer if it can produce one from the flow record, null if it cannot.
const QUESTION = [
  ["which two endpoints talked", (l) => `${l[0].host} -> ${l[0].dest}:${l[0].port}`],
  ["how many times did it connect", (l) => `${l.length} times`],
  ["at what intervals", (l) => `${median(l.slice(1).map((x, i) => x.t - l[i].t))} min`],
  ["how much data went out", (l) => bytes(l.reduce((a, x) => a + x.sent, 0))],
  ["how long did it last", (l) => `at most ${Math.max(...l.map((x) => x.duration))} min`],
  ["was the destination seen before",
    (l) => (BASELINE.has(`${l[0].dest}|${l[0].port}`) ? "yes" : "no")],
  ["what was sent", () => null],
  ["which account did it", () => null],
  ["was the request accepted", () => null],
  ["which process started it", () => null],
];
const fmt = (B) => (h) => h.map((v, j) => (j ? String(v).padStart(B[j])
  : String(v).padEnd(B[0]))).join("");
const line = fmt([34, 16, 28]);
console.log(`d1 item: ${pair.length} flow records. The same record set is asked ten questions.`);
console.log();
console.log(line(["question", "answerable", "answer"]));
for (const [q, f] of QUESTION) {
  const y = f(pair);
  console.log(line([q, y === null ? "no" : "yes", y === null ? "-" : y]));
}
const answerable = QUESTION.filter(([, f]) => f(pair) !== null).length;
console.log(`\n${answerable} of the ten questions are answered from metadata alone, ` +
  `${QUESTION.length - answerable} need another source`);

// Early decision: legitimate work cut if a connection is severed based only on the flow metric.
const SOURCE = { "measurement-transfer": ["meter collection stops", 0], "backup": ["nightly backup", 0],
  "new-work-item": ["new work connection", 0], "admin-session": ["admin access", 0] };
const flagged = new Map();
for (const [, f] of Object.entries(METRIC))
  for (const [k, l] of f()) if (!flagged.has(k)) flagged.set(k, l);
let legitPairs = 0, legitRecords = 0, legitBytes = 0;
for (const [, l] of flagged) {
  const e = LABEL[l[0].i];
  if (!SOURCE[e]) continue;
  legitPairs++; legitRecords += l.length;
  legitBytes += l.reduce((a, x) => a + x.sent + x.received, 0);
  SOURCE[e][1]++;
}
const line2 = fmt([22, 10, 30]);
console.log(`\n${flagged.size} distinct pairs are flagged, ${legitPairs} of them are legitimate; ` +
  `cutting them stops ${legitRecords} flows and ${bytes(legitBytes)}`);
console.log(line2(["work type cut", "pairs", "work stopped"]));
for (const [e, [ne, n]] of Object.entries(SOURCE)) console.log(line2([e, n, ne]));

d1 item: 96 flow records. The same record set is asked ten questions.

question                                answerable                      answer
which two endpoints talked                     yes  uc-108 -> 203.0.113.24:443
how many times did it connect                  yes                    96 times
at what intervals                              yes                      15 min
how much data went out                         yes                      0.1 MB
how long did it last                           yes               at most 2 min
was the destination seen before                yes                         yes
what was sent                                   no                           -
which account did it                            no                           -
was the request accepted                        no                           -
which process started it                        no                           -

6 of the ten questions are answered from metadata alone, 4 need another source

43 distinct pairs are flagged, 39 of them are legitimate; cutting them stops 630 flows and 8.8 GB
work type cut              pairs                  work stopped
measurement-transfer           6        meter collection stops
backup                        22                nightly backup
new-work-item                  6           new work connection
admin-session                  5                  admin access

The question table turns network flow data’s limit into a number: six of ten questions get answered, four stay unanswered. The answered ones alone are not enough to make a decision. uc-108 connected to the same destination 96 times every fifteen minutes and sent a total of 0.1 megabytes; that description fits both a recurring health check and something else. The fields that would tell them apart — which account, which process, whether the request was accepted — are not in the flow record.

Each of the four unanswered questions names a specific source. The question of which account did it requires the address mapping from the second lesson, and that mapping could answer 190 of 404 records in a five-minute window. The question of which process started it wants the endpoint record; whether the request was accepted wants the control-plane record. Network flow data is not a decision source, it is a routing source: it says which host and which time window to look at.

The last table writes the cost of that early decision. Of the 43 flagged pairs, 39 are legitimate, and cutting all of them stops 630 flows and 8.8 gigabytes of work. That is not a flat number: six of the cut pairs are meter-collection transfers, and when that transfer stops, measurement data never reaches the center at all. Twenty-two pairs are nightly backup; going a day without a backup is a loss that directly affects the recovery decision later in the course.

Summary

• A flow record has seven fields and no content field; despite that, four metrics that look only at shape flag all four fictional items.
• Discriminative power is low: regular interval flags meter transfer in 6 of 7 pairs, exit ratio flags nightly backup in 22 of 23 pairs, new destination flags a new work connection in 6 of 7 pairs.
• No single metric catches all four items at once; because d1’s destination is in the baseline, the new-destination metric never sees it, and coverage comes from the union of the four metrics.
• 6 of ten questions are answered from metadata alone; the four questions of what was sent, which account did it, whether the request was accepted, and which process started it need another source.
• Of the 43 flagged pairs, 39 are legitimate; cutting all of them stops 630 flows and 8.8 gigabytes of work, six pairs of which are meter-collection transfers.

Next Step

This lesson’s four metrics ran with fixed thresholds, and the result was that 39 of 43 flags turned out legitimate. Tightening the thresholds is the first fix that comes to mind: making the coefficient of variation 0.05 instead of 0.15, requiring an exit ratio of two hundred instead of twenty. The next lesson runs that as a sweep and plots two curves on the same axis — as the threshold tightens, false positives drop, but missed detections rise. What is measured is not which threshold is right but the point where the two curves do not cross: in the band where the fictional items’ scores overlap with legitimate work’s scores, no threshold can reduce both at once, and how many items that band contains can be counted.