--- title: 'Security Tooling and Penetration Testing' source: 'https://academia.sh/en/courses/penetration-testing' language: en updated: '2026-09-26T03:02:55+00:00' license: 'CC BY-SA 4.0' lessons: 18 outcomes: - 'Turns the scope and rules of engagement of an authorized testing process into programmatic checks' - 'Converts the candidates produced by reconnaissance, scanning, and analysis tools into evidence' - 'Counts the trace each step leaves on the defense side and reports a finding in a reproducible form' --- # Security Tooling and Penetration Testing The gap between a candidate finding and an evidenced finding in an authorized assessment: extracting attack surface candidates from publicly available information, the value of domain name and registration lookup as proof of ownership, the dangling item that subdomain and asset discovery uncovers, the trace that scan types leave on the defense side, service and version fingerprinting, the evidential value of packet capture and protocol analysis, request manipulation with intercepting proxies, sandbox analysis, the majority verdict of multi-engine reputation services, the validation-only limit of exploitation frameworks, legal practice labs, rules of engagement and programmatic scope checking, black, gray, and white box assessments, eliminating false positives, expressing a finding in terms of business impact, reproduction steps, and responsible disclosure. ## Reconnaissance The externally visible inventory of an authorized assessment: how many of the attack-surface candidates surfacing from public sources actually belong to the organization, which record in domain name and registration lookup proves ownership and which remains only a lead, the dangling item found by subdomain and asset discovery, the balance between the candidates port scan types find and the record they leave on the defense side, and how many candidates service fingerprinting can narrow down to a single version. ### Open-Source Intelligence Source: https://academia.sh/en/courses/penetration-testing/open-source-intelligence From a publicly available item to an attack surface candidate: 29 items yielding 26 mentions and 20 unique candidates, 9 of which are evidence and 6 false positives (30%), the most accurate source producing 83% evidence and the least productive 0%, 12 of 45 validation steps returning zero evidence, the evidence count flattening after four sources while accuracy drops from 69% to 45%, and only 9 of a 13-item inventory being visible from outside. The previous course counted, on paper, which link could be broken, and closed with a question: only someone authorized, actually trying, can show whether a defense truly holds. This course is that attempt itself. Its first step is knowing **what to look at** — and that list is not handed over by the organization itself; it is drawn out of items that are already sitting out in the open. **Open-source intelligence** is the extraction, from publicly available items, of assets that may belong to the target. The critical word here is **may**. This course's unit of measure is not the number of findings, it is the number of **evidenced findings**, and the distinction starts at the very first step: a name appearing in a list does not show that the name belongs to the organization. #### Candidate, Evidence, and False Positive This assessment is **authorized**: the scope owner has granted written authorization and defined which assets may be tested. Performing the same steps outside written authorization and a defined scope is both illegal and unprofessional. **RE1:** each source type carries four numbers — how many items it takes, how many records it leaves on the target side, how many alerts it triggers, and how many validation steps it requires per candidate. **RE2:** a candidate appearing in a source is called a *mention*; the same candidate can appear in multiple sources, so the mention count is larger than the candidate count. **RE3:** the inventory the scope owner has approved is the referee of this measurement; a candidate in the inventory counts as **evidence**, one not in the inventory but seen to serve the organization counts as a **lead**, and one that does not overlap with the inventory at all counts as a **false positive**. ```js // recon/source.mjs — a fictional measurement network's set of publicly available items (a // model): no real domain name, no real public address, no real organization. No randomness. // RE1: a source type carries these four numbers — items taken, records it leaves on the // target, alerts it triggers, and validation steps required per candidate. A source sitting // on a third party produces no record on the target; an item taken from the organization's // own endpoint leaves an access record on every fetch. export const SOURCE = ` certificate-record 4 0 0 1 document-metadata 3 0 0 2 job-posting 2 0 0 2 public-code-repo 3 0 0 2 archived-copy 5 0 0 3 corporate-website 12 12 0 1` .trim().split("\n").map((r) => { const [type, items, record, alert, step] = r.trim().split(/\s+/); return { type, items: +items, record: +record, alert: +alert, step: +step }; }); // RE2: each row is one candidate appearing in one source; the same candidate can appear in // multiple sources. export const MENTION = ` certificate-record subscriber-portal certificate-record field-app certificate-record test-portal certificate-record shared-interface certificate-record meter-update-endpoint certificate-record old-report-endpoint document-metadata report-endpoint document-metadata shared-interface document-metadata branch-vpn-endpoint document-metadata old-report-endpoint job-posting work-order-endpoint job-posting branch-vpn-endpoint job-posting training-environment public-code-repo collector-gateway public-code-repo cloud-storage-bucket public-code-repo test-portal public-code-repo spare-domain archived-copy old-subscriber-portal archived-copy old-field-app archived-copy support-portal archived-copy dealer-portal corporate-website subscriber-portal corporate-website report-endpoint corporate-website marketing-page corporate-website identity-provider-endpoint corporate-website payment-provider-endpoint` .trim().split("\n").map((r) => { const [source, candidate] = r.trim().split(/\s+/); return { source, candidate }; }); // RE3: the fictional inventory is, in this assessment, the real asset list the scope owner // approved, and it is the referee of the measurement. A candidate not in the inventory is a // false positive. export const INVENTORY = [ "subscriber-portal", "field-app", "branch-vpn-endpoint", "collector-gateway", "report-endpoint", "test-portal", "shared-interface", "work-order-endpoint", "meter-update-endpoint", "internal-network-management-endpoint", "backup-endpoint", "measurement-store", "identity-store", ]; // Items whose ownership cannot be proven from outside, sitting on a third party but serving // the organization. export const LEAD = ["old-subscriber-portal", "marketing-page", "cloud-storage-bucket", "identity-provider-endpoint", "training-environment"]; export const CANDIDATE = [...new Set(MENTION.map((a) => a.candidate))]; export const classify = (a) => INVENTORY.includes(a) ? "evidence" : LEAD.includes(a) ? "lead" : "false-positive"; ``` ```js // recon/candidate.mjs — from a publicly available item to an attack surface candidate: how // many candidates come out, how many are evidence. import { SOURCE, MENTION, CANDIDATE, INVENTORY, classify } from "./source.mjs"; const write = (g, ...s) => console.log(s.map((v, i) => (g[i] < 0 ? String(v).padEnd(-g[i]) : String(v).padStart(g[i]))).join("")); const count = (l, s) => l.filter((a) => classify(a) === s).length; const ITEMS = SOURCE.reduce((t, k) => t + k.items, 0); console.log(`${SOURCE.length} source types, ${ITEMS} publicly available items, ${MENTION.length} mentions, ` + `${CANDIDATE.length} unique candidates; inventory ${INVENTORY.length} items`); const A = [-19, 7, 6, 7, 7, 8, 9]; console.log("\n1. candidates and accuracy per source type"); write(A, "source type", "items", "ment", "evid", "leads", "falseP", "evid %"); for (const k of SOURCE) { const l = MENTION.filter((a) => a.source === k.type).map((a) => a.candidate); write(A, k.type, k.items, l.length, count(l, "evidence"), count(l, "lead"), count(l, "false-positive"), ((count(l, "evidence") / l.length) * 100).toFixed(0)); } write(A, "TOTAL (unique)", ITEMS, CANDIDATE.length, count(CANDIDATE, "evidence"), count(CANDIDATE, "lead"), count(CANDIDATE, "false-positive"), ((count(CANDIDATE, "evidence") / CANDIDATE.length) * 100).toFixed(0)); const B = [-19, 9, 7, 8, 12, 12]; console.log("\n2. the cost of validation and the trace left on the defense side"); write(B, "source type", "records", "alerts", "steps", "steps/evid", "rec/evid"); let stepTotal = 0; for (const k of SOURCE) { const l = MENTION.filter((a) => a.source === k.type).map((a) => a.candidate); const kn = count(l, "evidence"), st = l.length * k.step; stepTotal += st; write(B, k.type, k.record, k.alert, st, kn ? (st / kn).toFixed(2) : "-", kn ? (k.record / kn).toFixed(2) : "-"); } const TRACE = SOURCE.reduce((t, k) => t + k.record, 0); write(B, "TOTAL", TRACE, SOURCE.reduce((t, k) => t + k.alert, 0), stepTotal, (stepTotal / count(CANDIDATE, "evidence")).toFixed(2), (TRACE / count(CANDIDATE, "evidence")).toFixed(2)); const C = [-19, 10, 12, 13]; console.log("\n3. if one source type is dropped"); write(C, "dropped source", "evid left", "missed", "missed %"); const EVID = CANDIDATE.filter((a) => classify(a) === "evidence"); for (const k of SOURCE) { const remaining = [...new Set(MENTION.filter((a) => a.source !== k.type).map((a) => a.candidate))] .filter((a) => classify(a) === "evidence"); write(C, k.type, remaining.length, EVID.length - remaining.length, (((EVID.length - remaining.length) / EVID.length) * 100).toFixed(1)); } const D = [-19, 7, 7, 9, 9, 9]; console.log("\n4. as sources are added (from most to least accurate)"); write(D, "added source", "cand", "evid", "falseP", "evid %", "trace"); const sorted = [...SOURCE].sort((x, y) => { const o = (k) => { const l = MENTION.filter((a) => a.source === k.type).map((a) => a.candidate); return count(l, "evidence") / l.length; }; return o(y) - o(x); }); const accumulated = []; let traceTotal = 0; for (const k of sorted) { accumulated.push(k.type); traceTotal += k.record; const l = [...new Set(MENTION.filter((a) => accumulated.includes(a.source)).map((a) => a.candidate))]; write(D, k.type, l.length, count(l, "evidence"), count(l, "false-positive"), ((count(l, "evidence") / l.length) * 100).toFixed(0), traceTotal); } const visible = INVENTORY.filter((e) => MENTION.some((a) => a.candidate === e)).length; console.log(`\nexternally invisible inventory items: ` + INVENTORY.filter((e) => !MENTION.some((a) => a.candidate === e)).join(", ")); console.log(`${visible}/${INVENTORY.length} inventory items are visible ` + `(${((visible / INVENTORY.length) * 100).toFixed(1)}%); ` + `${count(CANDIDATE, "false-positive")}/${CANDIDATE.length} candidates are false positives ` + `(${((count(CANDIDATE, "false-positive") / CANDIDATE.length) * 100).toFixed(1)}%)`); ``` ``` 6 source types, 29 publicly available items, 26 mentions, 20 unique candidates; inventory 13 items 1. candidates and accuracy per source type source type items ment evid leads falseP evid % certificate-record 4 6 5 0 1 83 document-metadata 3 4 3 0 1 75 job-posting 2 3 2 1 0 67 public-code-repo 3 4 2 1 1 50 archived-copy 5 4 0 1 3 0 corporate-website 12 5 2 2 1 40 TOTAL (unique) 29 20 9 5 6 45 2. the cost of validation and the trace left on the defense side source type records alerts steps steps/evid rec/evid certificate-record 0 0 6 1.20 0.00 document-metadata 0 0 8 2.67 0.00 job-posting 0 0 6 3.00 0.00 public-code-repo 0 0 8 4.00 0.00 archived-copy 0 0 12 - - corporate-website 12 0 5 2.50 6.00 TOTAL 12 0 45 5.00 1.33 3. if one source type is dropped dropped source evid left missed missed % certificate-record 7 2 22.2 document-metadata 9 0 0.0 job-posting 8 1 11.1 public-code-repo 8 1 11.1 archived-copy 9 0 0.0 corporate-website 9 0 0.0 4. as sources are added (from most to least accurate) added source cand evid falseP evid % trace certificate-record 6 5 1 83 0 document-metadata 8 7 1 88 0 job-posting 10 8 1 80 0 public-code-repo 13 9 2 69 0 corporate-website 16 9 3 56 12 archived-copy 20 9 6 45 12 externally invisible inventory items: internal-network-management-endpoint, backup-endpoint, measurement-store, identity-store 9/13 inventory items are visible (69.2%); 6/20 candidates are false positives (30.0%) ``` The numbers belong to the **measurement** class; their inputs are RE1–RE3. #### Twenty Candidates, Nine Evidence Twenty-nine publicly available items produced twenty-six mentions; once mentions were deduplicated, **20 candidates** remained. Of these twenty candidates, **9 are evidence**, 5 are leads, and **6 are false positives**. 30% of the raw list does not belong to the organization, and if it went into a report without validation, it would produce six items with no one on the other end to fix them. The first table's real information is the accuracy gap between source types. Certificate record produces evidence in five of six mentions — **83%**. Archived copy produces four mentions and **none of them is evidence**; three are outright false positives. Both fall under the heading "publicly available source," and both enter the candidate list the same way, but one is nearly pure evidence and the other is nearly pure noise. The difference comes from the source's **nature**. A certificate record shows that a certificate was issued for a name, and the party who can request that is limited; an archived copy shows only that something once appeared on a page, and something that was true in the past does not have to be true today. The `old-field-app` item that came out of the archive once belonged to the organization, was handed off, and is now the responsibility of another party. The `support-portal` and `dealer-portal` items sitting in the same list, meanwhile, entered it only because of **name similarity**. #### The Step That Leaves a Trace and the One That Does not The second table opens the column that will carry through this course: how many **records** a step produces on the defense side. **Five of the six** source types leave zero records on the target, because the item sits in a third party's hands; there is no trace of these steps in the target's log at all. Only the twelve items taken from the organization's own endpoint produce twelve access records. That these twelve records produce **0 alerts** is the table's second finding. Fetching a portal's twelve pages in sequence is indistinguishable from what a normal visitor would do; on the defense side, these rows sit inside the noise. From the blue team's perspective, the conclusion is this: most of the reconnaissance phase happens where the defense cannot see it, and the visible part stays below the threshold. **Detecting this phase is not a matter of alerts, but a matter of retrospective correlation.** The same table's `step` column gives the cost. Forty-five validation steps were spent, and nine pieces of evidence came out of them: **5.00 steps per evidence item**. The distribution is not even. Certificate record spends 1.20 steps per evidence item; public code repo spends 4.00 steps; archived copy **spends 12 steps and produces no evidence at all**. A reconnaissance effort that orders sources by habit rather than accuracy sinks a quarter of its budget into a source with zero return. #### Two Sides of Adding a Source The third table measures each source's **indispensability**. When three of the six sources are dropped, no evidence is lost at all: document metadata, archived copy, and the organization's own website share their evidence with other sources. Dropping certificate record loses **2 pieces of evidence** (22.2%), and job posting and public code repo each take away one piece of evidence. So reaching all nine pieces of evidence does not require six sources — but the fact that three of them are individually dispensable does not mean the three of them can be dropped **together**; the next table shows that four are enough. The fourth table shows this cumulatively, and it is the lesson's sharpest finding. When sources are added one by one in order of accuracy, the evidence count moves 5 → 7 → 8 → **9** and **stops growing entirely** after the fourth source. The candidate count, by contrast, climbs from 13 to 16 to 20, and false positives rise from 2 to 6. Accuracy drops from **69% to 45%** over those same two steps. The last two sources do this: they add seven new candidates, bring **zero new evidence**, produce four new false positives, and leave **12 records** on the defense side. The reconnaissance-phase assumption that "more sources means better coverage" breaks down in this table. Adding a source stops increasing coverage past a certain point; what it increases is the **elimination burden**. #### The Lead Class Stops at the Scope Boundary Five of the twenty candidates are neither evidence nor false positives. The old subscriber portal, marketing page, cloud storage bucket, identity provider endpoint, and training environment **serve** the organization but do not sit in the organization's inventory; all of them run on third-party infrastructure. These five items are a quarter of the candidate list, and under a binary elimination rule — "evidence if it is in the inventory, noise if it is not" — they fall on the wrong side. The reason this class is kept separate is not measurement, it is **authorization**. An item may well serve the organization, but acting on it means touching infrastructure that belongs to the third party who owns that item, and the written authorization granted does not cover that party. Each of the five lead items is an item that **touches the scope boundary**, and none of them can be tested without the scope owner's approval. That is also where it differs from a false positive: a false positive **drops out** of the list, while a lead stays on the list and **waits for a decision**. Numerically, it comes to this: nine pieces of evidence can be acted on directly, six false positives get eliminated, and five items move to a scope conversation. Reconnaissance's output is not a single list, it is **three separate lists**; a report that collapses the three into one either works on an item it should not, or drops an item it should have worked on. #### Four Items Invisible from Outside The last line gives the measurement's upper bound. There are thirteen items in the inventory, and open-source intelligence sees **nine of them** — **69.2%**. The internal network management endpoint, backup endpoint, measurement store, and identity store do not appear in any publicly available item. These four items say two things by their absence. First, open-source intelligence's coverage is limited to the organization's **outward-facing side**; assets that sit internally and leave no external signature do not enter the list at this stage. Second, the list produced at this stage is not an **inventory**, it is a projection of the inventory as seen from outside. In an assessment that cannot be compared against the inventory the scope owner provides, this 69.2% is never known at all; the only thing that is known is twenty candidates, and that nine of those twenty are evidence can only be said while a referee exists. #### Summary - This course's unit of measure is not the candidate, it is the evidence: 29 publicly available items produced 26 mentions and 20 unique candidates; 9 are evidence, 5 are leads, 6 are false positives (30.0%). - The accuracy gap between source types is large: certificate record produces 83% evidence while archived copy produces 0%, and three of its four mentions are false positives. - Five of the six source types leave zero records on the target; the 12 items taken from the organization's own endpoint produce 12 access records and **0 alerts** — this phase stays below the threshold. - The validation cost is 5.00 steps per evidence item and is not evenly distributed: certificate record spends 1.20 steps, archived copy spends 12 steps and produces no evidence at all. - Evidence count flattens at 9 after four sources; the last two sources add 7 candidates, 0 evidence, 4 false positives, and 12 records, dropping accuracy from 69% to 45%. - Only 9 of the inventory's 13 items are visible from outside (69.2%); four items do not appear in any publicly available source. #### Next Step What separated nine of the twenty candidates as evidence and six as false positives in this lesson was a referee that came from outside: the inventory the scope owner approved. In a real assessment, that inventory is the assessment's **output**, not its input. So the question is this: without a referee, how is it shown that a candidate belongs to the organization? For some of them, it can be shown. Domain name and registration systems carry verifiable traces of who controls a name; some records can only be written by the name's owner, while others point to a third party hosting the name and prove nothing about ownership. The next lesson treats records as **inventory evidence**: the same twenty candidates are run through registration lookup, and it is counted how many have ownership that can be proven, how many stay at the lead level only, and how many records this lookup leaves on the target side. ### Domain Name and Registration Lookup Source: https://academia.sh/en/courses/penetration-testing/domain-name-and-registration-lookup Measuring a record's evidentiary strength: four record criteria's hit rates on 20 candidates ranging from 100% to 25%, the three strong criteria combined splitting off 10 candidates and capturing 8 evidence with zero false positives, the weak fourth criterion bringing 0 new evidence and dropping the hit rate from 80% to 62%, 60% of the 10 silent candidates being false positives while one is a real asset, and the cache path leaving 0 records while missing 1 evidence item and adding 2 false positives. The previous lesson split twenty candidates into nine evidence, five leads, and six false positives — but what made that split was a referee that came from outside: the inventory the scope owner approved. In a real assessment, that inventory is the output, not the input. This lesson tries a split without a referee: the candidates' names are queried, and the **evidentiary strength** of the records that come back is measured. What record types are for and how they are managed was covered in the Endpoint and System Hardening course. What is measured here is different: a record carries a **claim**, and the claim's value depends on who could write that record. Some records can only be created by the party controlling the name; others are the business of the third party hosting the name and say nothing about ownership. #### A Record Is a Claim, Not Evidence **RE4:** four pieces of information are kept for each candidate — the real class the referee assigned, whether the organization's name appears in the registration record, whether a verification record exists that only the party controlling the name could write, and the address block the name resolves to. **RE5:** a *criterion* is a rule that counts a candidate as evidence; it is referred to by two numbers — **hit rate** (of the candidates it triggers, how many are actually evidence) and **coverage** (how much of all the evidence it captures). **RE6:** a response from the cache leaves no record on the target but can be stale; a response queried directly from the authoritative server is fresh, and every query leaves a record. This lookup is part of an authorized assessment and is limited strictly to the names within the defined scope. ```js // recon/record.mjs — fictional record-lookup results for the twenty candidates from lesson 01 // (a model): no real domain name, no real address, no real registration record. No randomness. // RE4: each row carries four things for a candidate — the real class per the scope owner's // inventory (referee), whether the organization's name appears in the registration record, // whether a verification record exists that only the party controlling the name could write, // and the address block the name resolves to. Block A is org-assigned, B is shared hosting, // C is third party, "-" means it does not resolve. "zone" says who answers the name's query, // and "stale" gives the cache's outdated block value. export const CANDIDATE = ` subscriber-portal evidence 1 1 A org - field-app evidence 1 1 A org - branch-vpn-endpoint evidence 1 0 A org - collector-gateway evidence 0 0 A org C report-endpoint evidence 1 1 A org - test-portal evidence 0 0 C org - shared-interface evidence 1 0 B org - work-order-endpoint evidence 1 1 A org - meter-update-endpoint evidence 0 1 A org - old-subscriber-portal lead 1 0 C org - marketing-page lead 0 0 C org - cloud-storage-bucket lead 0 0 C outside - identity-provider-endpoint lead 0 1 C outside - training-environment lead 0 0 B org - payment-provider-endpoint falseP 0 0 C outside - old-field-app falseP 0 0 B outside - spare-domain falseP 0 0 - outside - dealer-portal falseP 0 0 B outside - support-portal falseP 0 0 C outside A old-report-endpoint falseP 0 0 - org A` .trim().split("\n").map((r) => { const [name, cls, registration, ownership, block, zone, stale] = r.trim().split(/\s+/); return { name, class: cls, registration: +registration, ownership: +ownership, block, zone, stale }; }); // RE5: a rule is a criterion that counts a candidate as evidence. Hit rate = of the triggered // candidates, how many are actually evidence; coverage = how much of all the evidence this // rule alone finds. export const RULE = [ { name: "ownership record", f: (a, b) => a.ownership === 1 }, { name: "registration record", f: (a, b) => a.registration === 1 }, { name: "org-assigned block", f: (a, b) => b === "A" }, { name: "shared hosting block", f: (a, b) => b === "B" }, ]; // RE6: a response from the cache leaves no record on the target but can be stale; a response // queried directly from the authoritative server is fresh and leaves two records for every // name in the org's zone. export const blockOf = (a, path) => (path === "cache" && a.stale !== "-" ? a.stale : a.block); export const EVIDENCE = CANDIDATE.filter((a) => a.class === "evidence").length; ``` ```js // recon/ownership.mjs — which record proves ownership, which one only leaves a lead. import { CANDIDATE, RULE, EVIDENCE, blockOf } from "./record.mjs"; const write = (g, ...s) => console.log(s.map((v, i) => (g[i] < 0 ? String(v).padEnd(-g[i]) : String(v).padStart(g[i]))).join("")); const count = (l, s) => l.filter((a) => a.class === s).length; const rate = (l) => l.length ? ((count(l, "evidence") / l.length) * 100).toFixed(0) : "-"; console.log(`${CANDIDATE.length} candidates, ${EVIDENCE} evidence, ${count(CANDIDATE, "lead")} leads, ` + `${count(CANDIDATE, "falseP")} false positives; ${RULE.length} record criteria, 3 queries per candidate`); const A = [-24, 7, 7, 7, 9, 9, 10]; console.log("\n1. hit rate and coverage per record criterion"); write(A, "criterion", "cand", "evid", "lead", "falseP", "hit %", "cover %"); for (const k of RULE) { const l = CANDIDATE.filter((a) => k.f(a, a.block)); write(A, k.name, l.length, count(l, "evidence"), count(l, "lead"), count(l, "falseP"), rate(l), ((count(l, "evidence") / EVIDENCE) * 100).toFixed(1)); } const B = [-32, 7, 7, 7, 9, 9, 10]; console.log("\n2. as criteria are combined (strongest to weakest)"); write(B, "set", "cand", "evid", "lead", "falseP", "hit %", "cover %"); const combined = []; for (const k of RULE) { combined.push(k); const l = CANDIDATE.filter((a) => combined.some((x) => x.f(a, a.block))); write(B, `+ ${k.name}`, l.length, count(l, "evidence"), count(l, "lead"), count(l, "falseP"), rate(l), ((count(l, "evidence") / EVIDENCE) * 100).toFixed(1)); } const strong = RULE.slice(0, 3); const triggered = CANDIDATE.filter((a) => strong.some((k) => k.f(a, a.block))); const silent = CANDIDATE.filter((a) => !strong.some((k) => k.f(a, a.block))); const C = [-32, 7, 7, 7, 9, 10]; console.log("\n3. the two lists the first three criteria split off"); write(C, "list", "cand", "evid", "lead", "falseP", "falseP %"); for (const [name, l] of [["triggered", triggered], ["no criterion triggered", silent]]) write(C, name, l.length, count(l, "evidence"), count(l, "lead"), count(l, "falseP"), ((count(l, "falseP") / l.length) * 100).toFixed(1)); console.log("evidence not triggered: " + silent.filter((a) => a.class === "evidence").map((a) => a.name).join(", ")); const D = [-26, 9, 7, 9, 8, 8, 10]; console.log("\n4. two query paths: trace vs. certainty"); write(D, "path", "records", "alerts", "stale", "evid", "falseP", "hit %"); for (const path of ["cache", "direct"]) { const l = CANDIDATE.filter((a) => strong.some((k) => k.f(a, blockOf(a, path)))); const record = path === "direct" ? CANDIDATE.filter((a) => a.zone === "org").length * 2 : 0; const stale = path === "cache" ? CANDIDATE.filter((a) => a.stale !== "-").length : 0; write(D, path, record, 0, stale, count(l, "evidence"), count(l, "falseP"), rate(l)); } const cacheTriggered = CANDIDATE.filter((a) => strong.some((k) => k.f(a, blockOf(a, "cache")))); console.log(`evidence missed by the cache path: ` + triggered.filter((a) => !cacheTriggered.includes(a)).map((a) => a.name).join(", ") + `; false positives added: ` + cacheTriggered.filter((a) => !triggered.includes(a)).map((a) => a.name).join(", ")); ``` ``` 20 candidates, 9 evidence, 5 leads, 6 false positives; 4 record criteria, 3 queries per candidate 1. hit rate and coverage per record criterion criterion cand evid lead falseP hit % cover % ownership record 6 5 1 0 83 55.6 registration record 7 6 1 0 86 66.7 org-assigned block 7 7 0 0 100 77.8 shared hosting block 4 1 1 2 25 11.1 2. as criteria are combined (strongest to weakest) set cand evid lead falseP hit % cover % + ownership record 6 5 1 0 83 55.6 + registration record 9 7 2 0 78 77.8 + org-assigned block 10 8 2 0 80 88.9 + shared hosting block 13 8 3 2 62 88.9 3. the two lists the first three criteria split off list cand evid lead falseP falseP % triggered 10 8 2 0 0.0 no criterion triggered 10 1 3 6 60.0 evidence not triggered: test-portal 4. two query paths: trace vs. certainty path records alerts stale evid falseP hit % cache 0 0 3 7 2 64 direct 26 0 0 8 0 80 evidence missed by the cache path: collector-gateway; false positives added: support-portal, old-report-endpoint ``` The numbers belong to the **measurement** class; their inputs are RE4–RE6. #### Four Criteria, Four Different Hit Rates The first table runs all four criteria over the same twenty candidates, and their hit rates spread **between 100% and 25%**. That spread is the lesson's main finding: the sentence "a record lookup was performed" says nothing at all, because which record carries which claim changes from criterion to criterion. The org-assigned address block criterion triggers seven candidates and **all seven are evidence** — 100% hit rate. The reason is structural: if an address block is allocated to an organization, for a name resolving into that block to belong to another organization requires the block owner's cooperation. Registration record follows at 86% and ownership record at 83%; both produce evidence, but both also cover only a little more than half. The fourth criterion shows the opposite. Shared hosting block triggers four candidates; **one is evidence, one is a lead, two are false positives** — 25% hit rate. On a shared hosting block, hundreds of organizations' names resolve to the same address; the observation "it resolves to the same address" says almost nothing about ownership there. Two observations that look identical — a name resolves to an address — carry a fourfold difference between evidence and noise depending on the block's kind. The single lead item triggered by the ownership record criterion is also instructive. `identity-provider-endpoint` has a valid ownership record; the organization has proven it controls that name. The item still is not evidence, because the record proves control of the **name**, not who the **asset** belongs to. The name is the organization's; the service running on it is a third party's. #### Where the Return on Combining Stops The second table combines criteria from strongest to weakest. The first three criteria combined trigger **10 candidates** and contain 8 evidence, 2 leads, **0 false positives**: 80% hit rate, 88.9% coverage. When the fourth criterion is added, candidates rise to 13, evidence **stays at 8**, false positives rise from 0 to 2, and the hit rate drops to 62%. This is the same curve as the previous lesson's, and here its shape is stricter: the weak criterion **brings in no new evidence at all**, it only adds elimination burden. The coverage column reads 88.9% on both rows. The question to ask when adding a new rule to a criterion set is not "does it find more candidates," it should be "**does it increase coverage**"; in this table the fourth criterion's contribution to coverage is zero. #### The Ten Candidates That Stay Silent The third table shows the other half of the list. None of the three strong criteria fire for **10 candidates**, and **6 of these ten are false positives** — 60%. So record lookup really does work as an elimination tool: **all six** of the six false positives are on this silent list, and none of them has leaked into the evidence list. In return, one real asset is lost. `test-portal` belongs to the organization, sits in the inventory, and no criterion triggers for it: it is not in a dedicated block because it is hosted by a third party, it does not appear in the registration record, and it has no ownership record. Record lookup tells this candidate "I cannot say anything about it," and a binary elimination rule reads that as "does not belong to the organization." The numeric summary is this: eight of nine pieces of evidence can be confirmed by a record (88.9%); one cannot be confirmed and falls into the silent list. The silent list **is not a list to discard, it is a separate list**; inside it, the 10% real asset stands next to the 60% noise. #### The Trade-off Between Trace and Freshness The fourth table opens this course's trace column for the second time, and for the first time shows a **trade-off**. If queries are served from a resolver's cache, **0 records** remain on the target; if the target's authoritative server is queried directly, two queries for each of the thirteen names in the org's zone produce **26 records**. The path that looks free is not free. The cache holds three stale responses, and the result breaks in three places: `collector-gateway` stops being evidence (the stale response places it in the third-party block), while `support-portal` and `old-report-endpoint` show up in the org-assigned block because of their stale records and **enter the evidence list as false positives**. The hit rate drops from 80% to 64%. The choice between the two paths carries a pattern this course will run into again and again: **leaving no trace means giving up certainty**. Read from the blue team's side, it is symmetric too — the 26 records on the authoritative server produce no alert at all, because a name query is the network's most ordinary event. This step is visible on the defense side but not noticed; noticing it depends on later correlating **which names** the queries targeted and in what order. #### The Cost of Sixty Queries Each of the twenty candidates required three queries: registration record, address, and ownership record. A total of **60 queries** produced eight pieces of evidence — **7.5 queries per evidence item**. The previous lesson's validation cost was 5.00 steps per evidence item; this lesson runs above that cost and gives something back in return: the previous lesson's nine pieces of evidence were read from a referee, this lesson's eight **verify themselves**. The cost is not evenly distributed either. The ten candidates that return no record still spend their three-query share; the silent list's total cost is 30 queries and it produces **zero evidence** in return. Half of the validation budget goes to a list whose outcome is "unknown" — and this spending is not wasted, because it is what drops six false positives from the list. There is also a boundary worth distinguishing in terms of scope: **seven** of the twenty names sit outside the organization's zone. Querying these names is reading publicly available data and creates no scope problem; **acting on** those names, however, does step outside scope. An item that is free to query is not necessarily an item that is free to work on. #### Summary - A record carries a claim, and the claim's strength depends on who can write the record: the four criteria's hit rates are 100%, 86%, 83%, and 25%. - The org-assigned address block criterion is correct on 7 of 7 candidates; the shared hosting block criterion only on 1 of 4 — the same kind of observation carries a fourfold difference in value depending on the block's kind. - Ownership record proves control of the name, not ownership of the asset: a candidate carrying a valid ownership record is a third party's service and does not count as evidence. - The three strong criteria combined trigger 10 candidates, capture 8 evidence, and produce 0 false positives; the fourth criterion adds 3 candidates and 2 false positives, brings 0 new evidence, and drops the hit rate from 80% to 62%. - 6 of the 10 candidates no criterion triggers for are false positives (60%), but one is a real asset; the silent list is not a list to discard, it is a separate list. - The cache path leaves 0 records on the target, misses 1 evidence item due to 3 stale responses, and adds 2 false positives; the direct path leaves 26 records, produces 0 alerts, and keeps the hit rate at 80%. #### Next Step Up to this point the candidate list arrived ready-made: twenty names came out of the previous lesson's publicly available items, and this lesson only classified them. But the list is not a closed set. Under an organization's domain name there can be names that appear in no publicly available document and have a link on no page; the way to find them is not reading documents, it is **scanning the name space**. This scan produces two new numbers. The first is coverage: how many candidates do different discovery methods pull out of the same name space, how much do they overlap, and which one alone is enough. The second is this course's most attention-grabbing item: some of the names found point to **a target that no longer exists** — items whose record has stopped, whose counterpart has fallen away, whose owner is unclear. The next lesson measures subdomain and asset discovery along these two axes and counts how many records each method leaves on the defense side. ### Subdomain and Asset Discovery Source: https://academia.sh/en/courses/penetration-testing/subdomain-and-asset-discovery Counting dangling items in the name space: 7 of 24 zone records pointing to a target that no longer responds, three methods' coverage ranging from 66.7% to 29.2%, the method that leaves no trace finding the most candidates, the 120-attempt method producing 8.57 records per name found, linked-item inference finding none of the dangling items, the request that triggers 4 alerts bringing back 0 candidates, and the combination finding 21 records while never seeing 3 of them. The previous two lessons worked on a closed list: twenty candidates came out of publicly available items and were classified by record lookup. Under an organization's domain name, though, sit names that appear in no document and have a link on no page. This lesson searches for those names and counts two things at once while doing it: the methods' coverage, and records that **point to a target that no longer exists**. That a zone file is kept as an inventory and that orphaned records are cleaned up was covered from the zone owner's side in the Endpoint and System Hardening course. Here the same zone is viewed **from the outside**: an item the owner has not cleaned up because they do not know about it sits as a candidate on the assessing party's list. #### Dangling Record and the Forgotten Item **RE7:** each name carries three things — whether its target responds, whether it is in the inventory, and its responsible team. A record whose target does not respond is **dangling**: the record stands, but what it pointed to has fallen away. A record whose target responds but has no owner is **forgotten**: it works, but no one is responsible for it. **RE8:** a method produces a candidate set; if a candidate has no counterpart in the zone, it is a false positive. **RE9:** a method's trace is the record it leaves on the target and the number of alerts it triggers; an attempt count can produce trace even when it produces no candidate. All of these steps are limited to the domain name in the scope definition and cannot be done in a name space that falls outside written authorization. ```js // recon/zone.mjs — a fictional measurement network's subdomain records and three discovery // methods' findings (a model): no real domain name, no real name server, no real target. No // randomness. // RE7: each row is a name — whether its target responds, whether it is in the inventory, its // responsible team, and the methods that found it. A record whose target does not respond counts // as dangling; one whose target responds but has no owner counts as forgotten. In the last // column p is passive record compilation, s is dictionary attempt, b is linked-item inference; // "-" means that method did not find it. export const NAME = ` subscriber live 1 portal psb field live 1 field psb vpn live 1 network ps- collector live 1 measurement p-- report live 1 data psb test live 1 portal psb shared live 1 portal p-b work-order live 1 field p-b meter live 1 measurement p-- old-subscriber live 0 - p-- marketing live 0 marketing -sb training live 0 - -s- demo dead 0 - ps- old-report dead 0 - p-- campaign-2 dead 0 - p-- archive dead 0 - ps- staging live 0 data ps- backup dead 1 network -s- monitoring live 1 network ps- log live 1 data -s- survey dead 0 - -s- old-field dead 0 - --- branch-1 live 1 network --- branch-2 live 1 network --- campaign-1 none 0 - p--` .trim().split("\n").map((r) => { const [name, target, inv, owner, found] = r.trim().split(/\s+/); return { name, target, inv: +inv, owner, found }; }); // RE8: a method produces a candidate set; if a candidate has no counterpart in the zone, it is a // false positive. RE9: a method's trace is the record it leaves on the target and the alerts it // triggers. An attempt count can produce trace even when it produces no candidate: the zone // transfer request is refused by all four name servers and returns no names at all. export const METHOD = [ { code: "p", name: "passive compilation", attempts: 0, records: 0, alerts: 0 }, { code: "s", name: "dictionary attempt", attempts: 120, records: 120, alerts: 1 }, { code: "b", name: "linked-item inference", attempts: 12, records: 12, alerts: 0 }, { code: "z", name: "zone transfer request", attempts: 4, records: 4, alerts: 4 }, ]; export const RECORD = NAME.filter((a) => a.target !== "none"); export const finds = (y, a) => a.found.includes(y.code); export const dangling = (a) => a.target === "dead"; export const forgotten = (a) => a.target === "live" && a.owner === "-"; ``` ```js // recon/asset.mjs — three discovery methods over the same name space: how many candidates, how // many dangling items, how much trace. import { NAME, METHOD, RECORD, finds, dangling, forgotten } from "./zone.mjs"; const write = (g, ...s) => console.log(s.map((v, i) => (g[i] < 0 ? String(v).padEnd(-g[i]) : String(v).padStart(g[i]))).join("")); const DANGLING = RECORD.filter(dangling).length; console.log(`${RECORD.length} zone records: ${RECORD.filter((a) => a.target === "live").length} ` + `targets respond, ${DANGLING} dangling, ${RECORD.filter(forgotten).length} forgotten, ` + `${RECORD.filter((a) => !a.inv && a.target === "live").length} outside inventory`); const A = [-23, 9, 7, 8, 9, 10, 10]; console.log("\n1. candidates and dangling items per method"); write(A, "method", "attempts", "cand", "true", "falseP", "dangling", "cover %"); for (const y of METHOD) { const l = NAME.filter((a) => finds(y, a)); const d = l.filter((a) => a.target !== "none"); write(A, y.name, y.attempts, l.length, d.length, l.length - d.length, d.filter(dangling).length, ((d.length / RECORD.length) * 100).toFixed(1)); } const B = [-23, 8, 8, 11, 15, 14]; console.log("\n2. the trace a method leaves"); write(B, "method", "records", "alerts", "rec/cand", "rec/dangling", "cand/attempt"); for (const y of METHOD) { const d = NAME.filter((a) => finds(y, a) && a.target !== "none"); const sz = d.filter(dangling).length; write(B, y.name, y.records, y.alerts, d.length ? (y.records / d.length).toFixed(2) : "-", sz ? (y.records / sz).toFixed(2) : "-", y.attempts ? (d.length / y.attempts).toFixed(2) : "-"); } const TRACE = METHOD.reduce((t, y) => t + y.records, 0); const ALERTS = METHOD.reduce((t, y) => t + y.alerts, 0); const C = [-23, 9, 9, 11, 11]; console.log("\n3. combination and overlap"); write(C, "set", "cand", "true", "dangling", "cover %"); const set = []; for (const y of METHOD) { set.push(y); const l = NAME.filter((a) => set.some((x) => finds(x, a))); const d = l.filter((a) => a.target !== "none"); write(C, `+ ${y.name}`, l.length, d.length, d.filter(dangling).length, ((d.length / RECORD.length) * 100).toFixed(1)); } const howMany = (a) => METHOD.filter((y) => finds(y, a)).length; const D = [-23, 9, 11, 12]; console.log("\n4. how many methods find it"); write(D, "method count", "records", "dangling", "forgotten"); for (const n of [0, 1, 2, 3]) { const l = RECORD.filter((a) => howMany(a) === n); write(D, `${n} methods`, l.length, l.filter(dangling).length, l.filter(forgotten).length); } const found = RECORD.filter((a) => howMany(a) > 0); console.log(`record no method found: ` + RECORD.filter((a) => !howMany(a)).map((a) => a.name).join(", ")); console.log(`combination finds ${found.length}/${RECORD.length} records (` + `${((found.length / RECORD.length) * 100).toFixed(1)}%) and ` + `${found.filter(dangling).length}/${DANGLING} dangling items; ` + `total trace ${TRACE} records, ${ALERTS} alerts`); ``` ``` 24 zone records: 17 targets respond, 7 dangling, 2 forgotten, 4 outside inventory 1. candidates and dangling items per method method attempts cand true falseP dangling cover % passive compilation 0 17 16 1 4 66.7 dictionary attempt 120 14 14 0 4 58.3 linked-item inference 12 7 7 0 0 29.2 zone transfer request 4 0 0 0 0 0.0 2. the trace a method leaves method records alerts rec/cand rec/dangling cand/attempt passive compilation 0 0 0.00 0.00 - dictionary attempt 120 1 8.57 30.00 0.12 linked-item inference 12 0 1.71 - 0.58 zone transfer request 4 4 - - 0.00 3. combination and overlap set cand true dangling cover % + passive compilation 17 16 4 66.7 + dictionary attempt 22 21 6 87.5 + linked-item inference 22 21 6 87.5 + zone transfer request 22 21 6 87.5 4. how many methods find it method count records dangling forgotten 0 methods 3 1 0 1 methods 9 4 2 2 methods 8 2 0 3 methods 4 0 0 record no method found: old-field, branch-1, branch-2 combination finds 21/24 records (87.5%) and 6/7 dangling items; total trace 136 records, 5 alerts ``` The numbers belong to the **measurement** class; their inputs are RE7–RE9. #### Seven Dangling Records Twenty-four zone records — **7 of them are dangling**: the record stands, but the target it points to does not respond. The rate is 29.2% — one in every three records in the zone points at nothing. Alongside these there are **2 forgotten items** whose target responds but has no owner, and **4 items** that never appear in the inventory at all yet still run. These three classes do not carry the same weight for assessment purposes. The four items outside the inventory are a scope problem: they run, they serve the organization, but their names never came up in the scope conversation. The two forgotten items are a responsibility gap: if a finding comes out of them, who the report gets written to stays unclear. The seven dangling records are the most attention-grabbing, though, because **a record is a promise binding a name to a target**, and when the target disappears, that promise stays open to whichever other party is ready to claim it. The party keeping the record is the organization; the party providing the target is no one anymore. #### Three Methods, Three Different Coverages The first table runs all three methods over the same name space. Passive record compilation produces 17 candidates, 16 of them have a counterpart in the zone, and its coverage is **66.7%**. Dictionary attempt produces 14 candidates and covers 58.3%. Linked-item inference produces only 7 candidates — **29.2%**. The difference does not come from the methods' quality, it comes from **what they look at**. Linked-item inference follows links running from one page to another; a name is found only if someone linked to it. Dangling items are exactly the items no one links to, which is why this method finds **none of the seven dangling records**. In the same column, passive compilation produces 4 and dictionary attempt produces 4 dangling items. A method is eliminated not because it is "the one producing the fewest candidates," but because it is **structurally blind to the class it is looking for**. The false-positive column is filled by a single item. `campaign-1`, produced by passive compilation, does not exist in the zone at all: it was once entered into a certificate record, and the name was later removed from the zone. This property of passive sources is the same as the archived copy from the previous lesson — something that was true in the past does not have to be true today. #### One Hundred Twenty Attempts, Fourteen Names The second table ties this course's trace column to a method choice for the first time. Passive record compilation leaves **zero records** on the target and is the method that finds the most candidates. Dictionary attempt makes 120 attempts, finds 14 names, and leaves **120 records** on the target; **8.57 records** per name, **30.00 records** per dangling item found. Linked-item inference leaves 12 records and produces 1.71 trace per name. Dictionary attempt's hit rate shows up in the `cand/attempt` column: **0.12**. A hundred and six of the hundred twenty attempts go to waste, and the wasted attempts leave records too. Read from the defense side: the blue team notices this method not by looking at queries for names that exist, but by looking at **the density of queries for names that do not exist**. The single alert produced across this lesson is triggered by exactly that density. The fourth row is the lesson's sharpest comparison. The zone transfer request is refused by four name servers: **0 candidates**, 4 records, **4 alerts**. Four of the total five alerts come from this single step that returns no names at all. Had it been allowed, it would have handed over the entire zone in one step; when it is not, all that is left is trace. A step attempted because its expected return is high becomes **pure cost** once its realized return turns out to be zero. #### Where the Combination Stops The third table combines the methods. The first two together find 21 records and coverage climbs from 66.7% to **87.5%**; dangling items rise from 4 to 6. When the third and fourth methods are added, all three numbers **do not change at all**. All seven names linked-item inference finds have already come out of the first two methods. The fourth table shows this item by item. Of the twenty-four records, 4 come from all three methods, 8 from two, 9 from only one, and **3 from none**. Of the nine records that come from only one method, 4 are dangling and 2 are forgotten: **the items the organization is least aware of are the items the fewest methods see.** None of the four records that come from all three methods is dangling or forgotten; they are the zone's most visible, best-maintained names. The three records no method finds mark the list's boundary. `branch-1` and `branch-2` are numbered names with no dictionary counterpart; `old-field` is both dangling and has left no trace in any passive record. So **six** of the seven dangling items are found, one is not — and in this run, that it is not found is only known because the referee inventory is on hand. In a real assessment, this record's existence would never be known at all; the only thing known would be the 21 names. #### A Dangling Record Is not Evidence Yet Finding six dangling records means, by this course's measure, six **candidates**, not six findings. A record's target might not respond because the target is temporarily down, because the name only resolves from inside the organization's network, or because the space the target sits in is available for another party to reclaim. Only the third case is a finding that belongs in a report; the first two are false positives, and none of the discovery steps can tell the three apart. The cost of making that distinction is two items. The first is an extra step: for every dangling record, whether the service the target sits in is open to reclaiming has to be tested. The second, and more restrictive, is **scope**: that test is run not on the organization's infrastructure but on the third party's. None of the targets the six dangling records point to is an asset named in the scope definition. This is the second encounter with the rule written for the lead class in the previous lesson, and it is sharper here: **finding** an item is unrestricted, **confirming** it requires separate authorization. How many of the six candidates turn into evidence is not settled in this lesson — it is settled after the scope conversation. #### Summary - Of twenty-four zone records, 7 are dangling (29.2%): the record stands, its target does not respond. Alongside them sit 2 forgotten and 4 outside-inventory items. - The three methods' coverage is 66.7%, 58.3%, and 29.2%; linked-item inference finds none of the seven dangling records, because a dangling item is exactly one no one links to. - Passive record compilation leaves 0 records on the target and finds the most candidates; its single false positive is a name already removed from the zone. - Dictionary attempt finds 14 names with 120 attempts (0.12 hit rate), leaves 8.57 records per name and 30.00 per dangling item found; one of this course's five alerts comes from its density. - The zone transfer request brings back 0 candidates and leaves 4 records and 4 alerts: 80% of all alerts come from this single step that produces no names. - The combination plateaus at 87.5% after two methods; of the 9 records from a single method, 4 are dangling and 2 are forgotten, while none of the 4 records from all three methods are. #### Next Step At the end of three lessons the externally visible name list is ready: twenty-one names, six of them dangling. A name is not an asset, though. The name `subscriber` resolves to a target, but **what runs** on that target has never been asked until now. The machine a record points to might accept no connections at all; it might serve three services at once; or it might keep a service reachable only from the organization's network closed off from the outside. All three look identical in the same record. The step that tells these apart is testing which ports the target responds on, and this step differs from the previous three: it sends requests **directly** and **in large numbers** to the target. The next lesson compares scan types on a single axis — how many candidates each type finds, how many it misses, how many records and alerts it produces on the defense side. This lesson accumulated 136 records and 5 alerts; in that lesson both numbers move into a different order of magnitude. ### Port Scanning Source: https://academia.sh/en/courses/penetration-testing/port-scanning Comparing scan types on the trace axis: in a 96-port space, 23 open ports being definitively distinguished by type between 78.3% and 0%, the one-way probe producing 21 ambiguous candidates and 3 false positives, the trace per piece of evidence ranging from 3.13 to 8.00 records, the same type slowed down producing the same 18 pieces of evidence with the same 72 records but 0 alerts instead of 2, and 91.3% coverage requiring a combination of two types. At the end of three lessons the externally visible name list is ready, but a name is not an asset. A record resolves to a target; that target might accept no connections at all, or it might offer three services at once. The two look identical at the record level. This lesson covers the step that tests the difference, and the point where it departs from the previous three is this: requests go to the target **directly** and **in large numbers**. This changes the weight of the measurement too. The trace accumulated so far was 136 records and 5 alerts. In this lesson a single scan type on its own can exceed half that number. The comparison is made on this axis: how many open ports each type definitively distinguishes, how many it leaves ambiguous, how many it misses, and how many records and alerts it produces on the defense side. This step can only be done against endpoints covered by written authorization and the scope definition; even a single request sent to an out-of-scope endpoint invalidates the authorization. #### What Defines a Type Is What It Distinguishes Scan types are referred to here not as a tool, a command, or an option, but as **discriminating power**. What defines a type is which port state it definitively distinguishes and which it leaves ambiguous. **RE10:** the scan space is 9 connection-oriented and 3 connectionless ports per endpoint; the common port list is 4 ports per endpoint. A port is in the `open`, `filtered`, or `closed` state. **RE11:** a type is defined by which state of its space it **definitively** distinguishes and which it leaves **ambiguous**; among the ports left ambiguous, the ones that are not actually open are false positives. **RE12:** the trace is three items — every attempt is a network record, every open port whose connection completes is an application record, and an alert follows from the speed threshold and from whether an application record exists. ```js // recon/port.mjs — a fictional eight-endpoint port state and five scan types (a model): no real // host, no real address, and no real service. No tool name, command, or flag sequence either: // scan types are defined only by "which state it definitively distinguishes, which it leaves // ambiguous." // RE10: the scan space is 9 connection-oriented (conn) and 3 connectionless (disc) ports per endpoint; the // common port list is 4 ports per endpoint. Only the non-closed ports are listed below, the rest // are closed. "response" says whether the connectionless port answered the probe. export const ENDPOINT = ["subscriber", "field", "vpn", "collector", "report", "shared", "monitoring", "log"]; export const SPACE = { conn: ENDPOINT.length * 9, disc: ENDPOINT.length * 3, common: ENDPOINT.length * 4 }; export const PORT = ` subscriber n1 conn open 1 - subscriber n2 conn open 1 - subscriber n3 conn open 0 - subscriber n4 conn filtered 1 - field n1 conn open 1 - field n5 conn open 0 - field n6 conn open 1 - field b1 disc open 1 yes vpn n1 conn filtered 1 - vpn n6 conn open 1 - vpn b2 disc open 0 no collector n6 conn open 1 - collector n7 conn open 0 - collector n8 conn open 0 - collector b1 disc open 1 yes report n1 conn open 1 - report n6 conn open 1 - report n9 conn filtered 0 - shared n1 conn open 1 - shared n3 conn open 0 - monitoring n1 conn open 1 - monitoring n7 conn open 0 - monitoring b1 disc open 1 yes log n1 conn open 1 - log n5 conn open 0 - log b3 disc open 1 no` .trim().split("\n").map((r) => { const [endpoint, id, proto, state, common, response] = r.trim().split(/\s+/); return { endpoint, id, proto, state, common: +common, response }; }); export const OPEN = PORT.filter((n) => n.state === "open"); const isOpen = (n) => n.state === "open"; const isResponsive = (n) => n.state === "open" && n.response === "yes"; // RE11: a scan type is defined by which state of its space it definitively distinguishes and // which it leaves ambiguous. "unit" is a fictional time unit for one attempt, "app" is whether // the connection completes and leaves a record at the application layer. export const TYPE = [ { name: "full connection", space: "conn", unit: 3, app: 1, confirmed: isOpen, ambiguous: () => false }, { name: "half connection", space: "conn", unit: 1, app: 0, confirmed: isOpen, ambiguous: () => false }, { name: "one-way probe", space: "conn", unit: 1, app: 0, confirmed: () => false, ambiguous: (n) => n.state !== "closed" }, { name: "connectionless protocol", space: "disc", unit: 5, app: 0, confirmed: isResponsive, ambiguous: (n) => isOpen(n) && !isResponsive(n) }, { name: "limited list", space: "common", unit: 3, app: 1, confirmed: isOpen, ambiguous: () => false }, { name: "half connection (slow)", space: "conn", unit: 10, app: 0, confirmed: isOpen, ambiguous: () => false }, ]; export const inScope = (t, n) => t.space === "common" ? n.common === 1 : n.proto === t.space; // RE12: the trace is three items — every attempt is a network record, every open port whose // connection completes is an application record, and an alert follows from the speed threshold // and from whether an application record exists. export const measure = (t) => { const seen = PORT.filter((n) => inScope(t, n)); const confirmed = seen.filter((n) => t.confirmed(n)); const ambiguous = seen.filter((n) => t.ambiguous(n)); const attempts = SPACE[t.space]; const duration = attempts * t.unit; const rate = attempts / duration; const netRecords = attempts; const appRecords = t.app * confirmed.length; const alerts = (rate > 0.5 ? 2 : rate > 0.25 ? 1 : 0) + (appRecords > 0 ? 1 : 0); return { t, attempts, confirmed, ambiguous, duration, rate, netRecords, appRecords, records: netRecords + appRecords, alerts, falseP: ambiguous.filter((n) => n.state !== "open").length, missed: OPEN.length - confirmed.length - ambiguous.filter(isOpen).length }; }; ``` ```js // recon/scan.mjs — scan types on the trace axis: how many candidates, how many missed, how many // records, how many alerts. import { PORT, OPEN, TYPE, SPACE, measure } from "./port.mjs"; const write = (g, ...s) => console.log(s.map((v, i) => (g[i] < 0 ? String(v).padEnd(-g[i]) : String(v).padStart(g[i]))).join("")); const M = TYPE.map(measure); console.log(`${SPACE.conn} connection-oriented + ${SPACE.disc} connectionless port space, ` + `${PORT.length} not-closed ports: ${OPEN.length} open, ` + `${PORT.filter((n) => n.state === "filtered").length} filtered`); const A = [-23, 8, 8, 10, 9, 11, 10]; console.log("\n1. found and missed per scan type"); write(A, "scan type", "attempts", "conf", "ambiguous", "falseP", "missed", "cover %"); for (const o of M) write(A, o.t.name, o.attempts, o.confirmed.length, o.ambiguous.length, o.falseP, o.missed, ((o.confirmed.length / OPEN.length) * 100).toFixed(1)); const B = [-23, 9, 10, 8, 9, 8, 7]; console.log("\n2. the trace left on the defense side"); write(B, "scan type", "net rec", "app rec", "records", "duration", "rate", "alerts"); for (const o of M) write(B, o.t.name, o.netRecords, o.appRecords, o.records, o.duration, o.rate.toFixed(2), o.alerts); const C = [-23, 12, 12, 12]; console.log("\n3. cost per piece of evidence"); write(C, "scan type", "records/conf", "dur/conf", "alerts/conf"); for (const o of M) { const k = o.confirmed.length; write(C, o.t.name, k ? (o.records / k).toFixed(2) : "-", k ? (o.duration / k).toFixed(2) : "-", k ? (o.alerts / k).toFixed(3) : "-"); } const D = [-50, 8, 10, 8, 9, 8]; console.log("\n4. combining two scan types"); write(D, "combination", "conf", "cover %", "records", "duration", "alerts"); const pair = [[1, 3], [4, 3], [5, 3], [0, 3]]; for (const [i, j] of pair) { const k = new Set([...M[i].confirmed, ...M[j].confirmed]).size; write(D, `${M[i].t.name} + ${M[j].t.name}`, k, ((k / OPEN.length) * 100).toFixed(1), M[i].records + M[j].records, M[i].duration + M[j].duration, M[i].alerts + M[j].alerts); } const slow = M[5], fast = M[1]; console.log(`same type slowed down: confirmed ${fast.confirmed.length} -> ${slow.confirmed.length}, ` + `records ${fast.records} -> ${slow.records}, duration ${fast.duration} -> ${slow.duration}, ` + `alerts ${fast.alerts} -> ${slow.alerts}`); ``` ``` 72 connection-oriented + 24 connectionless port space, 26 not-closed ports: 23 open, 3 filtered 1. found and missed per scan type scan type attempts conf ambiguous falseP missed cover % full connection 72 18 0 0 5 78.3 half connection 72 18 0 0 5 78.3 one-way probe 72 0 21 3 5 0.0 connectionless protocol 24 3 2 0 18 13.0 limited list 32 15 0 0 8 65.2 half connection (slow) 72 18 0 0 5 78.3 2. the trace left on the defense side scan type net rec app rec records duration rate alerts full connection 72 18 90 216 0.33 2 half connection 72 0 72 72 1.00 2 one-way probe 72 0 72 72 1.00 2 connectionless protocol 24 0 24 120 0.20 0 limited list 32 15 47 96 0.33 2 half connection (slow) 72 0 72 720 0.10 0 3. cost per piece of evidence scan type records/conf dur/conf alerts/conf full connection 5.00 12.00 0.111 half connection 4.00 4.00 0.111 one-way probe - - - connectionless protocol 8.00 40.00 0.000 limited list 3.13 6.40 0.133 half connection (slow) 4.00 40.00 0.000 4. combining two scan types combination conf cover % records duration alerts half connection + connectionless protocol 21 91.3 96 192 2 limited list + connectionless protocol 15 65.2 71 216 2 half connection (slow) + connectionless protocol 21 91.3 96 840 0 full connection + connectionless protocol 21 91.3 114 336 2 same type slowed down: confirmed 18 -> 18, records 72 -> 72, duration 72 -> 720, alerts 2 -> 0 ``` The numbers belong to the **measurement** class; their inputs are RE10–RE12. #### Zero Confirmed, Twenty-One Ambiguous The most striking row of the first table is the third one. The one-way probe makes 72 attempts and **cannot definitively distinguish a single port**: its coverage is 0%. Instead it produces 21 ambiguous candidates. The reason lies in the type's definition — in this type, no response arriving is the same outcome for both an open and a filtered port; the two cannot be told apart. **3 of the 21 candidates produced are false positives**; the three filtered ports sit in the same list as the open ones. This row is the plainest example of this course's unit of measure. The sentence "the scan found twenty-one open ports" is false for this type: twenty-one **candidates** were found, zero pieces of evidence. Closing that gap requires each candidate to be confirmed with a separate step, and the trace of those steps does not appear in this table at all. At the other end, the full and half connection types definitively distinguish 18 ports, leave nothing ambiguous, and their coverage is **78.3%**. The five ports they miss are on the connectionless protocol: a type that scans the connection-oriented space cannot see a single port in the connectionless space. The limited list finds 15 ports with 32 attempts (65.2%) and misses 8 — the ones it misses are the ports outside the common list, and those are exactly the ones most interesting in an assessment. #### Same Evidence, Different Trace The second table opens up the trace column. Full and half connection produce the same 18 pieces of evidence, but full connection leaves **90 records** and half connection leaves **72 records**. The difference is in the application record: full connection completes the connection at every open port, and the application at that port writes a line. Half connection does not complete the connection; 72 lines remain at the network layer, none at the application layer. This distinction also determines the **value** of the two records from the blue team's side. A network record only says that an attempt was made; an application record says who connected to which port on which endpoint, often together with an identity or a request line. A type chosen on the scanning side for "leaving less trace" means, on the defense side, **trace that carries less information**. The third table gives cost per piece of evidence. The limited list is the cheapest type at **3.13 records** per piece of evidence, but it never sees eight ports at all. The connectionless-protocol scan spends **8.00 records** and **40.00 duration** units per piece of evidence; it is the most expensive in the records column, ties the slow half connection in the duration column, and gives only 3 pieces of evidence in return. In this table, the cheapness ranking and the coverage ranking are **each other's reverse**. #### The Balance Between Speed and Trace The last line of the second table is this lesson's main finding. When the half connection type is slowed down tenfold, **the confirmed count stays at 18, records stay at 72, and alerts drop from 2 to 0**. Duration rises from 72 to 720. In other words, the relationship between speed and trace is not direct. Slowing down **does not reduce the number of records** the defense sees at all; what it reduces is those records crossing a threshold and turning into an alert. The same 72 lines get written to the log in both runs; in one it triggers the rate rule because it fits inside a minute, in the other it does not because it is spread over a window ten times wider. The defense side's takeaway reads directly from this: **a detection based on a speed threshold will never see a slow scan**. The way to see a slow scan is to accumulate attempts from the same source over a wide window and tie the threshold not to speed but to **spread** — how many separate endpoints and how many separate ports were touched. In this run, both types touch 8 endpoints and 72 ports; the spread is the same in both runs. The `alerts/conf` column of the third table writes the same thing from the reverse direction. The slow half connection and the connectionless-protocol scan produce **0.000** alerts; full and half connection produce 0.111, the limited list 0.133. The type producing the most alerts is not the type making the most attempts — the limited list gives the highest rate with 32 attempts, while the connectionless-protocol scan produces no alerts at all with 24 attempts; because an alert depends not on the number of attempts but on the attempt's **density and layer**. #### Twenty-One of Twenty-Three Ports The fourth table shows that a single type is not enough. The highest single-type coverage of the twenty-three open ports is 78.3%; climbing to **91.3%** requires combining two types that scan the connection-oriented and connectionless spaces separately. The remaining two ports are not found by any combination: the two open ports in the connectionless space that do not respond to the probe cannot be tied to a definitive result by any of these types. The combinations' cost differs. Half connection with connectionless protocol spends 96 records, 192 duration, and 2 alerts for 21 pieces of evidence. When the same evidence is obtained with full connection, records rise to 114 and duration to 336; alerts stay at 2. When obtained with the slow half connection, records stay at 96, duration is **840**, and alerts are **0**. The three options give the same 21 pieces of evidence; the difference between them is only in cost and trace. #### Summary - What determines a scan type is the state it distinguishes: for 23 open ports in a 96-port space, the coverages are 78.3%, 78.3%, 0%, 13.0%, and 65.2%. - The one-way probe cannot definitively distinguish a single port; it produces 21 ambiguous candidates, 3 of them false positives — this type produces candidates, not evidence. - The same 18 pieces of evidence leave 90 records with full connection and 72 with half connection; the difference is at the application layer, and an application record carries more information on the defense side. - Cost per piece of evidence ranges from 3.13 to 8.00 records; the cheapest type never sees 8 ports, the most expensive gives only 3 pieces of evidence. - When the same type is slowed down tenfold, evidence stays at 18, records stay at 72, and alerts drop from 2 to 0: slowing down does not erase the trace, it keeps the trace from crossing the threshold. - 21 of the twenty-three open ports (91.3%) can only be reached by combining two types; two ports cannot be tied to a definitive result in any combination. #### Next Step This lesson definitively distinguished that a port is **open**: 21 ports were marked at the evidence level. Yet a port being open does not say what is running there. Each of the twenty-one open ports is, for now, only a number; what software sits behind it, which version it is running, and whether that version is associated with a known vulnerability have not been asked at all. The step that asks this question is fingerprinting, and it brings a new measure: **precision**. The signatures collected from an endpoint point to a version set; the set sometimes has a single member, sometimes contains dozens of versions, and sometimes is the **wrong** set because of a banner the target deliberately leaves misleading. The next lesson counts how many candidates can be narrowed down to a single version, how much extra probing raises precision and how much it grows the trace, and how many false positives incorrect version information carries into the finding list. It is the topic's closing lesson. ### Service and Version Detection Source: https://academia.sh/en/courses/penetration-testing/service-and-version-detection Measuring fingerprinting in terms of precision: at the first probe level, 13 of 21 services narrowing to a single version but 4 of them being assigned the wrong version, the evidence rate rising from 42.9% to 76.2% while the trace climbs from 42 records to 168 records and 6 alerts, the confirmed count dropping from 14 to 11 at the third level while accuracy is completed, and incorrect version information carrying 9 wrong lines into the finding list while missing 10 lines. The previous lesson definitively distinguished that a port is open: 21 open ports were marked at the evidence level. A port being open, though, does not say what is running there. For now, each of the twenty-one ports is only a location; what software sits behind it, which version it is running, and whether that version is associated with a known vulnerability have not been asked at all. The step that asks this question is **fingerprinting** and it brings a new measure: **precision**. The signatures collected from an endpoint point to a version set. The set sometimes has a single member, sometimes contains dozens of versions — and sometimes has a single member but carries the wrong one. The third case is this lesson's main subject, because a wrong answer that looks certain is more expensive than a correct answer that stays ambiguous. #### Precision Is the Size of the Candidate Set **RE13:** every service carries three things — the fictional true version, the banner string's status (correct, misleading, none), and the size of the candidate version set per probe level. **RE14:** a service is **confirmed** after N probes (candidate set has a single member); a service with a misleading banner reaches the true version only at the third level, and is confirmed but assigned the wrong one at earlier levels. **RE15:** the trace comes from the probe level — N requests per service, each request a network and an application record; from the third level on, unusual probes produce alerts. ```js // recon/version.mjs — fictional fingerprinting results for 04's 21 evidence-level open ports // (a model): no real product, no real version, and no real vulnerability identifier; the version // labels are fictional too. // RE13: every service carries three things — the fictional true version, the banner string's // status (correct, misleading, none), and the candidate version set size for probe levels 1..4. // For a service with a misleading banner, "wrongVersion" is the version the string points to. export const SERVICE = ` subscriber/n1 A4 correct - 1 1 1 1 subscriber/n2 B2 correct - 2 1 1 1 subscriber/n3 C7 none - 12 5 2 1 field/n1 A4 correct - 1 1 1 1 field/n5 D1 misleading D0 1 1 3 1 field/n6 E3 correct - 1 1 1 1 field/b1 F2 none - 9 4 2 2 vpn/n6 E3 correct - 1 1 1 1 collector/n6 E3 correct - 1 1 1 1 collector/n7 G5 none - 14 6 3 2 collector/n8 H1 misleading H9 1 1 4 2 collector/b1 F2 none - 9 4 2 2 report/n1 A4 correct - 1 1 1 1 report/n6 E3 correct - 1 1 1 1 shared/n1 A4 correct - 2 1 1 1 shared/n3 C7 none - 12 5 2 1 monitoring/n1 A4 correct - 1 1 1 1 monitoring/n7 I2 misleading I5 1 2 3 1 monitoring/b1 F2 none - 9 4 2 2 log/n1 A4 correct - 1 1 1 1 log/n5 J8 misleading J1 1 1 2 1` .trim().split("\n").map((r) => { const [id, trueVersion, banner, wrongVersion, ...d] = r.trim().split(/\s+/); return { id, trueVersion, banner, wrongVersion, set: d.map(Number) }; }); // Fictional version-to-finding mapping: how many lines a version label carries into the finding list. export const FINDINGS = { A4: 1, B2: 0, C7: 2, D1: 3, E3: 2, F2: 1, G5: 2, H1: 4, I2: 1, J8: 2, D0: 0, H9: 1, I5: 3, J1: 5 }; // RE14: a service is confirmed after N probes (candidate set has a single member). A service with // a misleading banner reaches the true version only at the third level; at earlier levels it is // confirmed but assigned the wrong one. export const confirmed = (s, n) => s.set[n - 1] === 1; export const correct = (s, n) => (s.banner === "misleading" ? n >= 3 : true); export const assigned = (s, n) => (correct(s, n) ? s.trueVersion : s.wrongVersion); // RE15: the trace comes from the probe level — N requests per service, each request a network and // an application record; from the third level on, unusual probes produce alerts. export const trace = (n) => ({ requests: SERVICE.length * n, records: SERVICE.length * n * 2, alerts: n >= 3 ? (n - 2) * 3 : 0 }); ``` ```js // recon/precision.mjs — fingerprinting in terms of precision: how many candidate versions narrow // down to a single version. import { SERVICE, FINDINGS, confirmed, correct, assigned, trace } from "./version.mjs"; const write = (g, ...s) => console.log(s.map((v, i) => (g[i] < 0 ? String(v).padEnd(-g[i]) : String(v).padStart(g[i]))).join("")); const LEVEL = [1, 2, 3, 4]; const K = (n) => SERVICE.filter((s) => confirmed(s, n)); console.log(`${SERVICE.length} services: ${SERVICE.filter((s) => s.banner === "correct").length} ` + `with correct banner, ${SERVICE.filter((s) => s.banner === "misleading").length} misleading, ` + `${SERVICE.filter((s) => s.banner === "none").length} bannerless`); const A = [-8, 8, 8, 10, 9, 8, 11]; console.log("\n1. precision per probe level"); write(A, "level", "requests", "conf", "ambiguous", "falseP", "evid", "prec %"); for (const n of LEVEL) { const k = K(n), fp = k.filter((s) => !correct(s, n)).length; write(A, n, trace(n).requests, k.length, SERVICE.length - k.length, fp, k.length - fp, (((k.length - fp) / SERVICE.length) * 100).toFixed(1)); } const B = [-8, 8, 8, 12, 12, 15]; console.log("\n2. the trace cost of precision"); write(B, "level", "records", "alerts", "rec/evid", "alert/evid", "avg set size"); for (const n of LEVEL) { const k = K(n), evid = k.length - k.filter((s) => !correct(s, n)).length; const avg = SERVICE.reduce((t, s) => t + s.set[n - 1], 0) / SERVICE.length; write(B, n, trace(n).records, trace(n).alerts, (trace(n).records / evid).toFixed(2), (trace(n).alerts / evid).toFixed(3), avg.toFixed(2)); } const C = [-8, 10, 8, 8, 8, 10]; console.log("\n3. lines carried into the finding list"); write(C, "level", "reported", "correct", "wrong", "missed", "wrong %"); for (const n of LEVEL) { const k = K(n); const rep = k.reduce((t, s) => t + FINDINGS[assigned(s, n)], 0); const cor = k.filter((s) => correct(s, n)).reduce((t, s) => t + FINDINGS[s.trueVersion], 0); const tru = k.reduce((t, s) => t + FINDINGS[s.trueVersion], 0); write(C, n, rep, cor, rep - cor, tru - cor, ((rep - cor) / rep * 100).toFixed(1)); } const D = [-10, 7, 7, 7, 7, 7]; console.log("\n4. confirmed count by banner status"); write(D, "banner", "count", "level1", "level2", "level3", "level4"); for (const t of ["correct", "misleading", "none"]) { const l = SERVICE.filter((s) => s.banner === t); write(D, t, l.length, ...LEVEL.map((n) => l.filter((s) => confirmed(s, n)).length)); } const b = SERVICE.filter((s) => s.banner === "misleading"); console.log(`misleading banner: at level 1, ${b.filter((s) => confirmed(s, 1)).length} services ` + `are confirmed but assigned the wrong version; their true finding count is ` + `${b.reduce((t, s) => t + FINDINGS[s.trueVersion], 0)}, reported is ` + `${b.reduce((t, s) => t + FINDINGS[s.wrongVersion], 0)}`); ``` ``` 21 services: 11 with correct banner, 4 misleading, 6 bannerless 1. precision per probe level level requests conf ambiguous falseP evid prec % 1 21 13 8 4 9 42.9 2 42 14 7 3 11 52.4 3 63 11 10 0 11 52.4 4 84 16 5 0 16 76.2 2. the trace cost of precision level records alerts rec/evid alert/evid avg set size 1 42 0 4.67 0.000 3.90 2 84 0 7.64 0.000 2.10 3 126 3 11.45 0.273 1.71 4 168 6 10.50 0.375 1.24 3. lines carried into the finding list level reported correct wrong missed wrong % 1 22 13 9 10 40.9 2 20 14 6 9 30.0 3 14 14 0 0 0.0 4 24 24 0 0 0.0 4. confirmed count by banner status banner count level1 level2 level3 level4 correct 11 9 11 11 11 misleading 4 4 3 0 3 none 6 0 0 0 2 misleading banner: at level 1, 4 services are confirmed but assigned the wrong version; their true finding count is 10, reported is 9 ``` The numbers belong to the **measurement** class; their inputs are RE13–RE15. #### Thirteen Confirmed, Nine Pieces of Evidence The first row of the first table sets up this lesson's question all on its own. At the first probe level, 13 of 21 services narrow to a single version — by the precision measure, 61.9% are confirmed. Yet 4 of those thirteen are assigned the wrong version. The evidence count is not 13, it is **9**, and the real precision is 42.9%. The gap is not a measurement error; it has to do with the measure itself. A candidate set having a single member only shows that ambiguity has been eliminated; it does not show that the member is correct. A step that assigns a version by reading a banner string reads a piece of text the target wrote and counts that text as evidence. Nothing checks whether the text is true. The fourth table breaks this down by banner status. Of the 11 services with a correct banner, 9 are confirmed at the first level, two remain stuck between two versions and get resolved at the second level. Of the 4 services with a misleading banner, **all four are confirmed at the first level** — at a higher rate than the correctly-bannered services, in fact. Of the 6 bannerless services, none is confirmed in the first three levels. This is this lesson's most counterintuitive finding: **a misleading banner produces the highest apparent precision.** A claim of "I am this version" from an endpoint narrows the candidate set to one, regardless of whether the claim is true or false. The side that sees ambiguity is the bannerless endpoint; the side that gets a wrong answer is the misleading one. #### The Level Where the Confirmed Count Drops The third row of the first table looks, at first glance, like a regression: the confirmed count drops from 14 at the second level to **11**. In the same row, false positives fall from 3 to 0, and the evidence count stays at 11. Here is what is happening. The behavioral probes at the third level produce signatures that contradict the banner string; three misleading services now point to three or four candidates instead of a single version. The candidate set **grows**, because the measurement accounts for the first time for the banner possibly being a lie. Apparent precision falls, real precision holds steady, and wrong assignments drop to zero. In an assessment, this row reads as follows: **a drop in the confirmed count is not a loss, it is the previous precision turning out to have been fake.** Adding a probe does not have to increase precision; what it increases is accuracy, and when accuracy increases, some items rightly turn back into ambiguous. At the fourth level, the confirmed count rises to 16 and all of them are correct — precision is **76.2%**. The remaining five services still sit at two-candidate sets even after four probes; four of them are bannerless endpoints. Fingerprinting's ceiling in this run is 16 of 21. #### The Wrong Lines Carried Into the Finding List The third table measures the cost of incorrect version information in the report itself. Every version label carries a certain number of lines into the finding list; a wrong version carries wrong lines. At the first level, the finding list has 22 lines. 13 of them are correct, **9 are wrong** — 40.9% of the list. At the same level, 10 real lines never show up at all. So on one side there are nine items that waste the remediation budget, on the other side ten items that never get looked at. The closing line gives this via the four misleading services: the true finding count totals 10 while the reported count is 9; the two numbers are close to each other and **share not a single line**. At the second level, the wrong ratio falls to 30%. At the third level, the finding list drops to 14 lines and all of them are correct; missed lines are zero, because all eleven services counted as confirmed are correctly assigned. At the fourth level, the list rises to 24 lines and, again, all of them are correct. The comparison between the first and fourth levels is this lesson's practical takeaway: both produce a list of roughly the same size (22 versus 24), but in one, 40.9% of the lines are wrong, in the other, 0%. **The finding count does not tell the two apart; the only thing that does is verification.** This table has a counterpart on the defense side, and it shows the asymmetry clearly. In the organization's own software inventory, the version of the same 21 services can be read in a single step and with zero trace; installation records say which version is installed, which patch is applied, and which component is not used at all, without looking at a single misleading banner string. The measurement done from outside spends 84 requests and 168 records to reach 16 services, while the same measurement done from inside covers all 21 services. The assessment's finding at this step is an expensive and incomplete copy of information the organization **could read from its own inventory**; its real value shows up in the places where that inventory **does not match** what is visible from outside. #### The Trace Cost of Precision The second table opens up, for the last time in this topic, the column that has been carried throughout. The average candidate set size falls from 3.90 to 1.24; in return, the records accumulating at the target rise from 42 to 168, and alerts from 0 to 6. The cost is not distributed evenly. The first two levels leave 84 records and produce **no alerts at all**: these probes are indistinguishable from ordinary requests. The unusual probes used from the third level onward produce a protocol-level anomaly and turn into an alert. In other words, the step that completes accuracy is, at the same time, **the first step the defense notices**. The `rec/evid` column reduces this to a single number: 4.67, 7.64, 11.45, and 10.50. The most expensive level is the third; the fourth level leaves more records but brings five new pieces of evidence, so its cost per piece of evidence falls. Read from the blue team's side, it comes down to this: **most of this topic's steps are seen but not noticed.** Across the total of five lessons, the records accumulating at the target are measured in the hundreds; the items that turn into an alert do not exceed the fingers of two hands, and every one of them comes from either density or a protocol anomaly — none of them is a record in the shape of "reconnaissance is under way." #### Summary - Precision is the size of the candidate version set: at the first probe level, 13 of 21 services narrow to a single version but 4 are assigned the wrong version; evidence is 9, real precision is 42.9%. - A misleading banner produces the highest apparent precision: all four services with a misleading banner are confirmed at the first level, none of the six bannerless services are confirmed in the first three levels. - At the third level, the confirmed count drops from 14 to 11 while wrong assignments fall from 3 to 0: a drop in precision is the previous precision turning out to have been fake. - At the first level, wrong version information carries 9 wrong lines into the finding list (40.9%) and misses 10 real lines; the two sets share not a single line. - At the fourth level, all 24 lines are correct; it is a list of similar size to the 22 lines at the first level, and the only difference between them is verification. - The cost of raising precision from 42.9% to 76.2% is rising from 42 records to 168 and from 0 alerts to 6; the first two levels produce no alerts at all, and the third level, which completes accuracy, produces the first alert. #### Next Step At the end of five lessons, the externally visible inventory has been drawn up, and candidates were counted at every step: 20 candidates from public items, 8 verified ownerships from registration lookup, 21 records and 6 dangling items from the name space, 21 open ports from scanning, 16 confirmed versions from fingerprinting. The trace each number leaves behind was written down too. Yet all of these measurements share a common limit. No step in this topic can say whether a candidate is **actually exploitable**. A version label being associated with a known vulnerability does not show that the installation at that endpoint is actually affected: a patch may have been applied, the affected component may not be in use at all, or a control in front of it may never let the relevant request through. The fourth level's 24 lines, even with full precision, are still, each one, a **candidate**. The only way to close this gap is to look at the traffic coming from and going to the endpoint: which packets flow, how the response changes when a request is modified, what a file does when it is run, what independent sources say about the same item. The next topic — **Analysis Tools** — takes up each of these tools with the same two questions: what does it produce, and **what cannot it prove**. ## Analysis Tools Measuring the gap between a candidate and an evidenced finding produced by traffic, request, and file-analysis tools: which field in packet capture is direct evidence, which is inference, and which is a spoofable claim; how protocol analyzers leave only metadata in an encrypted flow; how many requests and which risk class intercepting-proxy validation requires; the suppressed behavior in sandbox analysis; the false-positive rate of the majority decision in multi-engine reputation services; what exploitation frameworks can and cannot prove; and the skill that legal practice labs do not cover. ### Packet Capture and Analysis Source: https://academia.sh/en/courses/penetration-testing/packet-capture-and-analysis Measuring the evidentiary value of captured traffic: 10 flows from 24 fictional packets, 18, 44, and 85 claim candidates at three capture points, 50 of the 85 candidates at the core mirror turning out to be evidence and 7 spoofable, low-value claims being proven at a rate of 20/20 while high-value claims are proven at only 10/25, and passive capture leaving zero records on the target while requiring one configuration change. The previous topic produced the externally visible inventory and counted candidates; the question left open at its close was this — no reconnaissance step can say whether a candidate is truly open, and answering that requires looking at the traffic and the request itself. This topic covers the tools that look there, and it tests all of them against a single measure: the gap between the **candidate** they produce and the **evidenced finding** that remains. The first tool class is the **packet capturer**: it records the frames passing through the network at one point. How protocols work was already established in the Network Models course in the Computer Networks curriculum, and it is not repeated here. The only question asked here is this: which field in this recorded pile counts as **direct evidence**, which is only an **inference**, and which is a claim the other side can write freely. Even in an authorized assessment, recording traffic belonging to another organization without written authorization and a defined scope is both illegal and unprofessional. #### A Field's Class Depends on Its Producer Reading every field in a packet with the same eye is analysis's most common mistake. **The capture point itself** produces one portion of the fields: the time of the record, which leg the frame entered on, direction, size, count. These are not the sender's claim; they are the measuring instrument's observation, and the sender cannot change them. **Headers** carry another portion: addresses, the port number, the server name field. These have to stay consistent along the network path, so they are not entirely free either, but they are not an observation — they are **inference**. The remaining portion is carried by the **payload**: username, meter ID, client clock. The party writing these is the sender, and unless they are cryptographically bound, the sender can write whatever it wants. There is a fourth class: the **signature**. If a field inside the payload is cryptographically bound, that field stops being the sender's claim and turns into evidence that can be tested — but if the flow is encrypted, the capture point cannot see the signature either. - **AT1.** The packet set, service table, capture points, and claim list are a **model**; no real capture was performed. No real address, hostname, or organization is written. - **AT2.** A flow is the set of packets carrying the same source, destination, and service triple. - **AT3.** A field's class depends on its producer: a point observation and a signature are **evidence**, a header field is **inference**, a payload field is **spoofable**. - **AT4.** In an encrypted flow, payload fields and the signature are not visible; the server name field is counted as visible. - **AT5.** A **claim** is a proposition derived from captured traffic. If at least one of its bases is visible, the claim is a **candidate**; its class is the class of the strongest visible basis. - **AT6.** Every claim is given a **value**: how much weight it carries in the report. The value was assigned by hand. - **AT7.** Capture is passive; it produces no record on the target systems. ```js // pkt/flow.mjs — fictional packet set, flow extraction, and field classification are a MODEL. // There is no real capture; no real address, hostname, or organization is written. // A field's class depends on who produced it. export const FIELD = { 'capture-time': 'point', 'ingress-leg': 'point', 'direction': 'point', 'frame-size': 'point', 'packet-count': 'point', 'plaintext-presence': 'point', 'source-address': 'header', 'destination-address': 'header', 'port-number': 'header', 'server-name-field': 'header', 'signature-verification': 'signature', 'username': 'payload', 'password-field': 'payload', 'meter-id': 'payload', 'reading-value': 'payload', 'subscriber-no': 'payload', 'session-token': 'payload', 'client-clock': 'payload', 'device-id': 'payload', 'transaction-id': 'payload', }; export const CLASS = { point: 'evidence', signature: 'evidence', header: 'inference', payload: 'spoofable' }; export const STRENGTH = { evidence: 3, inference: 2, spoofable: 1 }; // Service: whether it is encrypted, whether the payload is signed, which fields the payload carries. export const SERVICE = { 'meter-collection': { encrypted: false, signed: true, payload: ['meter-id', 'reading-value', 'client-clock'] }, 'validation-call': { encrypted: true, signed: true, payload: ['transaction-id', 'subscriber-no'] }, 'portal-session': { encrypted: true, signed: false, payload: ['session-token', 'username'] }, 'field-sync': { encrypted: false, signed: false, payload: ['username', 'device-id'] }, 'admin-access': { encrypted: false, signed: false, payload: ['username', 'password-field'] }, 'external-notify': { encrypted: true, signed: false, payload: ['subscriber-no'] }, }; // Capture point: which legs' packets it sees. export const POINT = { 'branch-leg': ['branch-1'], 'core-mirror': ['branch-1', 'branch-2', 'core', 'external'], 'collector-endpoint': ['core'], }; // row: time(s) | leg | source | destination | service | size(bytes) export const PACKET = ` 0 branch-1 s1-end collector meter-collection 420 1 branch-1 s1-end collector meter-collection 418 2 branch-1 s1-end collector meter-collection 421 3 branch-2 s2-end collector meter-collection 419 4 branch-2 s2-end collector meter-collection 420 5 core collector validator validation-call 1180 6 core collector validator validation-call 1174 7 core collector validator validation-call 1190 8 external sub-end portal portal-session 2240 9 external sub-end portal portal-session 2180 10 external sub-end portal portal-session 2260 11 branch-1 field-end field-svc field-sync 960 12 branch-1 field-end field-svc field-sync 955 13 branch-2 field-end field-svc field-sync 962 14 core admin collector admin-access 310 15 core admin collector admin-access 305 16 core admin validator admin-access 308 17 external billing ext-end external-notify 1420 18 external billing ext-end external-notify 1415 19 branch-1 s1-end collector meter-collection 417 20 core collector billing validation-call 1160 21 core collector billing validation-call 1168 22 branch-2 s2-end collector meter-collection 422 23 core admin field-svc admin-access 312 `.trim().split('\n').map((r) => { const [t, leg, source, destination, service, size] = r.trim().split(/\s+/); return { t: +t, leg, source, destination, service, size: +size }; }); // Flow: packets carrying the same (source, destination, service) triple. export function flows(packets) { const m = new Map(); for (const p of packets) { const k = `${p.source}>${p.destination}/${p.service}`; if (!m.has(k)) m.set(k, { name: k, service: p.service, leg: new Set(), packets: 0, bytes: 0 }); const a = m.get(k); a.leg.add(p.leg); a.packets += 1; a.bytes += p.size; } return [...m.values()]; } // Fields visible in a flow: point and header fields always; payload fields and the // signature only in an unencrypted flow. export function visibleFields(a) { const h = SERVICE[a.service]; const g = ['capture-time', 'ingress-leg', 'direction', 'frame-size', 'packet-count', 'source-address', 'destination-address', 'port-number']; if (h.encrypted) g.push('server-name-field'); else { g.push('plaintext-presence', ...h.payload.filter((f) => FIELD[f])); if (h.signed) g.push('signature-verification'); } return g; } // Claim: a proposition derived from captured traffic. If at least one of its basis fields is // visible, the claim becomes a CANDIDATE; the claim's class is the class of the strongest visible basis. export const CLAIM = [ { name: 'flow-exists', value: 'low', basis: ['direction', 'frame-size'] }, { name: 'volume-and-timing', value: 'low', basis: ['packet-count', 'capture-time'] }, { name: 'transaction-time', value: 'medium', basis: ['client-clock', 'capture-time'] }, { name: 'traffic-from-this-leg', value: 'medium', basis: ['ingress-leg'] }, { name: 'service-spoken', value: 'medium', basis: ['port-number'] }, { name: 'counterparty-identity', value: 'medium', basis: ['destination-address', 'server-name-field'] }, { name: 'fields-in-plaintext', value: 'high', basis: ['plaintext-presence'] }, { name: 'sender-identity', value: 'high', basis: ['source-address', 'signature-verification'] }, { name: 'meter-reading-value', value: 'high', basis: ['reading-value', 'meter-id', 'signature-verification'] }, { name: 'credential-field-contents', value: 'high', basis: ['password-field', 'session-token'] }, { name: 'person-performing-action', value: 'high', basis: ['username', 'device-id'] }, ]; export function claimClass(claim, visible) { const v = claim.basis.filter((f) => visible.includes(f)); if (v.length === 0) return null; return v.map((f) => CLASS[FIELD[f]]).sort((x, y) => STRENGTH[y] - STRENGTH[x])[0]; } export function pointMeasure(point) { const legs = POINT[point]; const p = PACKET.filter((x) => legs.includes(x.leg)); const s = { packets: p.length, bytes: p.reduce((t, x) => t + x.size, 0), flows: 0, candidates: 0, evidence: 0, inference: 0, spoofable: 0 }; for (const a of flows(p)) { s.flows += 1; const g = visibleFields(a); for (const i of CLAIM) { const c = claimClass(i, g); if (c) { s.candidates += 1; s[c] += 1; } } } return s; } ``` #### Three Capture Points ```js // pkt/measure.mjs — the same packet set is measured from three capture points. import { PACKET, POINT, CLAIM, SERVICE, flows, visibleFields, claimClass, pointMeasure } from './flow.mjs'; const y = (g, ...s) => console.log(s.map((v, i) => (g[i] < 0 ? String(v).padEnd(-g[i]) : String(v).padStart(g[i]))).join('').trimEnd()); console.log(`${PACKET.length} packets -> ${flows(PACKET).length} flows, ` + `${Object.keys(SERVICE).length} services, ${CLAIM.length} claim types`); const A = [-22, -18, 9, 7, 11, 8, 8]; console.log('\n1. visible fields per flow'); y(A, 'flow', 'service', 'packets', 'bytes', 'encrypted', 'signed', 'fields'); for (const a of flows(PACKET)) { const h = SERVICE[a.service]; y(A, a.name.split('/')[0], a.service, a.packets, a.bytes, h.encrypted ? 'yes' : 'no', h.signed ? 'yes' : 'no', visibleFields(a).length); } const B = [-20, 9, 7, 12, 10, 11, 11]; console.log('\n2. claims per capture point'); y(B, 'point', 'packets', 'flows', 'candidates', 'evidence', 'inference', 'spoofable'); for (const n of Object.keys(POINT)) { const s = pointMeasure(n); y(B, n, s.packets, s.flows, s.candidates, s.evidence, s.inference, s.spoofable); } const C = [-27, 8, 10, 11, 11]; console.log('\n3. class distribution per claim type at the core mirror'); y(C, 'claim', 'value', 'evidence', 'inference', 'spoofable'); const ag = flows(PACKET).map(visibleFields); const box = { low: [0, 0], medium: [0, 0], high: [0, 0] }; for (const i of CLAIM) { const c = { evidence: 0, inference: 0, spoofable: 0 }; for (const g of ag) { const s = claimClass(i, g); if (s) c[s] += 1; } y(C, i.name, i.value, c.evidence, c.inference, c.spoofable); box[i.value][0] += c.evidence; box[i.value][1] += c.evidence + c.inference + c.spoofable; } console.log('\n4. evidence ratio by claim value'); for (const d of ['low', 'medium', 'high']) console.log(` ${d.padEnd(8)} evidence ${String(box[d][0]).padStart(3)} / ${box[d][1]}`); const t = pointMeasure('core-mirror'); console.log(`\nrecords produced on target systems 0`); console.log(`configuration change needed to enable capture 1 (mirror leg, written approval)`); console.log(`volume examined ${t.bytes} bytes`); console.log(`flows not visible from the branch leg ${t.flows - pointMeasure('branch-leg').flows} / ${t.flows}`); ``` ``` 24 packets -> 10 flows, 6 services, 11 claim types 1. visible fields per flow flow service packets bytes encrypted signed fields s1-end>collector meter-collection 4 1676 no yes 13 s2-end>collector meter-collection 3 1261 no yes 13 collector>validator validation-call 3 3544 yes yes 9 sub-end>portal portal-session 3 6680 yes no 9 field-end>field-svc field-sync 3 2877 no no 11 admin>collector admin-access 2 615 no no 11 admin>validator admin-access 1 308 no no 11 billing>ext-end external-notify 2 2835 yes no 9 collector>billing validation-call 2 2328 yes yes 9 admin>field-svc admin-access 1 312 no no 11 2. claims per capture point point packets flows candidates evidence inference spoofable branch-leg 6 2 18 12 5 1 core-mirror 24 10 85 50 28 7 collector-endpoint 9 5 44 23 15 6 3. class distribution per claim type at the core mirror claim value evidence inference spoofable flow-exists low 10 0 0 volume-and-timing low 10 0 0 transaction-time medium 10 0 0 traffic-from-this-leg medium 10 0 0 service-spoken medium 0 10 0 counterparty-identity medium 0 10 0 fields-in-plaintext high 6 0 0 sender-identity high 2 8 0 meter-reading-value high 2 0 0 credential-field-contents high 0 0 3 person-performing-action high 0 0 4 4. evidence ratio by claim value low evidence 20 / 20 medium evidence 20 / 40 high evidence 10 / 25 records produced on target systems 0 configuration change needed to enable capture 1 (mirror leg, written approval) volume examined 22436 bytes flows not visible from the branch leg 8 / 10 ``` #### Evidence Is Cheap, a Valuable Claim Is Expensive The fourth table is this lesson's result. **20 of the 20** low-value claims fall in the evidence class. "There is a flow between these two endpoints," "this many packets passed at this time" — all of it is true, none of it is disputable, and none of it is worth a line in the report. For medium-value claims the ratio drops by half: the leg and the time are evidence, while the spoken service and the counterparty's identity are read from the header and are inference. For high-value claims, only **10 of 25** fall into the evidence class. This inverse relationship is not a flaw; it is the structure of the method. The more a capture point observes, the less that observation says; the fields that actually say something are produced by the sender, and the sender is always suspect. On the measurement network this has three concrete counterparts. **Credential field contents** and **person performing the action** are never evidence in any flow; they are produced as candidates seven times, and all seven rest solely on a payload field. Seeing a username in a capture does not show that person performed the action — it only shows that name was written into that field. A report line that reads "user X performed this action" is an unproven candidate, and it spends the remediation budget in the wrong place. **Sender identity** is evidence in two of ten candidates, inference in eight. The two pieces of evidence come from the meter-collection flows, whose payload is signed and unencrypted: the signature is visible and can be verified. In the remaining eight, all that is on hand is the source address. In encrypted flows the signature also stays inside the payload and so is not visible — encryption is the right call, but it also hides from the analyst a field that could have been evidence. **Fields in plaintext** are in the evidence class in six flows, and this is the most solid finding on the fictional network. The three admin-access flows, the field-sync flow, and two meter-collection flows carry their fields unencrypted. The distinction matters here: that the fields **are carried in plaintext** is the capture point's observation and is evidence; **what those fields say** is the sender's claim and is not. #### The Point's Placement and the Trace It Leaves The second table is the measure of where the capturer is placed. The branch leg has 6 packets, 2 flows, and 18 candidates; the core mirror has 24 packets, 10 flows, and 85 candidates. An analyst looking from the branch leg **never sees eight of the ten flows** and can say nothing about a flow it does not see. The collector endpoint sits in the middle: it sees the core's traffic, not the branch legs, but because three of the five flows it sees are unencrypted, it is the point with the highest spoofable-candidate ratio — 6 of 44 candidates. The cost column consists of two items, and both are small. The volume examined is 22,436 bytes; in a real assessment this number is far larger, and it is the actual item that determines analysis time. The trace left on the defense side, though, is **zero records**: capture is passive, no request goes to the target systems, no log line is produced. This is the lowest-trace step we will see in this course. The one item that is not zero is what **makes the capture possible**. Opening a mirror leg is a configuration change; it produces a change record and requires the network team's approval and an explicit line in the scope document. Because subscriber data passes through the traffic, where the record will be stored and when it will be deleted are also written into the same document. It is the capture itself that is passive; the permission to access it is not. #### Summary - Twenty-four fictional packets produce 10 flows; the number of visible fields per flow is 9 in encrypted flows and between 11 and 13 in unencrypted flows. - A field's evidentiary value depends on its producer: the capture point's observation and a cryptographic signature are evidence, a header field is inference, a payload field is a spoofable claim. - At the core mirror, 50 of 85 claim candidates are evidence, 28 are inference, and 7 are spoofable; a claim's class is the class of its strongest visible basis. - The evidence ratio runs inverse to the claim's value: 20/20 of low-value claims are proven, but only 10/25 of high-value ones. None of the seven candidates that tie a user to an action are evidence. - The capture point's placement determines which flow is seen: the branch leg never sees 8 of the 10 flows. - Passive capture leaves zero records on the target; the only trace is the configuration change the mirror leg requires and the written approval tied to it. #### Next Step In this lesson, fields were classified one by one, but encrypted flows were passed over in a single line: the payload is not visible, the server name field is. Four of the ten flows on the fictional network are encrypted, and those four are exactly the flows carrying the most weight — the subscriber portal session, the validation call, the external notification. It is not true that analysis comes up empty-handed on these flows; its hands are **full in a different way**. The next lesson measures application-layer inspection in terms of visibility. The same set of flows is run through a protocol analyzer, and two columns are counted separately: how many fields can be read in an unencrypted flow, and how many fields remain only **metadata** in an encrypted flow. It measures how much of a claim metadata carries, which scope and privacy item decrypting requires, and when the analyzer's **own interpretation** amounts to an inference. ### Protocol Analyzers Source: https://academia.sh/en/courses/penetration-testing/protocol-analyzers Measuring application-layer inspection in terms of visibility: only 16 of the 30 application fields defined across ten flows can be read, two flows are mislabeled by an analyzer that looks at the transport number, 43 of 91 claim candidates are evidence and 48 are inference drawn from metadata, three of five encrypted flows have in-org access to key material, and passive analysis leaves zero records on the target. The previous lesson classified fields by their producer and passed over encrypted flows in a single line: the payload is invisible, the server-name field is visible. The busiest flows on the fictional measurement network are exactly the encrypted ones. It is not true that analysis comes up empty-handed on these flows; its hands are full in a different way, and this lesson's subject is how much that "different way" is worth. **A protocol analyzer** splits captured bytes into application-layer fields. It does two things: it **assumes a format** and names the fields according to that format. Neither produces evidence; the first is a guess, the second reads a field if the guess is correct. What is measured is how many flows these two steps hold up on. #### Visibility Is Two Columns A flow has two separate sets of fields. **Application fields** are readable only if encryption is off and the format is known correctly. **Metadata fields** are produced under every condition: endpoint addresses, transport number, direction sequence, packet-size sequence, timing interval, flow duration. Encrypted sessions add handshake parameters and the server-name field to this — encryption does not close off these two fields. Metadata answers its own questions at evidence level: how long the flow lasted, what its volume was, whether it was evenly spaced. These are the capture point's own observations. The problem starts when metadata is used **as if it answered other questions**. Looking at the size sequence and saying "the transaction succeeded," looking at the direction sequence and saying "this was a write operation," is a **surrogate inference**: the field's answer is guessed without reading the field. These inferences are often correct, and they are not evidence. - **AT8.** The format catalog, flow list, and claim list are a **model**; there is no real capture. - **AT9.** The analyzer assumes the format **from the transport number**. If the assumption is wrong, it cannot read the fields and mislabels the flow. Real analyzers also look at content signatures; this model looks only at the number and gives the mislabel count as an upper bound. - **AT10.** When a text-based format is mislabeled, a raw text fragment is counted as visible; in a binary format nothing is visible. - **AT11.** In an encrypted flow, no application field is read; metadata fields are produced in full. - **AT12.** A claim is **evidence** if its basis comes from a read application field; meta-type claims are also evidence when satisfied from a metadata field. A field-type claim satisfied from a surrogate field is **inference**. - **AT13.** Access to key material is written as `end-agent` if it is provided by an agent on the organization's own end, and `none` if it cannot be provided. - **AT14.** Passive analysis produces no record on the target; resolving by intercepting is an active step. ```js // prot/resolve.mjs — this is a MODEL of protocol-analyzer visibility. There is no real capture, // real address, or real organization. An analyzer ASSUMES a format from transport information; // if the assumption is wrong, it cannot read the fields and mislabels the flow. export const FORMAT = { 'text-request': { text: true, field: ['request-line', 'header-fields', 'body', 'status-code', 'session-id', 'user-field'] }, 'measurement-protocol': { text: false, field: ['function-code', 'record-address', 'value-field', 'unit-id'] }, 'file-transfer': { text: true, field: ['command-name', 'file-name', 'user-field', 'size-field'] }, 'name-resolution': { text: true, field: ['query-name', 'record-type', 'response-code'] }, 'binary-sync': { text: false, field: ['operation-code', 'record-count', 'timestamp'] }, 'encrypted-session': { text: false, field: [] }, }; // The format the analyzer assumes from the transport number. export const ASSUMPTION = { 80: 'text-request', 443: 'encrypted-session', 502: 'measurement-protocol', 21: 'file-transfer', 53: 'name-resolution', 8080: 'text-request' }; // Fields that remain even if encryption is stripped: the analyzer produces these in every flow. export const META = ['endpoint-addresses', 'transport-number', 'direction-sequence', 'packet-size-sequence', 'timing-interval', 'flow-duration']; export const ENCRYPTED_EXTRA = ['server-name-field', 'handshake-parameters']; // row: flow | transport number | real format | encrypted | access to key material export const FLOW = ` s1-end>collector 502 measurement-protocol yes end-agent s2-end>collector 502 measurement-protocol yes end-agent collector>validator 443 encrypted-session yes end-agent sub-end>portal 443 encrypted-session yes none field-end>field-svc 8080 binary-sync no none admin>collector 80 text-request no none admin>validator 80 text-request no none billing>ext-end 443 encrypted-session yes none collector>billing 502 binary-sync no none field-end>file-svc 21 file-transfer no none `.trim().split('\n').map((r) => { const [name, port, format, encrypted, key] = r.trim().split(/\s+/); return { name, port: +port, format, encrypted: encrypted === 'yes', key }; }); // Resolution result: read application field, produced metadata field, mislabel. export function resolve(a) { const assumed = ASSUMPTION[a.port]; const meta = [...META, ...(a.encrypted ? ENCRYPTED_EXTRA : [])]; if (a.encrypted) return { read: [], meta, label: assumed, mislabeled: false, depth: 'meta-only' }; if (assumed === a.format) return { read: FORMAT[a.format].field, meta, label: assumed, mislabeled: false, depth: 'full' }; const partial = FORMAT[a.format].text ? ['raw-text-fragment'] : []; return { read: partial, meta, label: assumed, mislabeled: true, depth: partial.length ? 'partial' : 'none' }; } // Claim: a proposition drawn from the analyzer's output. // 'meta'-type claims are satisfied directly from a metadata field and count as evidence (a point // observation). 'field'-type claims are proven only from an application field; surrogate is a // guess made from a metadata field and counts as INFERENCE. export const CLAIM = [ { name: 'flow-duration-and-volume', type: 'meta', basis: ['flow-duration'], surrogate: [] }, { name: 'regular-interval-present', type: 'meta', basis: ['timing-interval'], surrogate: [] }, { name: 'counterparty-name', type: 'meta', basis: ['server-name-field'], surrogate: [] }, { name: 'session-parameters', type: 'meta', basis: ['handshake-parameters'], surrogate: [] }, { name: 'format-spoken', type: 'field', basis: ['request-line', 'function-code', 'command-name', 'operation-code', 'query-name'], surrogate: ['transport-number'] }, { name: 'requested-resource', type: 'field', basis: ['request-line', 'query-name', 'file-name'], surrogate: ['server-name-field'] }, { name: 'transaction-result', type: 'field', basis: ['status-code', 'response-code'], surrogate: ['packet-size-sequence'] }, { name: 'write-or-read', type: 'field', basis: ['function-code', 'operation-code', 'command-name'], surrogate: ['direction-sequence'] }, { name: 'transferred-file', type: 'field', basis: ['file-name', 'size-field'], surrogate: ['packet-size-sequence'] }, { name: 'record-count', type: 'field', basis: ['record-count', 'record-address'], surrogate: ['packet-size-sequence'] }, { name: 'carried-identity-field', type: 'field', basis: ['user-field', 'session-id'], surrogate: [] }, { name: 'written-measurement-value', type: 'field', basis: ['value-field'], surrogate: [] }, ]; // A claim's status for a given flow: evidence | inference | unmet export function status(claim, res) { if (claim.basis.some((g) => res.read.includes(g))) return 'evidence'; if (claim.type === 'meta' && claim.basis.some((g) => res.meta.includes(g))) return 'evidence'; if (claim.surrogate.some((g) => res.meta.includes(g))) return 'inference'; return 'unmet'; } export function measure() { const s = { defined: 0, read: 0, meta: 0, mislabeled: 0, candidate: 0, evidence: 0, inference: 0 }; for (const a of FLOW) { const c = resolve(a); s.defined += FORMAT[a.format].field.length; s.read += c.read.length; s.meta += c.meta.length; if (c.mislabeled) s.mislabeled += 1; for (const claim of CLAIM) { const d = status(claim, c); if (d !== 'unmet') { s.candidate += 1; s[d] += 1; } } } return s; } ``` #### Ten Flows, Two Columns ```js // prot/visibility.mjs — ten flows are run through the analyzer; field and meta are counted separately. import { FLOW, FORMAT, CLAIM, resolve, status, measure } from './resolve.mjs'; const y = (g, ...s) => console.log(s.map((v, i) => (g[i] < 0 ? String(v).padEnd(-g[i]) : String(v).padStart(g[i]))).join('').trimEnd()); const A = [-21, -22, 11, 10, 8, 6, 22, 11]; console.log('1. resolution depth per flow'); y(A, 'flow', 'real format', 'transport', 'encrypted', 'defined', 'read', 'label', 'depth'); for (const a of FLOW) { const c = resolve(a); y(A, a.name, a.format, a.port, a.encrypted ? 'yes' : 'no', FORMAT[a.format].field.length, c.read.length, c.label, c.depth); } const B = [-27, 6, 10, 11, 7]; console.log('\n2. satisfaction per claim type (10 flows)'); y(B, 'claim', 'type', 'evidence', 'inference', 'unmet'); for (const claim of CLAIM) { const d = FLOW.map((a) => status(claim, resolve(a))); y(B, claim.name, claim.type, d.filter((x) => x === 'evidence').length, d.filter((x) => x === 'inference').length, d.filter((x) => x === 'unmet').length); } const s = measure(); console.log('\n3. totals'); console.log(` defined application fields ${s.defined}`); console.log(` read application fields ${s.read}`); console.log(` produced metadata fields ${s.meta}`); console.log(` mislabeled flows ${s.mislabeled} / ${FLOW.length}`); console.log(` produced claim candidates ${s.candidate} / ${CLAIM.length * FLOW.length}`); console.log(` evidence ${s.evidence}`); console.log(` inference ${s.inference}`); console.log(` flows with a wrong format guess ${s.mislabeled}`); const encrypted = FLOW.filter((a) => a.encrypted); console.log('\n4. options for seeing inside an encrypted flow'); console.log(` encrypted flows ${encrypted.length} / ${FLOW.length}`); console.log(` key access via an end-agent ${encrypted.filter((a) => a.key !== 'none').length}`); console.log(` unresolvable without interception ${encrypted.filter((a) => a.key === 'none').length}`); console.log(` trace passive analysis leaves on target 0 records`); console.log(` trace interception leaves on target 1 alert per flow (active step)`); ``` ``` 1. resolution depth per flow flow real format transport encrypted defined read label depth s1-end>collector measurement-protocol 502 yes 4 0 measurement-protocol meta-only s2-end>collector measurement-protocol 502 yes 4 0 measurement-protocol meta-only collector>validator encrypted-session 443 yes 0 0 encrypted-session meta-only sub-end>portal encrypted-session 443 yes 0 0 encrypted-session meta-only field-end>field-svc binary-sync 8080 no 3 0 text-request none admin>collector text-request 80 no 6 6 text-request full admin>validator text-request 80 no 6 6 text-request full billing>ext-end encrypted-session 443 yes 0 0 encrypted-session meta-only collector>billing binary-sync 502 no 3 0 measurement-protocol none field-end>file-svc file-transfer 21 no 4 4 file-transfer full 2. satisfaction per claim type (10 flows) claim type evidence inference unmet flow-duration-and-volume meta 10 0 0 regular-interval-present meta 10 0 0 counterparty-name meta 5 0 5 session-parameters meta 5 0 5 format-spoken field 3 7 0 requested-resource field 3 5 2 transaction-result field 2 8 0 write-or-read field 1 9 0 transferred-file field 1 9 0 record-count field 0 10 0 carried-identity-field field 3 0 7 written-measurement-value field 0 0 10 3. totals defined application fields 30 read application fields 16 produced metadata fields 70 mislabeled flows 2 / 10 produced claim candidates 91 / 120 evidence 43 inference 48 flows with a wrong format guess 2 4. options for seeing inside an encrypted flow encrypted flows 5 / 10 key access via an end-agent 3 unresolvable without interception 2 trace passive analysis leaves on target 0 records trace interception leaves on target 1 alert per flow (active step) ``` #### The Analyzer's Own Interpretation The last two columns of the first table are this lesson's quietest finding. Of the 30 defined application fields, only 16 are read. The loss comes from two sources: five flows are encrypted, and two flows are **mislabeled**. The mislabel is not a flaw in the tool; it is the result of an assumption. Field synchronization speaks a binary format but listens near the customary text-request number; the analyzer assumes a text request and cannot read any of the three defined fields. The flow from collector to billing runs a binary sync over the measurement number; the analyzer stamps a measurement-protocol label on it. In both cases the output is not empty — the output is **wrong**. If the line "this flow speaks the measurement protocol" goes into the report, that line is a false positive, and its source is the analyzer itself, not the system on the target. In the second table, the `format-spoken` row gives this as a number: in three flows the format is confirmed from an actually-read field, in seven it is inferred only from the transport number, and **two of those seven are wrong**. The way to turn an inference into evidence is to actually read a field that gives the format away. These two flows have a second reading, and it is written to the defense side. A service speaking outside its customary number is easy to miss in an inventory, and it misleads every control that classifies by looking at the number in the same way. What misleads the assessment's analyzer also misleads the defense's classifier; the finding is not written as "the analyzer was wrong," but as "these two flows do not fit any number-based classification." #### What Metadata Says and Does not Say Ninety-one candidates were produced; 43 are evidence, 48 are inference. The inference column is nearly half, and its content shows a clear pattern. `record-count` is inference in ten of ten flows, evidence in zero: looking at the packet-size sequence and saying "this many records were transferred" is possible in every flow and evidence in none. `transaction-result` is inference in eight flows — reading success from the response size is an old habit, and it looks like the only option in encrypted flows. By contrast, two rows produce no inference at all. `carried-identity-field` is evidence in three flows and unmet in seven; `written-measurement-value` is unmet in ten of ten flows. These two rows are where encryption really holds: the content of the identity field and the written measurement value cannot be derived from metadata — they have no surrogates. Writing "unmet" in the assessment report is not a shortcoming, it is the correct answer; writing an inference as evidence is the shortcoming. The four meta-type claims produce 30 pieces of evidence in total, and 10 of these come only from encrypted flows — the server-name field and handshake parameters. While encryption closes off the application fields, it leaves these two fields open and provides a surrogate for the `requested-resource` claim in five flows. Encrypted traffic does not close off the whole question; it keeps saying who was talked to and for how long. #### The Cost of Seeing Inside the Encryption In three of the five encrypted flows, access to key material sits on the organization's own end: the measurement ends and the collector are devices the organization manages, and an agent on the end makes analysis possible within scope. The remaining two are the subscriber-portal session and the external-notification flow. The first has a subscriber on the other end, the second has another organization. The only way to see inside these two flows is to intercept; intercepting means stopping being passive, producing records and alerts on the target, and touching a third party's traffic. If written authorization and defined scope do not cover this, the step is both illegal and unprofessional, and the two flows are **left unresolved**. The trace left on the defense side also splits in two here. Passive analysis produces zero records, as in the previous lesson. Resolving by intercepting produces at least one alert per flow — the established trust chain differing from what is expected is the first thing the control on the end will see. This is the first non-zero trace column in the course, and the reason is not that the tools changed, but that the step has become **active**. #### Summary - Of the 30 application fields defined across ten flows, 16 are read; five flows' worth of the loss comes from encryption, two flows' worth from the format assumption not holding. - The analyzer assumes the format from transport information; two flows are mislabeled and the output is not empty but wrong. This is a source of false positives that comes from the tool, not the target. - Of 91 candidates, 43 are evidence and 48 are surrogate inference drawn from metadata; `record-count` is inference in ten of ten flows and evidence in zero. - The content of the identity field and the written measurement value can be derived from metadata in no flow; these two rows are where encryption really holds. - Three of the five encrypted flows have in-org access to the key; two are left unresolved because they involve a subscriber and a third party. - Passive analysis leaves zero records; resolving by intercepting produces at least one alert per flow and moves the step into the active class. #### Next Step So far, both tools were only **watching** tools: they recorded the traffic that passed and read the fields they could read. Both share the same limit: seeing that a field passed with a particular value in a flow does not say what the system will do when **a different value** is written to that field. Showing that a candidate is really open requires changing the request and seeing the response. The next lesson treats the intercepting proxy as a validation step and asks a single question: how many requests does it take to show that a candidate is open. Requests are split into three risk classes — observation-only, state-changing on the target, and carrying a risk of service disruption — and each candidate's validation plan is counted in terms of these classes. Scope checking also comes into play for the first time: some requests cannot be sent at all because they touch a party outside scope, and those candidates remain unturned into evidence. ### Request Manipulation Source: https://academia.sh/en/courses/penetration-testing/request-manipulation Measuring intercepting-proxy request manipulation as a validation step: 18 candidates' 40 planned requests, 6 candidates decided with observation-only requests producing zero alerts, 14 candidates decided under the broadest rule set producing 16 alerts, the number of requests to an out-of-scope target staying at zero across all three rule sets, and two candidates that are actually open being unprovable under any rule set. The previous two tools only watched. Both shared the same limit: seeing that a field passed with a particular value does not say what the system will do when a different value comes into that field. The only way to show that a candidate is really open is to change the request and see the response — and this is the first step in the course that **touches** the target. An **intercepting proxy** sits between client and server, halts the request, changes it, and resends it. This lesson's subject is not the tool's use but the **accounting** of the work done with that tool: how many requests it takes to prove a candidate, how many of these requests change something persistent on the target, how many carry a risk of disrupting service, and which ones spill outside scope. No request content, string, or payload is written; requests are counted only by their class. #### Two Separate Classifications A validation request is classified along two independent axes, and if the two are mixed up, scope checking cannot be established. The first is the **risk class**. `o` is an observation-only request: it does not change the target's state, and sending the same request again gives the same result. `s` is a request that changes state on the target: it writes a record, opens a session, advances a counter; being reversible is a separate condition. `d` is a request carrying a risk of disrupting service; it becomes unavoidable when validating candidates related to rate limiting and resource consumption. The second is the **scope class**. An `in-scope` target is one named in the rules of engagement and permitted to be modified. A `shared` target is a place where other tenants also exist in the same environment: observing is allowed, changing state is not, because it carries the risk of touching someone else's data. An `out-of-scope` target belongs to a third party, and **no request can be sent**. Even sending a single observation request to an out-of-scope target voids the authorization granted; at that point the work being done becomes unauthorized access and carries legal consequences. - **AT15.** The candidate list, plan lengths, scope classes, and the `actual` field are a **model**. `actual` is the candidate's status as it will turn out at the end of validation; it is unknown before validation and is written only so the model can compute the outcome. - **AT16.** The validation plan is the **shortest** request sequence needed to show that a candidate is open. Plan lengths were written by hand. - **AT17.** A plan produces a result only if **all** of its steps can be run. A plan that runs incompletely does not say "not open"; it says `unverifiable`. - **AT18.** Every request produces a record on the target. The alert count depends on the risk class: `o` zero, `s` one, `d` three. Times were written by hand per request. - **AT19.** Scope checking is programmatic: the permitted request class is the intersection of the rule set and the scope class. If the intersection is empty, the request is not sent. - **AT20.** The three rule sets represent three separate strictness levels of the rules of engagement. - **AT21.** A false positive is a candidate whose plan runs in full and is seen not to be open; this candidate has no place in the report. ```js // vek/validate.mjs — this is a model of the STEP COUNT and RISK CLASS of intercepting-proxy // validation. No request content, string, or payload is written; requests are counted only by // class. o = observation-only (no state change), s = state-changing on the target, // d = carrying a risk of service disruption. export const RISK = { o: { record: 1, alert: 0, min: 2 }, s: { record: 1, alert: 1, min: 5 }, d: { record: 1, alert: 3, min: 15 }, }; // Scope class: which request classes can be sent to each target. export const SCOPE = { 'in-scope': ['o', 's', 'd'], 'shared': ['o'], // other tenants exist in the same environment 'out-of-scope': [], // third party; no request can be sent }; // Rules of engagement: which risk class is permitted. export const RULE = { 'observation-only': ['o'], 'reversible': ['o', 's'], 'written-extra-permission': ['o', 's', 'd'], }; // row: candidate | candidate class | target scope | actually open | validation plan export const CANDIDATE = ` c01 information-disclosure in-scope yes o,o c02 authorization-check in-scope yes o,o,s c03 session-management in-scope no o,o,s c04 configuration in-scope yes o c05 input-validation in-scope yes o,s,s c06 rate-limit in-scope yes o,d c07 information-disclosure shared no o,o c08 authorization-check shared yes o,s c09 configuration out-of-scope yes o c10 input-validation in-scope no o,s c11 session-management in-scope yes o,o,o c12 rate-limit in-scope no o,d c13 information-disclosure in-scope no o c14 authorization-check in-scope yes o,o,s,s c15 input-validation shared no o,s c16 configuration in-scope no o,o c17 session-management out-of-scope no o,s c18 rate-limit in-scope yes o,o,d `.trim().split('\n').map((r) => { const [name, cls, target, actual, plan] = r.trim().split(/\s+/); return { name, cls, target, actual: actual === 'yes', plan: plan.split(',') }; }); // A candidate's outcome under a given rule set. export function result(c, rule) { const allowed = RULE[rule].filter((s) => SCOPE[c.target].includes(s)); const ran = c.plan.filter((s) => allowed.includes(s)); const complete = ran.length === c.plan.length; const s = { ran: ran.length, skipped: c.plan.length - ran.length, record: 0, alert: 0, min: 0 }; for (const step of ran) { s.record += RISK[step].record; s.alert += RISK[step].alert; s.min += RISK[step].min; } s.status = complete ? (c.actual ? 'evidence' : 'false-positive') : 'unverifiable'; return s; } export function ruleMeasure(rule) { const t = { evidence: 0, falsePositive: 0, unverifiable: 0, request: 0, skipped: 0, record: 0, alert: 0, min: 0, outOfScopeTouch: 0 }; for (const c of CANDIDATE) { const s = result(c, rule); t[s.status === 'false-positive' ? 'falsePositive' : s.status] += 1; t.request += s.ran; t.skipped += s.skipped; t.record += s.record; t.alert += s.alert; t.min += s.min; if (c.target === 'out-of-scope' && s.ran > 0) t.outOfScopeTouch += 1; } return t; } ``` #### Eighteen Candidates, Three Rule Sets ```js // vek/measure.mjs — eighteen candidates are run through three rule sets. import { CANDIDATE, RULE, RISK, result, ruleMeasure } from './validate.mjs'; const y = (g, ...s) => console.log(s.map((v, i) => (g[i] < 0 ? String(v).padEnd(-g[i]) : String(v).padStart(g[i]))).join('').trimEnd()); const plan = CANDIDATE.flatMap((c) => c.plan); console.log(`${CANDIDATE.length} candidates, ${plan.length} planned requests: ` + ['o', 's', 'd'].map((s) => `${s}=${plan.filter((x) => x === s).length}`).join(' ')); console.log(`actually open candidates ${CANDIDATE.filter((c) => c.actual).length} (model; ` + `unknown before validation)`); const A = [-6, -24, -13, 6, 8, 16]; console.log('\n1. plan and outcome per candidate (rule: reversible)'); y(A, 'cand', 'class', 'target', 'plan', 'ran', 'status'); for (const c of CANDIDATE) { const s = result(c, 'reversible'); y(A, c.name, c.cls, c.target, c.plan.join(''), s.ran, s.status); } const B = [-26, 9, 16, 14, 9, 9, 8, 8, 7]; console.log('\n2. totals per rule set'); y(B, 'rule', 'evidence', 'false positive', 'unverifiable', 'request', 'skipped', 'record', 'alert', 'min'); for (const k of Object.keys(RULE)) { const t = ruleMeasure(k); y(B, k, t.evidence, t.falsePositive, t.unverifiable, t.request, t.skipped, t.record, t.alert, t.min); } console.log('\n3. scope check'); for (const k of Object.keys(RULE)) console.log(` ${k.padEnd(26)} requests to out-of-scope target: ${ruleMeasure(k).outOfScopeTouch}`); const g = ruleMeasure('observation-only'), e = ruleMeasure('written-extra-permission'); console.log(`\ncandidates decided by observation only ${g.evidence + g.falsePositive} / ${CANDIDATE.length}`); console.log(`candidates decided by the broadest rule set ${e.evidence + e.falsePositive} / ${CANDIDATE.length}`); console.log(`candidates unverifiable under any rule set ${e.unverifiable}`); const openButUnproven = CANDIDATE.filter((c) => c.actual && result(c, 'written-extra-permission').status !== 'evidence'); console.log(` of these, actually open ${openButUnproven.length} ` + `(${openButUnproven.map((c) => c.name).join(', ')})`); console.log(`evidence gained per alert ` + `${((e.evidence - g.evidence) / (e.alert - g.alert)).toFixed(2)}`); ``` ``` 18 candidates, 40 planned requests: o=27 s=10 d=3 actually open candidates 10 (model; unknown before validation) 1. plan and outcome per candidate (rule: reversible) cand class target plan ran status c01 information-disclosure in-scope oo 2 evidence c02 authorization-check in-scope oos 3 evidence c03 session-management in-scope oos 3 false-positive c04 configuration in-scope o 1 evidence c05 input-validation in-scope oss 3 evidence c06 rate-limit in-scope od 1 unverifiable c07 information-disclosure shared oo 2 false-positive c08 authorization-check shared os 1 unverifiable c09 configuration out-of-scope o 0 unverifiable c10 input-validation in-scope os 2 false-positive c11 session-management in-scope ooo 3 evidence c12 rate-limit in-scope od 1 unverifiable c13 information-disclosure in-scope o 1 false-positive c14 authorization-check in-scope ooss 4 evidence c15 input-validation shared os 1 unverifiable c16 configuration in-scope oo 2 false-positive c17 session-management out-of-scope os 0 unverifiable c18 rate-limit in-scope ood 2 unverifiable 2. totals per rule set rule evidence false positive unverifiable request skipped record alert min observation-only 3 3 12 25 15 25 0 50 reversible 6 5 7 32 8 32 7 85 written-extra-permission 8 6 4 35 5 35 16 130 3. scope check observation-only requests to out-of-scope target: 0 reversible requests to out-of-scope target: 0 written-extra-permission requests to out-of-scope target: 0 candidates decided by observation only 6 / 18 candidates decided by the broadest rule set 14 / 18 candidates unverifiable under any rule set 4 of these, actually open 2 (c08, c09) evidence gained per alert 0.31 ``` #### Validation's Real Yield The first two columns of the second table are read together. The observation-only rule yields 3 pieces of evidence and 3 false positives; the broadest rule yields 8 evidence and 6 false positives. From the report's point of view, these two columns carry **equal value**. Eliminating six false positives keeps the remediation team from dealing with six lines; each line means a review, a discussion, and often a change request. An **eliminated candidate** is as much a product of validation as an evidenced finding. The third column is the most honest part of the report. `unverifiable` does not mean "this candidate is not open"; it means "I could not decide this candidate while staying within the rules." Losing this distinction is a known flaw of penetration-test reports: an unverifiable candidate is silently dropped, and the organization thinks an item was checked when it never was. Even under the broadest rule set, four of the eighteen candidates stay in this column, and looking at the model's `actual` field shows that **two of them are really open**. Which two candidates those are is the most important line in this lesson. `c08` is in a shared environment: the state-changing request required to prove it cannot be sent because it might touch another tenant's data in the same environment. `c09` belongs to a third party, and not even a single observation request can be sent. Both are really open, and both are reported unproven — not as a finding, but as **risk left out of scope**. #### The Length of the Plan Is a Finding The `plan` column in the first table carries information on its own. Three candidates have a plan of a single request; one candidate's plan is four requests long, and two of those are state-changing. This difference is not random: the number of requests needed to prove a candidate is directly proportional to the **number of controls standing in front of it**. A configuration item that can be shown with a single request has stayed at one request precisely because no control stands in front of it; an authorization-check item that needs four requests only reveals itself after a chain of conditions is satisfied. This is used in the report in two directions. A finding with a short plan is more serious, because that same shortness also holds on the attacker's side. A finding with a long plan can drop in remediation priority, but the justification for that is not "it is hard" — it is **how many conditions have to be satisfied together**, and those conditions are written out one by one. The plan's length is also a direct multiplier on the cost side. Of the forty planned requests, 27 are in the observation class; of the 35 requests that run under the broadest rule set, 25 are observation, 7 are state-changing, and 3 carry a risk of disruption. Of the 130 total minutes, 50 come from observation requests, and the remaining 80 minutes come from only ten requests. Validation's duration is set not by the request count but by the **request class**. The request and response pairs the proxy records become raw material for reproduction steps at the next stage. What goes into the report is not this raw record: the class, the count, the condition, and the observed outcome are written. Adding a validation record to the report as-is means giving everyone who reads the report the ability to resend the same request, and this needlessly enlarges the risk of the remediation process. #### Step Count and the Trace Left Behind As the rule set widens, three numbers rise together. Requests go from 25 to 35, time from 50 minutes to 130 minutes, alerts from 0 to 16. The record count tracks the request count one-to-one, because every request leaves a record on the target; the **alert** column is the distinguishing one. Validation run with observation-only requests produces 25 records and **zero alerts**. Seen from the defense side, this is an ordinary pile of requests in the log file; it is noticed only later, through analysis based on volume and pattern. Opening up state-changing requests brings seven alerts; opening up service-disruption-risking requests brings sixteen. The real number here is the marginal rate: moving from observation-only to the broadest rule gains **five extra pieces of evidence** while producing **sixteen alerts** — 0.31 evidence per alert. This ratio explains why the rules of engagement are a subject of negotiation. Some of the alerts are a sign that the defense is really working, and the blue team is meant to see them. Some of them put the on-call team on the trail of what they think is a real incident and trigger a response; the test window, the communication channel, and the previously reported source list are written for exactly this reason. Keeping the alert count at zero is also an option, but its cost is clear: twelve of the eighteen candidates stay undecided. The third table gives the same number across all three rule sets: requests going to an out-of-scope target are **zero**. This number is not an outcome, it is an **invariant**; whichever rule set is chosen, it has to stay at zero. Running the scope check programmatically on the plan, rather than leaving it to the tool or the person, secures this: before a request is sent, the target's class is asked, if the intersection is empty the step is skipped, and the skip is written down. #### Summary - The validation plans of the eighteen candidates contain 40 requests in total: 27 observation-only, 10 state-changing, 3 carrying a risk of service disruption. - As the rule set widens, decided candidates go from 6 to 14; evidence goes from 3 to 8, and eliminated false positives go from 3 to 6. An eliminated candidate is also a product of validation. - Validation run with observation-only requests leaves 25 records and zero alerts; under the broadest rule, 35 records and 16 alerts arise. The marginal rate is 0.31 evidence per alert. - Four candidates cannot be verified under any rule set, and two of them are really open; these are reported not as findings but as risk left out of scope. - The plan's length measures the number of controls standing in front of a candidate; an item shown with a single request is more serious precisely because that same shortness also holds on the attacker's side. - Scope checking is programmatic, and the number of requests to an out-of-scope target is zero under all three rule sets; this is not an outcome, it is an invariant that must be preserved. #### Next Step All three of these tools worked over the network: they recorded traffic, read fields, changed a request and saw the response. A portion of the items that need examining during an assessment, though, come not from the network but from a **file**: an unknown executable found on an endpoint, an archive extracted from an email attachment, a helper tool the field team obtained from outside. There is no way to learn what these do by sending requests. The next lesson takes up sandbox analysis on this axis. The Endpoint and System Hardening course measured the sandbox's isolation strength; here that measurement is taken as input, and a single question is asked: how many of an item's behaviors can be observed, and how many of them notice that the environment is a sandbox and **hide**. Hidden behavior feeds the unverifiable-candidate column this time not from the network's limit, but from the analysis environment's own limit. ### Sandbox Analysis Source: https://academia.sh/en/courses/penetration-testing/sandbox-analysis Measuring sandbox analysis in terms of suppressed behavior: 5 of 14 items noticing the environment on a stock image and not showing their decisive behavior, the number of resolved items rising from 7 to 12 while total time rises from 288 minutes to 965 minutes, one environment hint being impossible to hide in any environment, and analysis leaving zero records on the target while dropping a trace on the other side. The previous three tools worked over the network. A portion of the items that need examining during an assessment, though, come from a file: an unknown executable found on an endpoint, an archive extracted from an attachment, a helper tool the field team obtained from outside. There is no way to learn what these do by sending requests; the only way to learn is to **run** the item in an isolated place and observe its behavior. The sandbox's isolation strength was measured in the Endpoint and System Hardening course and is taken as input here; what is measured in this lesson is not isolation but **analysis's yield**. The only question is: how many of an item's behaviors can be observed, how many of them notice that the environment is an analysis environment and hide, and how many items end up unable to be decided. #### Environment Hints and Hiding An item can look at the environment it is running in while it runs. What it looks at are the traces that separate analysis environments from real endpoints: the machine having been up for a very short time, the low number of devices, an empty user history, traces left by analysis tools, an accelerated clock, uniform network responses. If even one of these traces is still standing, the item recognizes the environment and **never shows its decisive behavior** — it runs, does a few harmless things, and stops. Every trace can be hidden, and each one carries a **prep cost**: refurbishing the image, building a fake user history, cleaning up tool traces. One trace falls outside this. The only way to fix uniform network responses is to let the item reach a real target; in an authorized assessment this step is out of scope, and connecting to another party's infrastructure without written authorization is both illegal and unprofessional. This trace is a boundary that prep cannot close. - **AT22.** The hint catalog, environments, item list, and behaviors are a **model**; no real item was run. Items are referred to only by the hints they check and the class of behavior observed. - **AT23.** Isolation strength is not measured here; it is taken as input. - **AT24.** An item that has **at least one** of the hints it checks not hidden notices the environment and does not show its decisive behaviors. It keeps showing its harmless behaviors. - **AT25.** For a behavior to be visible, the observation window has to be larger than its delay. Delays and windows were written by hand; there is no randomness. - **AT26.** The decision is three-valued. **Evidence**: a decisive behavior was observed. **Clean**: the item was not hidden, all its behaviors fit in the window, and none was decisive. **Undetermined**: every remaining case. - **AT27.** The decisive marker belongs to the behavior, not the item; a legitimate item can also show a behavior counted as decisive. - **AT28.** Analysis does not run in the target environment; it does not produce records on target systems. ```js // alan/observe.mjs — this is a model of SANDBOX ANALYSIS. No real item is run; // items are referred to only by the environment hints they check and the class of behavior // observed. Isolation strength is not measured here; it is taken as input from the Endpoint // and System Hardening course. // hint -> the prep cost of hiding that hint (min) export const HINT = { 'short-uptime': 8, 'low-device-count': 12, 'known-tool-traces': 15, 'empty-user-history': 20, 'accelerated-clock': 10, 'uniform-network-response': 0, }; // Analysis environment: which hints it hides, how many minutes the observation window is. // 'uniform-network-response' is not hidden in any environment: hiding it requires the item // to reach a real target, and that is out of scope in an authorized assessment. export const ENVIRONMENT = { 'stock-image': { hides: ['short-uptime'], window: 5 }, 'refurbished': { hides: ['short-uptime', 'low-device-count', 'known-tool-traces'], window: 20 }, 'near-real': { hides: ['short-uptime', 'low-device-count', 'known-tool-traces', 'empty-user-history', 'accelerated-clock'], window: 60 }, }; export const MANUAL_MIN = 30; // manual review per undetermined item // row: item | actual | hints it checks | behaviors (name:delay, * = decisive) export const ITEM = ` k01 bad short-uptime file-write:0,*persistence:2 k02 bad low-device-count,known-tool-traces process-creation:0,*credential-read:5 k03 bad empty-user-history file-write:0,*external-connection:10 k04 bad accelerated-clock file-write:0,*timed-trigger:45 k05 bad uniform-network-response process-creation:0,*external-connection:3 k06 bad - file-write:0,*persistence:1,external-connection:4 k07 bad short-uptime,uniform-network-response file-write:0,*credential-read:8 k08 clean - file-write:0,process-creation:1 k09 clean - file-write:0,external-connection:2 k10 clean - file-write:0,process-creation:0,external-connection:1 k11 clean - file-write:0,*credential-read:3 k12 clean - file-write:0,timed-trigger:40 k13 clean - process-creation:0,external-connection:35 k14 clean - file-write:0 `.trim().split('\n').map((r) => { const [name, actual, hint, dav] = r.trim().split(/\s+/); return { name, bad: actual === 'bad', controls: hint === '-' ? [] : hint.split(','), behavior: dav.split(',').map((d) => { const [n, g] = d.split(':'); return { name: n.replace('*', ''), decisive: n.startsWith('*'), delay: +g }; }) }; }); // An item that has one of the hints it checks not hidden notices the environment and HIDES: // it never shows its decisive behaviors. export const hides = (k, o) => k.controls.some((i) => !ENVIRONMENT[o].hides.includes(i)); export function observe(k, o) { const p = ENVIRONMENT[o].window; const g = hides(k, o); const observed = k.behavior.filter((d) => d.delay <= p && !(g && d.decisive)); const suppressed = k.behavior.length - observed.length; let decision = 'undetermined'; if (observed.some((d) => d.decisive)) decision = 'evidence'; else if (!g && suppressed === 0) decision = 'clean'; return { hidden: g, observed: observed.length, suppressed, decision, externalContact: observed.some((d) => d.name === 'external-connection') }; } export function measureEnvironment(o) { const s = { prep: 0, hidden: 0, observed: 0, suppressed: 0, evidence: 0, clean: 0, undetermined: 0, falseEvidence: 0, escaped: 0, externalTrace: 0 }; s.prep = ENVIRONMENT[o].hides.reduce((t, i) => t + HINT[i], 0); for (const k of ITEM) { const r = observe(k, o); if (r.hidden) s.hidden += 1; s.observed += r.observed; s.suppressed += r.suppressed; s[r.decision] += 1; if (r.decision === 'evidence' && !k.bad) s.falseEvidence += 1; if (r.decision !== 'evidence' && k.bad) s.escaped += 1; if (r.externalContact) s.externalTrace += 1; } s.observeMin = ITEM.length * ENVIRONMENT[o].window; s.manualMin = s.undetermined * MANUAL_MIN; s.totalMin = s.prep + s.observeMin + s.manualMin; return s; } ``` #### Fourteen Items, Three Environments ```js // alan/measure.mjs — fourteen items are observed in three analysis environments. import { HINT, ENVIRONMENT, ITEM, MANUAL_MIN, hides, observe, measureEnvironment } from './observe.mjs'; const y = (g, ...s) => console.log(s.map((v, i) => (g[i] < 0 ? String(v).padEnd(-g[i]) : String(v).padStart(g[i]))).join('').trimEnd()); console.log(`${ITEM.length} items (${ITEM.filter((k) => k.bad).length} bad), ` + `${Object.keys(HINT).length} environment hints, ${Object.keys(ENVIRONMENT).length} analysis environments`); console.log(`hint never hidden in any environment: ` + Object.keys(HINT).filter((i) => !Object.values(ENVIRONMENT).some((o) => o.hides.includes(i))).join(', ')); const A = [-14, 9, 10, 10, 12, 10, 8, 14, 16]; console.log('\n1. result per environment'); y(A, 'environment', 'window', 'hidden', 'observed', 'suppressed', 'evidence', 'clean', 'undetermined', 'false evidence'); for (const o of Object.keys(ENVIRONMENT)) { const s = measureEnvironment(o); y(A, o, ENVIRONMENT[o].window, s.hidden, s.observed, s.suppressed, s.evidence, s.clean, s.undetermined, s.falseEvidence); } const B = [-7, 8, 23, 23, 23]; console.log('\n2. decision per item'); y(B, 'item', 'actual', ...Object.keys(ENVIRONMENT)); for (const k of ITEM) y(B, k.name, k.bad ? 'bad' : 'clean', ...Object.keys(ENVIRONMENT).map((o) => { const r = observe(k, o); return r.hidden ? `${r.decision} (hidden)` : r.decision; })); const C = [-14, 8, 12, 11, 11, 13, 16]; console.log('\n3. cost and trace'); y(C, 'environment', 'prep', 'observe min', 'manual min', 'total min', 'escaped bad', 'external trace'); for (const o of Object.keys(ENVIRONMENT)) { const s = measureEnvironment(o); y(C, o, s.prep, s.observeMin, s.manualMin, s.totalMin, s.escaped, s.externalTrace); } const h = measureEnvironment('stock-image'), g = measureEnvironment('near-real'); console.log(`\nitems resolved ${h.evidence + h.clean} -> ${g.evidence + g.clean} / ${ITEM.length}`); console.log(`items detecting env ${h.hidden} -> ${g.hidden} / ${ITEM.length}`); console.log(`records produced on target 0 (analysis does not run in the target environment)`); console.log(`manual review time ${MANUAL_MIN} min/item`); ``` ``` 14 items (7 bad), 6 environment hints, 3 analysis environments hint never hidden in any environment: uniform-network-response 1. result per environment environment window hidden observed suppressed evidence clean undetermined false evidence stock-image 5 5 22 7 3 4 7 1 refurbished 20 4 23 6 4 4 6 1 near-real 60 2 27 2 6 6 2 1 2. decision per item item actual stock-image refurbished near-real k01 bad evidence evidence evidence k02 bad undetermined (hidden) evidence evidence k03 bad undetermined (hidden) undetermined (hidden) evidence k04 bad undetermined (hidden) undetermined (hidden) evidence k05 bad undetermined (hidden) undetermined (hidden) undetermined (hidden) k06 bad evidence evidence evidence k07 bad undetermined (hidden) undetermined (hidden) undetermined (hidden) k08 clean clean clean clean k09 clean clean clean clean k10 clean clean clean clean k11 clean evidence evidence evidence k12 clean undetermined undetermined clean k13 clean undetermined undetermined clean k14 clean clean clean clean 3. cost and trace environment prep observe min manual min total min escaped bad external trace stock-image 8 70 210 288 5 3 refurbished 35 280 180 495 4 3 near-real 65 840 60 965 2 5 items resolved 7 -> 12 / 14 items detecting env 5 -> 2 / 14 records produced on target 0 (analysis does not run in the target environment) manual review time 30 min/item ``` #### The Suppressed Behavior The `hidden` column of the first table is this lesson's measure. On a stock image, **five** of fourteen items notice the environment and never show their decisive behavior. As the environment is refurbished this number drops to 4, then to 2. While observed behavior rises from 22 to 27, suppressed behavior falls from 7 to 2. Hiding is a different thing from a short observation window, and the two need to be told apart. If the window is short, the behavior has not been seen **yet**; lengthening it fixes that. If the item is hiding, the behavior will **never** be seen; lengthening the window changes nothing, the hint has to be hidden. The second table reads this distinction item by item: `k04` has a behavior with a forty-five-minute delay and checks the clock hint; seeing it requires both the window to reach sixty minutes and the accelerated clock to be hidden — one without the other is not enough. By contrast, `k12` and `k13` check nothing at all; for them, lengthening the window alone is enough, and both come out clean in the widest environment. Which prep step unlocks which item is also read from this table, and the gain is not distributed evenly. Moving from the stock image to the refurbished environment asks for 27 extra minutes of prep and turns a single item — `k02`, which checks both device count and tool traces — into evidence. Moving from refurbished to near-real asks for 30 extra minutes of prep and a window three times as long; in return, `k03` and `k04` turn into evidence, and `k12` and `k13` turn clean. The second step does four times the work for a prep cost only slightly above the first, but it does so by pushing the window from 280 minutes to 840 minutes. Prep is cheap, the window is expensive, and they are two separate line items. Two items cannot be resolved in any environment: `k05` and `k07`. Both check the uniformity of network responses, and that hint is not hidden in any environment. Two of the seven bad items stay **undetermined** regardless of the analysis environment's quality. These two items get handed off to manual review and appear in the report with the line "analysis could not resolve this." #### Evidence, Clean, and False Evidence The candidate-evidence gap is read here as three values. Fourteen items arrive at analysis as candidates. On the stock image, 3 turn into evidence, 4 come out clean, 7 stay undetermined. In the best environment there are 6 evidence, 6 clean, and 2 undetermined. The number of resolved items goes from **7 to 12**. Items that come out clean are worth as much as evidence. If all of an item's behaviors were seen and none was decisive, that item drops out of the queue — this is the only mechanism that shortens the candidate list. But the condition for a "clean" decision is strict: the item must not be hidden **and** none of its behaviors may have been suppressed. If even a single behavior was suppressed, the decision goes to undetermined, not clean. This strictness is deliberate, because relaxing it would let `k05` and `k07` quietly count as clean. An error in the opposite direction stands in every environment: the **false evidence** column is 1 in all three. A legitimate item shows a behavior counted as decisive, and analysis flags it as evidence. Raising the prep, lengthening the window, and hiding hints do not change this line, because what is missing is not observation, it is **context**. The sandbox sees the behavior; it does not see whether that behavior belongs there, what job the item was set up to do. That is why analysis output alone is not a decision — it is input to one. #### Cost and the Trace Left on the Other Side The third table gives the cost, and it rises in one direction only. Prep goes from 8 minutes to 65; observation goes from 70 to 840. Manual review goes the other way — from 210 minutes to 60 — but the drop does not offset the rise: the total goes from 288 minutes to 965 minutes, more than tripling. In return, the number of resolved items rises by five, and the number of escaped bad items falls from 5 to 2. The cost paid per item is roughly 135 minutes. The last column is a kind of trace seen for the first time in this course. Records produced on target systems are **zero**; analysis does not run in the target environment, none of its requests reach the target. But the number of items that attempt an external connection during observation rises from 3 to 5, and those attempts go somewhere. Even if the controlled network does not let them reach a real target, the attempt itself and its timing can leak outward in a way that tells the other side something. The closer the analysis environment gets to reality, the more behavior an item shows; and every behavior it shows is also **a sign that it is being examined**. This splits the course's trace column in two: the trace the defense sees, and the trace the other side can see. The two point in opposite directions. The trace the defense sees is a sign that the assessment is running transparently inside the organization, and its excess only creates noise. The trace left on the other side is irreversible: learning that an item is being examined is enough for whoever is using it to change their behavior, and it corrupts every measurement after that. The next lesson measures the second one far more sharply. #### Summary - Of fourteen items, five notice the environment on a stock image and never show their decisive behavior; as the environment is refurbished this number falls to 2, and observed behavior rises from 22 to 27. - Hiding and a short window are separate problems: the first is solved by hiding a hint, the second by lengthening the window, and some items need both at once. - One environment hint cannot be hidden in any environment, because hiding it requires the item to reach a real target, which is out of scope; two of the seven bad items stay undetermined in every environment for this reason. - The number of resolved items goes from 7 to 12; items that come out clean are the only mechanism that shortens the candidate list, and its condition is strict. - False evidence is 1 in all three environments; what is missing is not observation, it is context, and prep does not close that gap. - Total time goes from 288 minutes to 965 minutes; records produced on target are zero, but the number of items attempting an external connection rises from 3 to 5, leaving a trace on the other side. #### Next Step This lesson measured a single analysis environment's decision, and two items stayed unresolved. There is a second way to resolve an item: ask the same item to **a large number of independent engines** and combine the results that come back. This route brings analysis time down from minutes to seconds, and in return raises a new question — how to combine results that contradict each other. The next lesson takes up multi-engine reputation services on this axis. The same set of items is run through a large number of engines, the false-positive and false-negative rate of the **majority decision** is measured threshold by threshold, and the cost of trusting a single engine is counted. Alongside this, a privacy cost is computed: sending a file to an outside service is publishing it, and a portion of the items in the fictional measurement network carry subscriber data. ### File and Link Reputation Services Source: https://academia.sh/en/courses/penetration-testing/file-and-link-reputation-services Measuring the combination of multi-engine reputation results: recall falling from 92 percent to 8 percent as the majority threshold rises across 24 items and 12 engines, precision instead rising from 50 percent to 86 percent at the seventh threshold and then returning to 50 percent, the majority decision within a single engine family turning out identical to picking a single engine, 10 of the 24 items never being queryable at all because of the privacy limit, and every query leaving a trace on the other side. In the previous lesson two items could not be resolved in any analysis environment, and a separate prep, observation, and manual-review budget was spent for every item. There is a second way to resolve an item: ask the same item to a large number of independent engines and combine the results that come back. This route takes seconds, and in return raises a new question — how to combine results that contradict each other. A **multi-engine reputation service** asks a large number of detection engines about an item or a link target and shows the flags that come back side by side. The output is not a decision, it is a **vote distribution**; the decision is made by whoever reads that distribution. What this lesson measures is the quality of that reading. #### Counting Votes and Independence A threshold has to be chosen: how many engines flagging an item counts it as bad. As the threshold drops, more bad items get caught and more clean items get mistakenly flagged; as it rises, the opposite happens. Two numbers measure this trade-off. **Precision** is how much of what we called bad is actually bad. **Recall** is how much of what is actually bad we caught. The majority decision paying off depends on a single condition: engines being wrong **independently of each other**. Two engines using the same method get the same items wrong; their combined vote says no more than a single vote. In this lesson engines are split into two families and are considered nested within a family: a highly sensitive engine flags every item the less sensitive one flags. There is no such relationship between families. - **AT29.** The engine list, item list, and threshold values are a **model**; there is no real reputation service, no real file, no real vulnerability identity. - **AT30.** An engine only flags an item if its sensitivity is greater than or equal to the item's threshold in that family. For bad items the threshold means "the sensitivity needed to catch it"; for clean items it means "the sensitivity needed to flag it falsely." - **AT31.** Engines are nested within a family; families are independent of each other. In reality there is partial dependence between engines, and this model shows the two extremes together. - **AT32.** There is no randomness; every threshold was written by hand. - **AT33.** An item with an `open` privacy class can have its file sent. `internal` and `subscriber-data` items have no file sent; only a hash is queried, and a hash query only returns a result if the item is **known** on the service side. - **AT34.** Sending a file to an outside service is publishing it; it is irreversible. - **AT35.** The query does not go to target systems; it does not produce a record on target. ```js // itibar/engine.mjs — this is a MODEL of a multi-engine reputation service. There is no real // service, no real file, and no real item; engines are referred to only by their family and // sensitivity value. Two families look at two separate signals. Engines within a family are // nested: a highly sensitive engine flags every item the less sensitive one flags. export const ENGINE = [ ...[30, 40, 50, 60, 75, 90].map((d, i) => ({ name: `s${i + 1}`, family: 'signature', sensitivity: d })), ...[30, 40, 50, 60, 75, 90].map((d, i) => ({ name: `b${i + 1}`, family: 'behavior', sensitivity: d })), ]; // Item: if bad, threshold = lowest sensitivity needed to catch it (per family). // If clean, threshold = lowest sensitivity needed to flag it falsely. 95 = no engine. // privacy: open = file can be sent; internal and subscriber-data = cannot be sent, only a hash is queried. // known: whether the hash query has a match on the service side. export const ITEM = ` k01 bad 30 30 open yes k02 bad 40 40 open yes k03 bad 50 40 internal yes k04 bad 60 50 open no k05 bad 50 60 internal no k06 bad 75 60 subscriber-data yes k07 bad 60 75 internal no k08 bad 75 75 open no k09 bad 90 75 internal yes k10 bad 95 60 subscriber-data no k11 bad 95 95 internal no k12 bad 40 50 open no k13 clean 95 95 open yes k14 clean 30 95 internal yes k15 clean 95 30 subscriber-data yes k16 clean 40 95 internal no k17 clean 95 40 open yes k18 clean 50 95 internal no k19 clean 95 50 subscriber-data yes k20 clean 60 95 internal no k21 clean 95 60 open yes k22 clean 75 95 internal no k23 clean 95 75 subscriber-data no k24 clean 30 30 subscriber-data no `.trim().split('\n').map((r) => { const [name, actual, sig, beh, privacy, known] = r.trim().split(/\s+/); return { name, bad: actual === 'bad', threshold: { signature: +sig, behavior: +beh }, privacy, known: known === 'yes' }; }); export const flags = (m, k) => m.sensitivity >= k.threshold[m.family]; export const flagCount = (k, set = ENGINE) => set.filter((m) => flags(m, k)).length; // Majority decision: an item is called bad if at least 'threshold' engines in the given set flagged it. export function majority(threshold, set = ENGINE) { const s = { tp: 0, fp: 0, fn: 0, tn: 0 }; for (const k of ITEM) { const calledBad = flagCount(k, set) >= threshold; if (calledBad && k.bad) s.tp += 1; else if (calledBad) s.fp += 1; else if (k.bad) s.fn += 1; else s.tn += 1; } s.precision = s.tp + s.fp ? s.tp / (s.tp + s.fp) : 1; s.recall = s.tp / ITEM.filter((k) => k.bad).length; return s; } export const singleEngine = (m) => majority(1, [m]); // Privacy limit: a file can only be sent for 'open' items. For the others a hash is queried, // and a hash query only returns a result if the item is known on the service side. export const queryable = (k) => k.privacy === 'open' || k.known; export const publishable = (k) => k.privacy === 'open'; ``` #### Twenty-Four Items, Twelve Engines ```js // itibar/measure.mjs — 24 items are run through 12 engines; the majority threshold and engine family are scanned. import { ENGINE, ITEM, flagCount, majority, singleEngine, queryable, publishable } from './engine.mjs'; const y = (g, ...s) => console.log(s.map((v, i) => (g[i] < 0 ? String(v).padEnd(-g[i]) : String(v).padStart(g[i]))).join('').trimEnd()); const percent = (x) => `${(x * 100).toFixed(0)}%`; const sig = ENGINE.filter((m) => m.family === 'signature'); const beh = ENGINE.filter((m) => m.family === 'behavior'); console.log(`${ITEM.length} items (${ITEM.filter((k) => k.bad).length} bad), ` + `${ENGINE.length} engines: ${sig.length} signature + ${beh.length} behavior`); const A = [-8, 8, 11, 10, 10, 18]; console.log('\n1. flag count per item'); y(A, 'item', 'actual', 'signature', 'behavior', 'total', 'privacy'); for (const k of ITEM) y(A, k.name, k.bad ? 'bad' : 'clean', flagCount(k, sig), flagCount(k, beh), flagCount(k), k.privacy); const B = [-11, 8, 8, 8, 11, 8]; console.log('\n2. majority threshold scan (12 engines)'); y(B, 'threshold', 'true+', 'false+', 'false-', 'precision', 'recall'); for (let e = 1; e <= ENGINE.length; e += 1) { const s = majority(e); y(B, e, s.tp, s.fp, s.fn, percent(s.precision), percent(s.recall)); } const C = [-34, 8, 8, 8, 11, 8]; console.log('\n3. comparison of decision rules'); y(C, 'decision rule', 'true+', 'false+', 'false-', 'precision', 'recall'); const row = (name, s) => y(C, name, s.tp, s.fp, s.fn, percent(s.precision), percent(s.recall)); const singles = ENGINE.map((m) => ({ m, s: singleEngine(m) })).filter((x) => x.s.tp > 0); const mostPrecise = singles.reduce((a, b) => (b.s.precision > a.s.precision ? b : a)); row(`most precise single engine (${mostPrecise.m.name})`, mostPrecise.s); row('signature only, threshold 2', majority(2, sig)); row('behavior only, threshold 2', majority(2, beh)); row('mixed, threshold 3', majority(3)); row('mixed, threshold 5', majority(5)); row('mixed, threshold 7', majority(7)); const identical = (set) => set.every((_, i) => { const a = majority(i + 1, set), b = singleEngine(set[set.length - 1 - i]); return a.tp === b.tp && a.fp === b.fp; }); console.log(` is majority within a single family identical to picking a single engine: ` + `signature ${identical(sig) ? 'yes' : 'no'}, behavior ${identical(beh) ? 'yes' : 'no'}`); console.log('\n4. privacy limit and trace'); const open = ITEM.filter(publishable), queried = ITEM.filter(queryable); console.log(` items whose file can be sent (publishable) ${open.length} / ${ITEM.length}`); console.log(` items matched via hash query ${queried.length - open.length}`); console.log(` items that cannot be queried in any form ${ITEM.length - queried.length}`); console.log(` of these, actually bad ` + `${ITEM.filter((k) => k.bad && !queryable(k)).length}`); console.log(` unsendable because they carry subscriber data ` + `${ITEM.filter((k) => k.privacy === 'subscriber-data').length}`); console.log(` submissions leaving a trace on the other side ${queried.length}`); console.log(` records produced on target systems 0`); console.log(`\nbad items no engine flags ` + `${ITEM.filter((k) => k.bad && flagCount(k) === 0).length}`); console.log(`clean items every engine flags ` + `${ITEM.filter((k) => !k.bad && flagCount(k) === ENGINE.length).length}`); console.log(`bad items a single family is blind to ` + `${ITEM.filter((k) => k.bad && (flagCount(k, sig) === 0 || flagCount(k, beh) === 0)).length}`); ``` ``` 24 items (12 bad), 12 engines: 6 signature + 6 behavior 1. flag count per item item actual signature behavior total privacy k01 bad 6 6 12 open k02 bad 5 5 10 open k03 bad 4 5 9 internal k04 bad 3 4 7 open k05 bad 4 3 7 internal k06 bad 2 3 5 subscriber-data k07 bad 3 2 5 internal k08 bad 2 2 4 open k09 bad 1 2 3 internal k10 bad 0 3 3 subscriber-data k11 bad 0 0 0 internal k12 bad 5 4 9 open k13 clean 0 0 0 open k14 clean 6 0 6 internal k15 clean 0 6 6 subscriber-data k16 clean 5 0 5 internal k17 clean 0 5 5 open k18 clean 4 0 4 internal k19 clean 0 4 4 subscriber-data k20 clean 3 0 3 internal k21 clean 0 3 3 open k22 clean 2 0 2 internal k23 clean 0 2 2 subscriber-data k24 clean 6 6 12 subscriber-data 2. majority threshold scan (12 engines) threshold true+ false+ false- precision recall 1 11 11 1 50% 92% 2 11 11 1 50% 92% 3 11 9 1 55% 92% 4 9 7 3 56% 75% 5 8 5 4 62% 67% 6 6 3 6 67% 50% 7 6 1 6 86% 50% 8 4 1 8 80% 33% 9 4 1 8 80% 33% 10 2 1 10 67% 17% 11 1 1 11 50% 8% 12 1 1 11 50% 8% 3. comparison of decision rules decision rule true+ false+ false- precision recall most precise single engine (b5) 11 6 1 65% 92% signature only, threshold 2 9 6 3 60% 75% behavior only, threshold 2 11 6 1 65% 92% mixed, threshold 3 11 9 1 55% 92% mixed, threshold 5 8 5 4 62% 67% mixed, threshold 7 6 1 6 86% 50% is majority within a single family identical to picking a single engine: signature yes, behavior yes 4. privacy limit and trace items whose file can be sent (publishable) 8 / 24 items matched via hash query 6 items that cannot be queried in any form 10 of these, actually bad 4 unsendable because they carry subscriber data 6 submissions leaving a trace on the other side 14 records produced on target systems 0 bad items no engine flags 1 clean items every engine flags 1 bad items a single family is blind to 2 ``` #### What a Threshold Buys The second table tries every threshold in sequence and shows that no single row raises both precision and recall at once. If one engine flagging is enough, recall goes up to 92 percent and precision stays at 50 percent: half of the 22 items called bad are clean. If seven engines are required, precision goes up to 86 percent and recall drops to 50 percent: six bad items are missed. Two items stand at the two ends of this curve, and both are independent of the threshold. `k11` is not flagged by any engine; whichever threshold is chosen, it slips through. `k24` is flagged by every engine; whichever threshold is chosen, it stays a false positive. These two items are an error floor that counting votes cannot fix, and they have to be handled in the report not with the line "the reputation query could not decide" but by another method. Choosing a threshold is therefore not a technical decision but a **business decision**: if the cost of a false positive is an hour of investigation, the threshold is pulled down; if it is recalling the field team, it is pulled up. The same table makes two different organizations pick two different rows. #### When Majority Adds Something The last row of the third table is this lesson's structural finding: **within a single family, the majority decision is identical to picking a single engine**. Requiring two of the six signature engines to flag an item gives the exact same result as listening to the signature family's fifth engine alone — the same true positives, the same false positives. The code checks this at all six thresholds and prints `yes` for both families. The reason is this: counting votes among nested engines is nothing more than picking a point on the sensitivity axis. The vote of twelve engines looking at the same method is the vote of a single engine. The mixed set behaves differently, but its gain is conditional. At 92 percent recall, the mixed threshold of 3 is **worse** than the best single engine: it produces 9 false positives against the same 11 true positives, where the single engine produces 6. By contrast, the mixed threshold of 7 reaches a precision no single engine can reach — 86 percent against the best single engine's 65 percent. The gain comes from the two families being wrong on **different items**: two bad items that one family is blind to are flagged by the other family, and the false positives one family produces do not find a vote in the other family. The rule that follows from this: what should be counted when reading a multi-engine result is not the number of engines, it is the **number of different methods**. If six of twelve flags come from the same family, those six flags are one flag. #### The Item That Cannot Be Sent The fourth block gives the cost, and most of that cost is not technical. Only eight of the twenty-four items can have their file sent. Six items cannot be sent under any condition because they carry subscriber data; uploading a file to an outside service is **publishing** it, and it is irreversible. For the remaining items only a hash can be queried, but a hash query only answers if the item is known on the service side. The result: **ten items cannot be queried in any form**, and four of these are actually bad. This is another instance of the same shape as the scope limit from earlier lessons. There, a request could not be sent to an out-of-scope target; here, the privacy class cannot leave the organization. Both produce an unverifiable candidate, and both are written in the report with the line "the method could not be applied to this item." The trace column on the defense side is again zero in this lesson: queries do not go to target systems. The trace lands elsewhere. Each of the fourteen queries enters the service's own records, and someone able to watch those records learns which item was investigated and when. The second of the two trace types separated at the end of the previous lesson takes its most concrete form here: querying an item can tell whoever is using it that it is being examined, and that information cannot be taken back. #### Summary - As the majority threshold rises, recall falls from 92 percent to 8 percent; precision is not monotonic — it rises from 50 percent to 86 percent at the seventh threshold, then returns to 50 percent at the twelfth. No threshold raises both at once, and the choice is a business decision. - Two items are independent of the threshold: one is a bad item no engine flags, the other a clean item every engine flags. These two are an error floor that counting votes cannot fix. - Within a single family the majority decision is identical to picking a single engine; counting votes among nested engines is nothing more than picking a point on the sensitivity axis. - The mixed set's gain is conditional: at the same recall it produces more false positives than a single engine, but at a high threshold it reaches 86 percent precision that no single engine reaches. What should be counted is the number of different methods, not the number of engines. - Eight of twenty-four items can have their file sent; six cannot be sent under any condition because they carry subscriber data, and ten items cannot be queried in any form. Four of these are actually bad. - Records produced on target are zero; fourteen queries land in the service's own records and can tell the other side it is being examined. #### Next Step Five tool classes have now hit the same limit. Traffic fields left inference, the analyzer dropped to metadata in an encrypted flow, the intercepting proxy had to leave two candidates unresolved because of scope, the sandbox could not resolve two items, the reputation service could not even query ten items. Every time, the same gap remained: the last step of showing that a candidate is **really** open was never taken. The next lesson examines the tool class that takes up that last step — **exploitation frameworks** — only for validation purposes and within their limits. Three things are measured: what a framework proves, what it does not prove, and under what condition it damages the target. The second point especially gets emphasis, because a failed validation attempt does not prove the vulnerability **does not exist**, and losing this distinction is the report's most expensive mistake. ### Exploitation Frameworks Source: https://academia.sh/en/courses/penetration-testing/exploitation-frameworks Measuring an exploitation framework strictly by its validation limit: of 16 candidates 9 are actually open while only 3 turn into evidence, only one of four validation options has a negative result that shows the vulnerability's absence, two of the three candidates that come back negative are actually open, three candidates get wrongly closed if negative and weak results are counted as closure grounds, and two targets are never touched at all because of scope. Up to this point five tool classes have hit the same limit: every time, the last step that would show an item is really open was left undone. The **exploitation framework** is the tool class that takes that last step, and for exactly that reason it is used within the narrowest limits. This lesson does not describe how a framework is used; no tool, module, step sequence, command, or payload is written. Three things are measured: what a validation attempt **proves**, what it **does not prove**, and under what condition it damages the target. Pointing a framework at a target without written authorization and a defined scope is unauthorized access regardless of the outcome, and it carries legal consequences. #### Four Options, Two Separate Questions There is more than one way to validate a candidate, and these are separated by two independent properties. The first is **evidence strength**. Passive indicator matching — comparing configuration and version information against a known definition — does not touch the target and never produces conclusive evidence; it only strengthens a candidate. Side-effect-free validation deterministically tests whether the condition needed for the vulnerability to exist is met. State-changing and disruption-risk options produce conclusive evidence and in return leave a trace and risk on the target. The second, and the less-discussed one, is **the meaning of a negative result**. When a condition test comes back negative, it is known that the condition is not met; this is proof of the vulnerability's absence. When an attempt comes back negative, nothing is known. The attempt may have failed because of the target's state at that moment, its load, a middle layer's behavior, or timing. **A failed validation attempt does not prove the vulnerability does not exist.** Losing this distinction is the report's most expensive mistake. - **AT36.** The option catalog, candidate list, and hosting classes are a **model**; there is no real target, no real vulnerability identity, and no real tool. - **AT37.** The `open` field is the candidate's actual state; it is not known before validation and is written only so the model can compute the result. - **AT38.** In `conditional` options the result depends on the target's environment condition at that moment; if the condition is not met, the attempt comes back negative even on an open item. - **AT39.** The hosting class narrows the option set. `managed` is a service operated by a third party, and no attempt is in scope there. - **AT40.** The rules of engagement set the highest risk level; the selection is the option with the highest evidence strength among those allowed, and the lowest risk in case of a tie. - **AT41.** Duration, record, and alert counts were written by hand per option. ```js // cerceve/limit.mjs — this is a model of an exploitation framework's VALIDATION LIMIT. No tool, // module, step sequence, command, or payload is written; options are referred to only by their // type, risk level, evidence strength, and THE MEANING OF A NEGATIVE RESULT. // negative: 'absence' = a negative result shows the vulnerability's absence; 'inconclusive' = it shows nothing. // conditional = the result depends on the target's environment condition at that moment; if the // condition is not met, the attempt comes back negative even on an open item. A conditional // attempt's negative result shows NOTHING. export const OPTION = { 'indicator-match': { risk: 0, evidence: 'weak', negative: 'inconclusive', conditional: false, min: 5, record: 1, alert: 0 }, 'side-effect-free-validation': { risk: 1, evidence: 'conclusive', negative: 'absence', conditional: false, min: 15, record: 2, alert: 1 }, 'state-changing': { risk: 2, evidence: 'conclusive', negative: 'inconclusive', conditional: true, min: 30, record: 4, alert: 3 }, 'disruption-risk': { risk: 3, evidence: 'conclusive', negative: 'inconclusive', conditional: true, min: 45, record: 6, alert: 5 }, }; // Hosting class: up to which risk level an operation can be performed on the target. // managed = a service operated by a third party; no attempt is in scope there. export const HOSTING = { own: 3, shared: 1, managed: -1 }; export const ALLOWED_RISK = 2; // the highest risk level the rules of engagement allow // row: candidate | is it open | passive indicator | is the environment condition met | hosting | options export const CANDIDATE = ` c01 open yes no own indicator-match,side-effect-free-validation c02 open yes yes own indicator-match,state-changing c03 open no yes own indicator-match,side-effect-free-validation c04 open yes no own indicator-match,state-changing c05 open yes yes shared indicator-match,state-changing c06 open no no own indicator-match,state-changing c07 open yes yes managed indicator-match,side-effect-free-validation c08 open yes yes own indicator-match c09 closed yes yes own indicator-match,side-effect-free-validation c10 closed yes yes own indicator-match,state-changing c11 closed no yes own indicator-match,side-effect-free-validation c12 closed yes yes own indicator-match c13 closed yes yes shared indicator-match,side-effect-free-validation c14 closed no yes own indicator-match,disruption-risk c15 closed yes yes managed indicator-match,state-changing c16 open no yes own indicator-match,disruption-risk `.trim().split('\n').map((r) => { const [name, state, indicator, cond, hosting, opts] = r.trim().split(/\s+/); return { name, open: state === 'open', indicator: indicator === 'yes', condition: cond === 'yes', hosting, options: opts.split(',') }; }); export const allowed = (a, s) => OPTION[s].risk <= Math.min(ALLOWED_RISK, HOSTING[a.hosting]); // Selected option: among those allowed, the one with the highest evidence strength, and the lowest risk in case of a tie. export function selection(a) { const fit = a.options.filter((s) => allowed(a, s)); if (fit.length === 0) return null; return fit.sort((x, y) => { const g = (s) => (OPTION[s].evidence === 'conclusive' ? 1 : 0); return g(y) - g(x) || OPTION[x].risk - OPTION[y].risk; })[0]; } // Result: evidence | absence-proof | inconclusive-negative | strong-candidate | weak-candidate | unverifiable export function result(a) { const s = selection(a); if (!s) return { option: '-', result: 'unverifiable', min: 0, record: 0, alert: 0 }; const o = OPTION[s]; let d; if (o.evidence === 'conclusive') { if (a.open && (a.condition || !o.conditional)) d = 'evidence'; else if (a.open) d = 'inconclusive-negative'; // open, but the attempt came back negative else d = o.negative === 'absence' ? 'absence-proof' : 'inconclusive-negative'; } else d = a.indicator ? 'strong-candidate' : 'weak-candidate'; return { option: s, result: d, min: o.min, record: o.record, alert: o.alert }; } export function total() { const t = { evidence: 0, 'absence-proof': 0, 'inconclusive-negative': 0, 'strong-candidate': 0, 'weak-candidate': 0, unverifiable: 0, min: 0, record: 0, alert: 0, unprovenOpen: 0 }; for (const a of CANDIDATE) { const r = result(a); t[r.result] += 1; t.min += r.min; t.record += r.record; t.alert += r.alert; if (r.result !== 'evidence' && a.open) t.unprovenOpen += 1; } return t; } ``` #### Sixteen Candidates, Six Result Types ```js // cerceve/measure.mjs — sixteen candidates, what the framework proves and does not prove. import { OPTION, HOSTING, ALLOWED_RISK, CANDIDATE, selection, result, total } from './limit.mjs'; const y = (g, ...s) => console.log(s.map((v, i) => (g[i] < 0 ? String(v).padEnd(-g[i]) : String(v).padStart(g[i]))).join('').trimEnd()); console.log(`${CANDIDATE.length} candidates (${CANDIDATE.filter((a) => a.open).length} actually open), ` + `${Object.keys(OPTION).length} validation options, highest allowed risk ${ALLOWED_RISK}`); const A = [-28, 6, 6, 7, 7, 13, 14]; console.log('\n1. option types'); y(A, 'option', 'risk', 'min', 'record', 'alert', 'conditional', 'negative'); for (const [s, o] of Object.entries(OPTION)) y(A, s, o.risk, o.min, o.record, o.alert, o.conditional ? 'yes' : 'no', o.negative); const B = [-10, 8, 11, 11, 10, 30, 23]; console.log('\n2. option selected and result per candidate'); y(B, 'candidate', 'actual', 'indicator', 'condition', 'hosting', 'selected', 'result'); for (const a of CANDIDATE) { const r = result(a); y(B, a.name, a.open ? 'open' : 'closed', a.indicator ? 'yes' : 'no', a.condition ? 'yes' : 'no', a.hosting, r.option, r.result); } const t = total(); console.log('\n3. decision distribution produced by the framework'); for (const k of ['evidence', 'absence-proof', 'inconclusive-negative', 'strong-candidate', 'weak-candidate', 'unverifiable']) console.log(` ${k.padEnd(24)} ${String(t[k]).padStart(3)}`); console.log('\n4. what it does not prove'); const inconclusiveNeg = CANDIDATE.filter((a) => result(a).result === 'inconclusive-negative'); const wronglyClosed = CANDIDATE.filter((a) => ['inconclusive-negative', 'weak-candidate'].includes(result(a).result) && a.open); console.log(` options whose negative result counts as absence-proof ` + `${Object.values(OPTION).filter((o) => o.negative === 'absence').length} / ${Object.keys(OPTION).length}`); console.log(` candidates that came back negative but show nothing ${inconclusiveNeg.length}`); console.log(` of these, actually open ` + `${inconclusiveNeg.filter((a) => a.open).length}`); console.log(` candidates wrongly closed if negative and weak results are read as 'not open' ` + `${wronglyClosed.length}`); console.log(` candidates that are actually open but cannot be proven ${t.unprovenOpen} / ` + `${CANDIDATE.filter((a) => a.open).length}`); console.log('\n5. cost and trace'); console.log(` total validation time ${t.min} min`); console.log(` records produced on target ${t.record}`); console.log(` alerts produced on target ${t.alert}`); console.log(` targets never touched because they are out of scope ` + `${CANDIDATE.filter((a) => a.hosting === 'managed').length}`); console.log(` targets whose risk limit was lowered in a shared environment ` + `${CANDIDATE.filter((a) => a.hosting === 'shared').length}`); ``` ``` 16 candidates (9 actually open), 4 validation options, highest allowed risk 2 1. option types option risk min record alert conditional negative indicator-match 0 5 1 0 no inconclusive side-effect-free-validation 1 15 2 1 no absence state-changing 2 30 4 3 yes inconclusive disruption-risk 3 45 6 5 yes inconclusive 2. option selected and result per candidate candidate actual indicator condition hosting selected result c01 open yes no own side-effect-free-validation evidence c02 open yes yes own state-changing evidence c03 open no yes own side-effect-free-validation evidence c04 open yes no own state-changing inconclusive-negative c05 open yes yes shared indicator-match strong-candidate c06 open no no own state-changing inconclusive-negative c07 open yes yes managed - unverifiable c08 open yes yes own indicator-match strong-candidate c09 closed yes yes own side-effect-free-validation absence-proof c10 closed yes yes own state-changing inconclusive-negative c11 closed no yes own side-effect-free-validation absence-proof c12 closed yes yes own indicator-match strong-candidate c13 closed yes yes shared side-effect-free-validation absence-proof c14 closed no yes own indicator-match weak-candidate c15 closed yes yes managed - unverifiable c16 open no yes own indicator-match weak-candidate 3. decision distribution produced by the framework evidence 3 absence-proof 3 inconclusive-negative 3 strong-candidate 3 weak-candidate 2 unverifiable 2 4. what it does not prove options whose negative result counts as absence-proof 1 / 4 candidates that came back negative but show nothing 3 of these, actually open 2 candidates wrongly closed if negative and weak results are read as 'not open' 3 candidates that are actually open but cannot be proven 6 / 9 5. cost and trace total validation time 220 min records produced on target 31 alerts produced on target 17 targets never touched because they are out of scope 2 targets whose risk limit was lowered in a shared environment 2 ``` #### What It Proves The third block counts six result types, and the first row is smaller than expected: only **three** of sixteen candidates turn into evidence. Nine candidates are actually open; the framework proves a third of them. Of the ones that turn into evidence, two were determined by side-effect-free validation, one by a state-changing attempt. This distribution points a direction: the cheapest way to produce conclusive evidence is not an attempt, it is a **condition test**. For fifteen minutes, two records, and one alert it produces both evidence and — if the candidate is not open — proof of absence. A state-changing attempt gives the same certainty for thirty minutes, four records, and three alerts, and when it comes back negative it says nothing. The second row counts something rarely discussed in the course: **three proofs of absence**. These three candidates drop off the list for certain; they do not go to the remediation team and are not looked at again. They do the same job as the false positives eliminated in earlier lessons; the difference is that the grounds are deterministic. #### What It Does not Prove The fourth block is this lesson's actual subject. Of four validation options, only **one** can count its negative result as proof of absence. In the remaining three, a negative result carries no information. Three candidates came back negative, and **two are actually open**. For `c04` and `c06`, the environment condition needed was not met during the attempt; the attempt came back negative, the vulnerability is still sitting there. If these go into the report as "vulnerability not found," two real vulnerabilities get closed. The correct line reads "validation was attempted, came back negative, result inconclusive," and it keeps candidate status in the remediation priority list. The second form of the same mistake is in weak results. `c16` is actually open, its passive indicator does not match, and the conclusive validation option cannot be used because it carries disruption risk. The result is "weak candidate." If negative and weak results are all read as "not open," **three candidates get wrongly closed**. In total, **six** of nine open candidates cannot be proven. The reasons are different and are written separately in the report: two had an unmet attempt condition, two had the risk limit close off the conclusive option, one had no conclusive option to begin with, one had an out-of-scope target. None of these is "no vulnerability." #### Damage Risk and Scope The fifth block gives the cost: 220 minutes, 31 records, 17 alerts. The alert count is higher than in any earlier lesson, and the reason is obvious — every option that produces conclusive evidence touches the target. Two targets were never touched at all. Two candidates running on a service operated by a third party were tested by no option because they are out of scope; one of them is actually open, and it is reported not as a finding but as **risk left outside the assessment**. On the two targets in the shared environment, the risk limit was lowered: every option that could affect other tenants in the same environment was closed off; one had only indicator matching left available, the other still allowed side-effect-free validation. There is one more risk item not shown in the table, and it explains why the scope check is run on a schedule before every attempt. An attempt's target comes by hand or from a list; the list may be stale, a record may have moved, or an address may have been reassigned. An attempt pointed at the wrong target is unauthorized access regardless of intent, and it invalidates the authorization given. Verifying a target's identity right before the attempt is therefore not a habit but an audit step. The disruption-risk option was not used on any candidate because the rules of engagement did not allow it. This is the normal decision in an environment like a measurement network: taking a meter collector offline stops field teams' work and shifts the billing cycle. When the cost of evidence is greater than the value of what it proves, the attempt is not made — the candidate stays a candidate, and the reason is written down. #### Summary - Nine of sixteen candidates are actually open; the framework proves only three of them. The cheapest way to conclusive evidence is not an attempt, it is a deterministic condition test. - Only one of four validation options has a negative result that shows the vulnerability's absence; in the remaining three, a negative result carries no information. - Two of the three candidates that come back negative are actually open; if negative and weak results are read as "not open," three candidates get wrongly closed. - The grounds for the six unprovable open candidates are written separately: condition not met, risk limit closed it off, no conclusive option, target out of scope. None of these is "no vulnerability." - Three proofs of absence drop off the list for certain and do the same job as eliminated false positives. - Validation produces 220 minutes, 31 records, and 17 alerts; two targets are never touched because of scope, two have their risk limit lowered, and the disruption-risk option is used on no candidate. #### Next Step Everything measured in this lesson — how many requests, which risk class, which option, how many alerts — must have been learned somewhere. A step tried for the first time during validation puts both the target and the assessment itself at risk. A real institutional network is not the place to practice. The next lesson takes up legal practice labs in terms of scope, and closes out the topic. What is measured is not what these environments teach, it is **what they do not teach**: the skill list produced by seven lessons is drawn up, each environment's coverage is counted, and the items no environment covers are written down separately. How many items the gap between a real assessment and a lab comes down to is this topic's final number. ### Test Distributions and Labs Source: https://academia.sh/en/courses/penetration-testing/test-distributions-and-labs Measuring legal practice labs in terms of scope: 13 of the 18 skill items produced by the topic's six lessons are covered by the union of three lab types, no single type exceeds 10, a security-focused test distribution covers zero skills on its own, and the five uncovered items stem from the absence of a real record and real risk. The previous lesson measured validation attempts by risk class, evidence strength, and the meaning of a negative result. All of these measures must have been learned somewhere; a step tried for the first time during validation puts both the target and the assessment itself at risk. A real institutional network is not the place to practice. An attempt pointed at someone else's system without written permission is unauthorized access even if done for learning, and it carries legal consequences. This lesson measures legal practice labs in terms of **scope**, and what it asks is not what they teach, it is **what they do not teach**. The skill items produced by the topic's six lessons are listed, how many each environment type covers is counted, and the items none of them cover are written down separately. #### Scope Is a Set Difference Four environment types are considered. A **self-built isolated lab** is an environment set up on your own hardware that does not reach an outside network: you build the topology yourself, so you also choose where capture happens. A **prebuilt vulnerable image set** consists of pre-prepared targets; setup is cheap, the topology is given. A **scenario-based practice lab** is one run by a provider, with scope and rules defined in advance; the terms of use are a contract, and reading it is part of the exercise. A **security-focused test distribution** is a toolset; it shortens setup time. The fifth row is the measure's reference: an **authorized real assessment**. By definition it covers every item, requires no setup, and requires a written contract and a defined scope. - **AT42.** The skill list, environment types, and scope sets are a **model**; no real platform, distribution, or provider name is written. - **AT43.** The skill items were drawn from this topic's six lessons; the four items sourced from `process` were not measured in any lesson and were put on the list as a limit. - **AT44.** An environment covering a skill means that skill can be worked through there **along with its result**; being able to run the tool is not enough. - **AT45.** Setup times were written by hand per environment type. - **AT46.** The union of labs covers three practice types; the real assessment does not join the union, it is the comparison side. - **AT47.** The real record produced and the real risk carried in a lab are zero. ```js // lab/scope.mjs — this is a SCOPE model of legal practice labs. No real platform, // distribution, or product name is written; environments are referred to only by type. // The skill list was drawn from this topic's six lessons; the ones sourced from 'process' // were not measured in any lesson and were put here as a limit. export const SKILL = ` b01 capture-point-selection 01 b02 field-class-separation 01 b03 analyzer-label-validation 02 b04 metadata-extraction-limit 02 b05 validation-plan-building 03 b06 request-risk-class-assignment 03 b07 scope-check-running 03 b08 sandbox-prep 04 b09 hidden-behavior-reading 04 b10 majority-threshold-selection 05 b11 privacy-class-decision 05 b12 validation-option-selection 06 b13 negative-result-interpretation 06 b14 target-identity-verification 06 b15 scope-negotiation process b16 client-communication process b17 production-impact-carrying process b18 finding-reporting process `.trim().split('\n').map((r) => { const [name, skill, source] = r.trim().split(/\s+/); return { name, skill, source }; }); // Environment type: which skills it covers, how many hours to set up, what its legal condition is. // 'security-focused-test-distribution' is a toolset, not a practice lab: it shortens // setup time and covers no skill on its own. export const ENVIRONMENT = { 'self-built-isolated-lab': { setupHours: 24, legal: 'your-own-hardware', covers: ['b01', 'b02', 'b03', 'b04', 'b05', 'b06', 'b08', 'b09', 'b12', 'b13'] }, 'prebuilt-vulnerable-image-set': { setupHours: 3, legal: 'your-own-hardware', covers: ['b03', 'b04', 'b05', 'b06', 'b12', 'b13'] }, 'scenario-based-practice-environment': { setupHours: 0, legal: 'provider-terms-of-use', covers: ['b03', 'b04', 'b05', 'b06', 'b07', 'b09', 'b10', 'b12', 'b13', 'b14'] }, 'security-focused-test-distribution': { setupHours: 1, legal: 'your-own-hardware', covers: [] }, 'authorized-real-assessment': { setupHours: 0, legal: 'written-contract-and-scope', covers: SKILL.map((b) => b.name) }, }; export const LAB = Object.keys(ENVIRONMENT).filter((o) => o !== 'authorized-real-assessment'); export function union(environments) { const k = new Set(); for (const o of environments) for (const b of ENVIRONMENT[o].covers) k.add(b); return k; } export function perLesson(covered) { const m = new Map(); for (const b of SKILL) { if (!m.has(b.source)) m.set(b.source, { total: 0, covered: 0 }); const s = m.get(b.source); s.total += 1; if (covered.has(b.name)) s.covered += 1; } return m; } ``` #### Eighteen Skills, Four Environments ```js // lab/measure.mjs — four environment types and a real assessment, across eighteen skills. import { SKILL, ENVIRONMENT, LAB, union, perLesson } from './scope.mjs'; const y = (g, ...s) => console.log(s.map((v, i) => (g[i] < 0 ? String(v).padEnd(-g[i]) : String(v).padStart(g[i]))).join('').trimEnd()); console.log(`${SKILL.length} skill items, ${Object.keys(ENVIRONMENT).length} environment types; ` + `skills were drawn from this topic's six lessons and from process`); const A = [-38, 10, 11, 28]; console.log('\n1. coverage per environment'); y(A, 'environment', 'covered', 'setup h', 'legal condition'); for (const [o, v] of Object.entries(ENVIRONMENT)) y(A, o, `${v.covers.length}/${SKILL.length}`, v.setupHours, v.legal); const lab = union(LAB); const open = SKILL.filter((b) => !lab.has(b.name)); const B = [-10, 9, 11, 12]; console.log('\n2. coverage per source lesson (union of three lab types)'); y(B, 'source', 'skill', 'covered', 'not covered'); for (const [k, s] of perLesson(lab)) y(B, k, s.total, s.covered, s.total - s.covered); const C = [-8, -32, 8, 17]; console.log('\n3. skill covered by no lab'); y(C, 'item', 'skill', 'source', 'where it exists'); for (const b of open) y(C, b.name, b.skill, b.source, 'real only'); console.log('\n4. lab vs. real assessment difference'); console.log(` union of three lab types ${lab.size} / ${SKILL.length}`); console.log(` difference ${open.length} items`); console.log(` broadest single lab type ` + `${Math.max(...LAB.map((o) => ENVIRONMENT[o].covers.length))} / ${SKILL.length}`); console.log(` distribution's coverage on its own ${ENVIRONMENT['security-focused-test-distribution'].covers.length}`); console.log(` total setup time (four environments) ` + `${LAB.reduce((t, o) => t + ENVIRONMENT[o].setupHours, 0)} hours`); console.log('\n5. trace and result'); console.log(` real record produced in the lab 0`); console.log(` real risk carried in the lab 0`); console.log(` authorization condition in a real assessment ` + `${ENVIRONMENT['authorized-real-assessment'].legal}`); console.log(` source of the ${open.length} uncovered items: ` + `${[...new Set(open.map((b) => b.source))].join(', ')}`); ``` ``` 18 skill items, 5 environment types; skills were drawn from this topic's six lessons and from process 1. coverage per environment environment covered setup h legal condition self-built-isolated-lab 10/18 24 your-own-hardware prebuilt-vulnerable-image-set 6/18 3 your-own-hardware scenario-based-practice-environment 10/18 0 provider-terms-of-use security-focused-test-distribution 0/18 1 your-own-hardware authorized-real-assessment 18/18 0 written-contract-and-scope 2. coverage per source lesson (union of three lab types) source skill covered not covered 01 2 2 0 02 2 2 0 03 3 3 0 04 2 2 0 05 2 1 1 06 3 3 0 process 4 0 4 3. skill covered by no lab item skill source where it exists b11 privacy-class-decision 05 real only b15 scope-negotiation process real only b16 client-communication process real only b17 production-impact-carrying process real only b18 finding-reporting process real only 4. lab vs. real assessment difference union of three lab types 13 / 18 difference 5 items broadest single lab type 10 / 18 distribution's coverage on its own 0 total setup time (four environments) 28 hours 5. trace and result real record produced in the lab 0 real risk carried in the lab 0 authorization condition in a real assessment written-contract-and-scope source of the 5 uncovered items: 05, process ``` #### A Distribution Is Not an Environment The first table's clearest row is its fourth: a security-focused test distribution covers **zero** of eighteen skill items. This is not a shortfall, it is a matter of definition. A distribution is a toolset; it cuts the time to install the tools down to an hour and stops there. Having installed the tools is not a skill; the skill is being able to separate the candidate a tool produces from evidence, and that requires a target, a scope, and a result. The remaining three rows show an unexpected relationship between setup and coverage. Your own lab, which needs twenty-four hours of setup, covers 10 items; the scenario-based environment, which needs zero setup, also covers 10 items. But the items they cover are not the same. In your own lab, because you build the topology yourself, **capture-point selection** and **field-class separation** can be worked through; in a prebuilt scenario the topology is given, and those two items drop out. In return, because a scenario carries a scope definition, **scope-check running**, **target-identity verification**, and **majority-threshold selection** can be worked through; your own lab has no concept of scope, because everything in it is yours. When the two are used together, the union rises to 13 items. The prebuilt vulnerable image set adds no new item to the union; its contribution is not coverage, it is cheapening repetition through a three-hour setup. The total setup time of twenty-eight hours is the combined cost of the three lab types and the distribution, and it is a one-time cost. It is worth saying here that the coverage count does not measure a skill. The table says an item can be worked through, not how well it was worked through. Building a validation plan is an item covered in all three environments; having built one plan is not the same as having built twenty and having seen which ones got shorter. This is exactly where the prebuilt image set's contribution lies: it adds nothing to the coverage column, it cheapens repetition. Coverage is a precondition, not sufficiency. The second table groups skills by source lesson and shows that almost all of the technical side can be worked through in a lab: in five of six lessons, there is no uncovered item. The one exception is the fifth lesson, and even there the uncovered item is not technical. This distribution confirms that the lab's limit has nothing to do with tool capability. #### The Five Items the Lab Cannot Teach The third table is this lesson's result. Five items are covered by no lab, and four of them come straight from process: scope negotiation, client communication, production-impact carrying, finding reporting. The fifth comes from a technical lesson, and stands out for exactly that reason: **privacy-class decision**. A lab has no subscriber data. Deciding not to send a file outside is a free choice there; what makes the decision hard is what happens if you send it, and in a lab, nothing happens. The same structure shows up in the other four items. Scope negotiation only means something when there is an organization on the other side. Carrying production impact is only learned when there is a service that actually goes down. Reporting a finding is only learned when someone reading the report has their budget shift. The fifth block gives this in a single line: the real record produced in the lab is zero, the real risk carried is zero. **What the lab cannot teach is the reason it is a lab.** Zero risk makes the exercise legal and repeatable; that same zero leaves consequential decisions off the list. The practical conclusion here is not about environment choice. Even if all three lab types are used, five items stay open, and these items come not from the next environment but from a **process**. The authorized real assessment row's coverage is 18, but its condition is a written contract and a defined scope; doing the same work without that condition is not practice, it is a crime. #### What the Lab Gives on Its Own The difference list should not be read one-sidedly. The lab has an advantage the real assessment does not, and it connects directly to this topic's measurement axis: in a lab, **the actual state is known**. Every model in this topic carried an `actual` field — whether the item was open, whether it was bad. The model carried that field only so it could compute the result; a real assessment has no such field. When a candidate is not proven, it is unknown whether the vulnerability exists and just could not be shown, or never existed at all. In a lab, the target's state is known in advance, so it is possible to **measure** your own false-positive and false-negative rate. This is the only measurable form of practice. How many of ten targets got the right decision, how many candidates got needlessly validated, which step got skipped how many times — all countable in a lab; in a real assessment it is only understood years later, when an incident is read back retroactively. The 13-item section of the coverage table is therefore worked through not as "less" but as "measurable." A rule of use follows from this. The lab's job is not to see a technique, it is to learn your own error rate at that technique. Running the same target set twice and reaching the same decision both times teaches nothing; changing the target set and comparing the decision distribution does. #### Legal Limit and Terms of Use The three lab types do not share a legal condition. In an environment set up on your own hardware, the limit is the limit of your own hardware; even a single outbound connection crosses it, and that is the reason isolation is built. In a scenario-based environment the limit is drawn by the provider's terms of use: which targets can be touched, which techniques are forbidden, and how a found vulnerability is to be reported are written there. Reading that text is part of the exercise itself — because its counterpart in a real assessment is the rules of engagement, and not reading it produces the same consequence. Failing to build isolation in your own lab is not just a legal problem; it also breaks the measurement. The fourth lesson showed that an environment hint cannot be hidden under any configuration, because hiding it required the item under examination to reach a real target. The same limit stands the same way in a practice lab: letting one item reach outside removes both the isolation and the legal ground at the same time. Controlling the lab's outbound connection is part of the exercise itself, and its counterpart in a real assessment is the scope document. This draws the topic's seven lessons together into a single line. Tools produced candidates, validation steps turned some of those candidates into evidence, every step left a trace on the defense side, and at every stage a scope limit wrote down how many items it left off the list. The scope limit behaved the same way in all seven lessons: it measured not a tool's capability, but the **permission** to use that capability. #### Summary - Thirteen of eighteen skill items are covered by the union of three lab types; no single type exceeds 10, and the difference is five items. - A security-focused test distribution covers zero skills on its own: a toolset shortens setup time, it does not substitute for a practice lab. - Your own lab and the scenario-based environment cover the same number of items but different ones: the first gives you the topology, the second gives you a scope definition. - Four of the five uncovered items come from process, one from a privacy decision; what they share is requiring a decision with a consequence. - Real record and real risk are zero in a lab; that zero makes the exercise legal and repeatable, and the same zero leaves five items off the list. - The legal limit varies by environment type: the limit of your own hardware, or a provider's terms of use. Reading the terms of use is the counterpart of the rules of engagement in a real assessment. #### Next Step This topic counted what seven tool classes produce and what they cannot prove. Packet capture split claims into three classes, the protocol analyzer dropped to metadata in an encrypted flow, the intercepting proxy measured validation in terms of request count and risk class, the sandbox counted suppressed behavior, the reputation service showed the limit of the majority decision, the exploitation framework wrote down what it does not prove, and this lesson measured where all of it can be learned, in terms of scope. The same column stayed missing in every lesson. The scope limit kicked in again and again — the request to an out-of-scope target was held at zero, two flows were left unresolved, ten items were never queried, two targets were never touched — but **who** drew that limit, **with what authority**, and **with what document** was never written down. Nor was it written down in what form each measured permission, each alert, and each piece of evidence is to be reported. The next topic, **Testing Process**, fills this gap: turning the rules of engagement into measurable items, the difference between black-box, gray-box, and white-box assessments, eliminating false positives, expressing a finding in terms of business impact, writing reproduction steps, and the timeline of responsible disclosure. ## Testing Process The process of an authorized assessment: turning the scope definition into a programmatic audit and catching out-of-scope items, the candidate gap produced by black-, gray- and white-box assessments, running the candidate list through validation rules and eliminating false positives, scoring the same finding in two business contexts, measuring the report in terms of reproducibility, and calculating responsible disclosure as time and steps. ### Rules of Engagement Source: https://academia.sh/en/courses/penetration-testing/rules-of-engagement A written scope definition for a 64-item fictional inventory is turned into a programmatic audit: 20 items come out in scope, 11 on the border, 33 out of scope. Through five shared front ends, 29 out-of-scope and borderline items can be reached by mistake; 11 of them are banned by name in the rules of engagement. The raw window exceeds the planned work by 4 hours, while subtracting the meter-reading interval leaves it 6 hours short; two of the four thresholds carry no on-call coverage, and for one of them the actual notification time comes to one and a half times what the contract states. The previous topic showed what tools produce and what they cannot prove: the flow extracted from a packet set, the request an intercepting proxy altered, the behavior a sandbox could observe, the majority verdict of reputation services. All of them carried the same silent assumption — that there was a right to run those tools against the target they ran on. Where that right came from, where it ended, and who measured where it ended was never written down. **Rules of engagement** are the written form of that right: they define what can be touched, when it can be touched, and who is notified within how much time when something goes wrong. But a written sentence is not an audit. The sentence "the subscriber portal is in scope, the branch network is out of scope" has to turn into a decision for every single item once there is an actual target list, and this lesson runs that turn. **Touching an out-of-scope target voids the authorization: from that moment on, the action taken stops being written permission and becomes unauthorized access — it carries legal consequences and it makes the entire assessment unusable.** - **TP1.** The regional measurement network inventory has 64 items; it is spread across four units, three environments and eight front ends. Both the inventory and the scope definition are **fictional** — there is no real target. - **TP2.** The scope definition consists of three allow criteria (network block, unit, environment) and two exclusion criteria; six allow entries and three exclusion entries in total. - **TP3.** The audit produces three decisions: **in scope**, **borderline**, **out of scope**. Borderline means either that only part of the criteria matched, or that the item falls under both an allow rule and an exclusion rule; these items are set aside and cannot be touched without separate written approval. - **TP4.** A **front end** is a shared entry point that publishes multiple items from the same point. A request sent to a front end can reach every item behind that front end. - **TP5.** The test window is five nights, eight hours a night, two people. One hour of the night is the meter-reading interval, and no item is touched during that interval. - **TP6.** The communication protocol defines four thresholds; the example incident occurs at 02:30 at night, and the workday starts at 09:00. All durations are fictional. #### Scope Definition as an Audit ```js // scope.mjs — the model that turns the scope definition into a programmatic audit. // THIS IS A MODEL; both the inventory and the scope are fictional, there is no real target. export const SEED = 20260806; const makeRng = (t) => { let s = t >>> 0; return () => (s = (s * 1664525 + 1013904223) >>> 0) / 4294967296; }; const r = makeRng(SEED); // Each item belongs to a unit, an environment, an internal network block and a front end. const UNIT = ["subscriber-portal", "field-app", "branch-network", "meter-collector"]; const ENVIRONMENT = ["production", "staging", "development"]; const POOL = { // which unit uses which front end "subscriber-portal": ["front-a1", "front-a2", "front-shared-1"], "field-app": ["front-s1", "front-shared-1", "front-shared-2"], "branch-network": ["front-b1", "front-shared-2"], "meter-collector": ["front-t1", "front-t2", "front-shared-2"], }; export const TARGET = Array.from({ length: 64 }, (_, i) => { const unit = UNIT[Math.floor(r() * 4)], pool = POOL[unit]; return { name: `${unit}-${String(i + 1).padStart(2, "0")}`, unit, environment: ENVIRONMENT[Math.floor(r() * 3)], block: `10.${20 + Math.floor(r() * 4)}.${Math.floor(r() * 6)}.0/24`, front: pool[Math.floor(r() * pool.length)] }; }); // The scope definition written in the rules of engagement: three allow criteria, two exclusion criteria. export const SCOPE = { allowedBlock: ["10.20.", "10.21."], allowedUnit: ["subscriber-portal", "field-app"], allowedEnvironment: ["production", "staging"], excludedName: ["meter-collector", "branch-network"], excludedEnvironment: ["development"], }; // Audit: the three criteria are checked separately. All match -> in scope, none match -> // out of scope, some match or it collides with an exclusion -> BORDERLINE. export function decide(h, k = SCOPE) { const criteria = [k.allowedBlock.some((b) => h.block.startsWith(b)), k.allowedUnit.includes(h.unit), k.allowedEnvironment.includes(h.environment)]; const excluded = k.excludedName.some((a) => h.name.startsWith(a)) || k.excludedEnvironment.includes(h.environment); const matched = criteria.filter(Boolean).length; if (excluded) return matched === 3 ? "borderline" : "out-of-scope"; return matched === 3 ? "in-scope" : matched === 0 ? "out-of-scope" : "borderline"; } export const DECISION = new Map(TARGET.map((h) => [h.name, decide(h)])); // Shared front end: if an out-of-scope item uses the same front end as an in-scope // item, a request sent to that front end reaches the out-of-scope item too. export function sharedRisk() { const fronts = new Map(); for (const h of TARGET) fronts.set(h.front, [...(fronts.get(h.front) ?? []), h]); const risky = []; for (const [name, list] of fronts) { const d = (k) => list.filter((h) => DECISION.get(h.name) === k).length; const banned = list.filter((h) => SCOPE.excludedName.some((a) => h.name.startsWith(a))).length; if (d("in-scope") && list.length > d("in-scope")) { risky.push({ front: name, inScope: d("in-scope"), outOfScope: d("out-of-scope"), borderline: d("borderline"), banned }); } } return risky.sort((a, b) => b.outOfScope - a.outOfScope || a.front.localeCompare(b.front)); } if (import.meta.url === `file://${process.argv[1]}`) { const gen = (B) => (h) => h.map((v, j) => (j ? String(v).padStart(B[j]) : String(v).padEnd(B[0]))).join(""); const yz = gen([19, 10, 12, 14, 7]); const count = (d) => TARGET.filter((h) => DECISION.get(h.name) === d).length; console.log(`model (seed ${SEED}): ${TARGET.length} targets, ${UNIT.length} units, ` + `${new Set(TARGET.map((h) => h.front)).size} front ends`); console.log("\n" + yz(["unit", "in scope", "borderline", "out of scope", "total"])); for (const u of UNIT) { const l = TARGET.filter((h) => h.unit === u); const c = (d) => l.filter((h) => DECISION.get(h.name) === d).length; console.log(yz([u, c("in-scope"), c("borderline"), c("out-of-scope"), l.length])); } console.log(yz(["TOTAL", count("in-scope"), count("borderline"), count("out-of-scope"), TARGET.length])); const risky = sharedRisk(), yz2 = gen([16, 10, 14, 12, 16]); console.log("\n" + yz2(["front end", "in scope", "out of scope", "borderline", "banned by name"])); for (const x of risky) console.log(yz2([x.front, x.inScope, x.outOfScope, x.borderline, x.banned])); const t = (field) => risky.reduce((a, x) => a + x[field], 0); console.log(`\nshared front ends ${risky.length}; a request sent to these front ends ` + `can also reach ${t("outOfScope") + t("borderline")} out-of-scope and borderline items, ` + `${t("banned")} of which are banned by name in the rules of engagement`); } ``` ``` model (seed 20260806): 64 targets, 4 units, 8 front ends unit in scope borderline out of scope total subscriber-portal 11 5 1 17 field-app 9 6 6 21 branch-network 0 0 9 9 meter-collector 0 0 17 17 TOTAL 20 11 33 64 front end in scope out of scope borderline banned by name front-shared-2 1 14 2 11 front-shared-1 7 3 5 0 front-a1 3 1 0 0 front-a2 5 0 3 0 front-s1 4 0 1 0 shared front ends 5; a request sent to these front ends can also reach 29 out-of-scope and borderline items, 11 of which are banned by name in the rules of engagement ``` The table above writes out what a two-sentence scope definition corresponds to across 64 items. The written text takes two units in and leaves two units out; the number, however, spreads across three columns. **20 items are in scope, 33 are out of scope, 11 are borderline.** All 11 borderline items come from allowed units: the unit criterion matches, but the network block or environment criterion does not. The written definition produces no decision for these items, because whoever wrote it did not account for a unit having items in three different blocks and three different environments. The measurement reduces this gap to a name: the number of items whose scope is uncertain is a little more than half the number of items in scope. The third decision being a separate column is not a detail — it is the actual information the audit carries. If a borderline item is counted as "in scope," the authorization limit is exceeded; if it is counted as "out of scope," the assessment keeps its own scope narrower than it should and items that genuinely need testing are never seen. Of the two mistakes, the first carries legal consequences, the second wastes the organization's money. The correct action is to flag the item with a third decision and send it back to whoever wrote the definition. This is also where the value of a scope audit is measured: not how many items it auto-approves, but **how many items it sends back to a human**. The table below gives a harsher result. Five of the eight front ends collect in-scope and out-of-scope items at the same point. The worst is `front-shared-2`: behind it there is a **single** in-scope item, against which stand 14 out-of-scope and 2 borderline items, and 11 of those 14 are items banned by unit name in the rules of engagement. For that one item, a request sent to that front end also reaches the 11 items banned by name. In total, across five front ends, **29 items** are reachable by mistake. This shows why writing a scope definition as a list of items is not enough. Scope intersects not with the item but with the **access point**; how many items stand behind a single point is a decision made by the infrastructure, not the inventory. The correct action here is to exclude the single item behind `front-shared-2` from the assessment: the value of that one item is immeasurably small next to the risk of touching 11 banned items. The 11 borderline items, meanwhile, are tied to a separate written approval, and no step runs before that approval arrives. #### Time Window and Communication Protocol ```js // window.mjs — the time window and communication protocol counted as measurable items. import { TARGET, DECISION } from "./scope.mjs"; const inScopeCount = TARGET.filter((h) => DECISION.get(h.name) === "in-scope").length; const STEP = [["recon", 0.4], ["scan", 1.1], ["validation", 2.3]]; // hours/item, fictional const NIGHTS = 5, HOURS = 8, PEOPLE = 2, BLACKOUT = 1; // one hour a night is the meter-reading interval const gen = (B) => (h) => h.map((v, j) => (j ? String(v).padStart(B[j]) : String(v).padEnd(B[0]))).join(""); const y1 = gen([12, 7, 12, 13]); const planned = STEP.reduce((a, [, s]) => a + s * inScopeCount, 0); const raw = NIGHTS * HOURS * PEOPLE, net = NIGHTS * (HOURS - BLACKOUT) * PEOPLE; console.log(`${inScopeCount} in-scope items, ${STEP.length} step classes`); console.log(y1(["step", "items", "hours/item", "total hours"])); for (const [name, s] of STEP) { console.log(y1([name, inScopeCount, s.toFixed(1), (s * inScopeCount).toFixed(1)])); } console.log(y1(["PLANNED", "", "", planned.toFixed(1)])); console.log(`window: ${NIGHTS} nights x ${HOURS} hours x ${PEOPLE} people = ${raw} person-hours; ` + `subtracting the meter-reading interval (${BLACKOUT} hour/night) leaves ${net} person-hours`); console.log(`the raw window exceeds the planned work by ${(raw - planned).toFixed(1)} hours, ` + `the net window falls ${(planned - net).toFixed(1)} hours short`); // Communication protocol: contract duration, on-call coverage and parties for each threshold. const THRESHOLD = [ ["critical finding", 1, true, ["test team", "org security"], false], ["service outage", 0.25, true, ["test team", "org security", "system owner"], true], ["scope excursion", 4, false, ["test team", "org security", "system owner", "legal"], true], ["third-party data", 24, false, ["test team", "org security", "legal", "data owner", "regulator"], true], ]; const INCIDENT = 2.5, WORKDAY = 9; // incident hour and workday start const y2 = gen([19, 10, 9, 9, 9, 7]); console.log("\n" + y2(["threshold", "contract", "on-call", "actual", "parties", "stops"])); let contractTotal = 0, actualTotal = 0; for (const [name, contract, onCall, parties, stops] of THRESHOLD) { const actual = onCall ? contract : Math.max(contract, WORKDAY - INCIDENT); contractTotal += contract; actualTotal += actual; console.log(y2([name, `${contract} h`, onCall ? "yes" : "no", `${actual.toFixed(2)} h`, parties.length, stops ? "yes" : "no"])); } const allParties = new Set(THRESHOLD.flatMap(([, , , t]) => t)); console.log(`\n${THRESHOLD.length} thresholds, ${allParties.size} distinct parties, ` + `testing stops at ${THRESHOLD.filter((e) => e[4]).length} thresholds`); console.log(`for an incident occurring at ${INCIDENT.toFixed(2)} at night, the contract promises ` + `${contractTotal.toFixed(2)} hours total, actual comes to ${actualTotal.toFixed(2)} hours`); ``` ``` 20 in-scope items, 3 step classes step items hours/item total hours recon 20 0.4 8.0 scan 20 1.1 22.0 validation 20 2.3 46.0 PLANNED 76.0 window: 5 nights x 8 hours x 2 people = 80 person-hours; subtracting the meter-reading interval (1 hour/night) leaves 70 person-hours the raw window exceeds the planned work by 4.0 hours, the net window falls 6.0 hours short threshold contract on-call actual parties stops critical finding 1 h yes 1.00 h 2 no service outage 0.25 h yes 0.25 h 3 yes scope excursion 4 h no 6.50 h 4 yes third-party data 24 h no 24.00 h 5 yes 4 thresholds, 6 distinct parties, testing stops at 3 thresholds for an incident occurring at 2.50 at night, the contract promises 29.25 hours total, actual comes to 31.75 hours ``` The time window is enough on paper, and it is not enough in reality. Five nights, two people and eight hours a night give 80 person-hours; the planned work is 76 hours, so the window exceeds the work by four hours. Once the meter-reading interval is subtracted, the usable time drops to 70 hours and the same plan falls **six hours short**. The difference is not a management decision, it is arithmetic: the untouchable interval is one-eighth of the window, and the plan's share drops by that same proportion. The only way to reclaim time is to cut the number of steps, and since validation is the most expensive class at 46 hours, that is where the cut has to come from — which is the question the next lesson asks. The second table measures communication. Two of the four thresholds carry on-call coverage, and for those two the actual notification time equals what the contract states. For the remaining two, the other party can only be reached during working hours; for a suspected scope excursion arising at 02:30 at night, the contract states four hours while the actual time is 6.5 hours. This means the protocol's two slowest thresholds are also the thresholds that stop testing: two of the three test-stopping thresholds depend on working hours. The four thresholds together involve six distinct parties, and the threshold that involves the most parties is also the longest, because the protocol requires reaching all of them. The stop column also deserves its own reading. Testing stops at three thresholds, and the moment it stops is not a decision but an obligation: once a suspected scope excursion arises, no new step runs until that suspicion is cleared. This directly increases the hours lost from the window — if a test that stops at 02:30 at night waits until 09:00, the remaining 3.5 hours of that night amount to seven person-hours for two people, added on top of a plan that is already six hours short. The time and communication sections of the rules of engagement are not separate documents for this reason; one is the input to the other. The conclusion comes down to a single sentence: the rules of engagement produce three separate numbers — the number of items that can be touched, the number of usable hours, and how long notification actually takes. None of the three shows up in the written text; all three appear only once it is run against the inventory and the calendar. #### Summary - In a 64-item inventory, a two-sentence scope definition leaves 20 items in scope, 33 out of scope, and **11 items borderline** — undecided; all of the borderline items come from allowed units with a disallowed block or environment. - Five of the eight front ends collect in-scope and out-of-scope items at the same point; 29 items are reachable by mistake through these front ends, and 11 of them are banned by name in the rules of engagement. - `front-shared-2` carries the risk of 11 banned items for the sake of a single in-scope item behind it; the correct decision is to exclude that one item from the assessment. - In the raw calculation the window exceeds the planned work by 4 hours, but once the meter-reading interval is subtracted it falls 6 hours short; the cut has to come from the 46-hour validation class. - Two of the four communication thresholds carry no on-call coverage; a suspected scope excursion at 02:30 at night takes 4 hours by contract and 6.5 hours in practice, and involves four distinct parties; the four thresholds together reach six parties. #### Next Step Scope has been turned into numbers, the window has been calculated, and notification times have been written down. These three numbers determine what can be done, but not **how** it will be done. An assessment over the same 20 items can be run with no prior information at all, or with credentials and architecture documents supplied. The two paths do not spend the same time and do not produce the same number of candidates. The next lesson compares black-, gray- and white-box assessments in terms of candidate production: how many candidates each type produces on the same target, how many turn into evidence, and what the time difference between them costs. ### Test Types Source: https://academia.sh/en/courses/penetration-testing/test-types Black-, gray- and white-box assessments are run against the same 20 items and the same 41 fictional flaws. Black box produces 78 candidates and yields 6 evidenced findings (7.7%), gray box turns 63 candidates into 20, white box turns 60 candidates into 31; 10 flaws cannot be evidenced by any type because proving them requires an action the rules of engagement forbid. Time per evidenced finding drops from 8.82 hours to 1.87 hours, and the defense-side trace per evidenced finding drops from 78 to 2.9; 17 of white box's 31 evidenced findings are meaningful only from the inside. The previous lesson turned scope into numbers: 20 items in scope, 11 items on the border, a net window of 70 person-hours. That calculation wrote down what could be done, not how it would be done. The same 20 items can be assessed with no prior information at all, with credentials supplied, or with configuration and source access opened up. The three paths do not carry the same name and do not produce the same result. This lesson compares the three paths on a single target. **Black-box assessment** starts with no internal information at all, **gray box** receives credentials and limited documentation, **white box** sees the configuration and the source. The measure of comparison is not the number of findings; the number of findings varies from type to type in a misleading way. The measure is the gap between the **candidates** produced and the findings that turn into evidence. - **TP7.** The assessment runs on the 20 in-scope items from the previous lesson. Both the item set and the flaw set are **fictional**. - **TP8.** Each flaw carries two separate access levels: the level required to **see** it and the level required to **prove** it. The levels are ordered as `external`, `credentialed` and `internal`. - **TP9.** Some flaws have a proof level of `impact`: their evidence can only be obtained through an action that would affect the service. Because the rules of engagement forbid that action, **no type** can reach this level. - **TP10.** Each item carries a **noise base**: the number of signals counted as candidates before validation. The less internal information a type sees, the higher its noise multiplier. - **TP11.** Time is made up of two components: setup per item and validation per candidate. All durations are fictional and given in person-hours. - **TP12.** **Trace** is the number of records a step produces on the defense side. The trace multiplier per candidate depends on the type; a type that proceeds by reading documentation produces fewer records. #### Three Types on the Same Target ```js // types.mjs — candidate production of black, gray and white box assessment on the same target. // THIS IS A MODEL; both the item set and the flaw set are fictional. export const SEED = 20260811, ITEM = 20; const makeRng = (t) => { let s = t >>> 0; return () => (s = (s * 1664525 + 1013904223) >>> 0) / 4294967296; }; const r = makeRng(SEED); // Each flaw carries two separate access levels: the one required to SEE it and the one // required to PROVE it. The proof level can exceed the visibility level; this is where // the candidate-versus-evidence gap comes from. const LEVEL = ["external", "credentialed", "internal"]; export const FLAW = []; for (let k = 0; k < ITEM; k++) { const n = 1 + Math.floor(r() * 3); // 1-3 real flaws per item for (let j = 0; j < n; j++) { const g = Math.floor(r() * 3); // "impact": a flaw whose evidence can only be obtained through an action that would // affect the service. Because the rules of engagement forbid this, no type can reach it. const proof = r() < 0.15 ? "impact" : LEVEL[Math.min(2, g + (r() < 0.42 ? 1 : 0))]; FLAW.push({ item: k, visible: LEVEL[g], proof }); } } // Noise base per item: the number of signals counted as a candidate before validation. export const NOISE = Array.from({ length: ITEM }, () => 1 + Math.floor(r() * 5)); // Type definition: the levels it accesses, the noise multiplier, setup time per item and // validation time per candidate (hours), trace left on the defense side per candidate. export const TYPE = [ { name: "black box", access: ["external"], noise: 1.0, setup: 0.5, validation: 0.55, trace: 6.0 }, { name: "gray box", access: ["external", "credentialed"], noise: 0.6, setup: 1.1, validation: 0.40, trace: 3.5 }, { name: "white box", access: ["external", "credentialed", "internal"], noise: 0.3, setup: 2.0, validation: 0.30, trace: 1.5 }, ]; // A type is measured over the given item set; if no list is given, the full set is used. export function measure(t, items = [...Array(ITEM).keys()]) { const set = new Set(items); const flaws = FLAW.filter((a) => set.has(a.item)); const visible = flaws.filter((a) => t.access.includes(a.visible)); const evidenced = visible.filter((a) => t.access.includes(a.proof)); const falsePositive = items.reduce((s, k) => s + Math.round(NOISE[k] * t.noise), 0); const candidates = visible.length + falsePositive; const time = items.length * t.setup + candidates * t.validation; return { candidates, evidenced: evidenced.length, falsePositive, pending: visible.length - evidenced.length, missed: flaws.length - visible.length, time, trace: Math.round(candidates * t.trace) }; } if (import.meta.url === `file://${process.argv[1]}`) { const gen = (B) => (h) => h.map((v, j) => (j ? String(v).padStart(B[j]) : String(v).padEnd(B[0]))).join(""); const yz = gen([13, 7, 8, 9, 9, 9, 9, 11]); console.log(`model (seed ${SEED}): ${ITEM} in-scope items, ${FLAW.length} real flaws`); const d = (k) => FLAW.filter((a) => a.visible === k).length; console.log(`by visibility level: external ${d("external")}, credentialed ${d("credentialed")}, ` + `internal ${d("internal")}; in ${FLAW.filter((a) => a.visible !== a.proof).length} flaws the ` + `proof level exceeds the visibility level`); console.log("\n" + yz(["type", "cand.", "evid.", "false+", "pending", "missed", "hours", "evid./hour"])); for (const t of TYPE) { const o = measure(t); console.log(yz([t.name, o.candidates, o.evidenced, o.falsePositive, o.pending, o.missed, o.time.toFixed(1), (o.evidenced / o.time).toFixed(2)])); } const [a, b, c] = TYPE.map((t) => measure(t)); console.log(`\nevidence rate: black ${(100 * a.evidenced / a.candidates).toFixed(1)}%, gray ` + `${(100 * b.evidenced / b.candidates).toFixed(1)}%, white ${(100 * c.evidenced / c.candidates).toFixed(1)}%`); } ``` ``` model (seed 20260811): 20 in-scope items, 41 real flaws by visibility level: external 16, credentialed 11, internal 14; in 19 flaws the proof level exceeds the visibility level type cand. evid. false+ pending missed hours evid./hour black box 78 6 62 10 25 52.9 0.11 gray box 63 20 36 7 14 47.2 0.42 white box 60 31 19 10 0 58.0 0.53 evidence rate: black 7.7%, gray 31.7%, white 51.7% ``` The first column of the table is this course's most misleading number. **The type that produces the most candidates yields the fewest evidenced findings.** Black box leaves 78 candidates: 62 of them are false positives, 10 are items that are seen but cannot be proven, and what remains is 6 evidenced findings — 7.7 percent. On the same target, white box turns 60 candidates into 31 evidenced findings. The gap between the "78 findings" written on a report's cover and "60 findings" runs in the opposite direction from what the organization actually receives. The source of this gap is in the second line: in 19 of the 41 flaws, the proof level is higher than the visibility level. **Seeing** a flaw and **proving** it require separate access, and every signal the reconnaissance stage produces stands on the wrong side of this distinction. Black box can see all 16 flaws visible from outside, but it can only reach the evidence for 6 of them from outside; the remaining 10 stay pending. A pending candidate that goes into the report is an unvalidated claim; if it is left out, the organization never hears about it at all. Neither option is correct on its own, and what this lesson measures is exactly the size of this third column. Gray box's place is also legible in this table. With 63 candidates, 20 evidenced findings and 14 missed flaws, it sits between the two extremes, but its real contribution is not in the numbers: gray box answers the question of how far an account with compromised credentials can reach. Black box cannot ask this question at all, because it has no access to the credentialed level; white box can ask it, but it reads the answer from the configuration, whereas the question belongs to runtime. Gray box's 6 internal evidenced findings fall exactly into this class and mark the boundary of a credential-compromise scenario. White box's 10 pending items are a separate class. White box sees every level, so the 10 candidates it leaves pending are flaws whose **evidence can only be obtained through an action that would affect the service.** The rules of engagement forbid that action, and the moment the prohibition is lifted, what gets produced is not an assessment but an outage. These 10 items cannot be evidenced by any type; in the report they have to stand not as an "evidenced finding" but as a "candidate with its conditions written down." The missed column shows the opposite: black box never sees 25 real flaws at all, gray box misses 14 of them, white box misses none. #### The Cost of Access and the Meaning of Evidence ```js // compare.mjs — the trace types leave and the external reachability of evidence. import { TYPE, ITEM, FLAW, measure } from "./types.mjs"; const gen = (B) => (h) => h.map((v, j) => (j ? String(v).padStart(B[j]) : String(v).padEnd(B[0]))).join(""); const yz = gen([13, 8, 10, 13, 10, 10, 13]); const full = [...Array(ITEM).keys()]; // How many of the evidenced flaws are visible at an externally-reachable level: evidence an // attacker could reach without prior access is separated from evidence meaningful only from inside. const spread = (t) => { const evidenced = FLAW.filter((a) => t.access.includes(a.visible) && t.access.includes(a.proof)); return { external: evidenced.filter((a) => a.visible === "external").length, internal: evidenced.filter((a) => a.visible !== "external").length }; }; console.log(`${ITEM} items, ${FLAW.length} real flaws; trace is the record count on the defense side`); console.log(yz(["type", "evid.", "trace", "trace/evid.", "external", "internal", "hours/evid."])); for (const t of TYPE) { const o = measure(t), d = spread(t); console.log(yz([t.name, o.evidenced, o.trace, (o.trace / o.evidenced).toFixed(1), d.external, d.internal, (o.time / o.evidenced).toFixed(2)])); } // Mix: white box on the first m items, gray box on the remaining items. const [, gray, white] = TYPE; const mix = (m) => { const a = measure(white, full.slice(0, m)), b = measure(gray, full.slice(m)); return { candidates: a.candidates + b.candidates, evidenced: a.evidenced + b.evidenced, pending: a.pending + b.pending, missed: a.missed + b.missed, time: a.time + b.time, trace: a.trace + b.trace }; }; const y2 = gen([17, 8, 8, 8, 8, 9, 10]); console.log("\n" + y2(["mix white / gray", "cand.", "evid.", "pending", "missed", "hours", "trace"])); for (let m = 0; m <= ITEM; m += 4) { const k = mix(m); console.log(y2([`${m} / ${ITEM - m}`, k.candidates, k.evidenced, k.pending, k.missed, k.time.toFixed(1), k.trace])); } const s = mix(0), b20 = mix(ITEM); console.log(`\nmoving from gray box to white box adds ${(b20.time - s.time).toFixed(1)} hours, ` + `raises evidence from ${s.evidenced} to ${b20.evidenced}, and drops missed flaws from ${s.missed} to 0`); console.log(`the same move brings the trace down from ${s.trace} to ${b20.trace}; pending candidates ` + `go from ${s.pending} to ${b20.pending}, because white box sees more flaws`); ``` ``` 20 items, 41 real flaws; trace is the record count on the defense side type evid. trace trace/evid. external internal hours/evid. black box 6 468 78.0 6 0 8.82 gray box 20 221 11.1 14 6 2.36 white box 31 90 2.9 14 17 1.87 mix white / gray cand. evid. pending missed hours trace 0 / 20 63 20 7 14 47.2 221 4 / 16 64 24 7 10 49.9 199 8 / 12 62 26 7 8 51.6 169 12 / 8 63 30 7 4 54.2 145 16 / 4 61 31 8 2 55.8 114 20 / 0 60 31 10 0 58.0 90 moving from gray box to white box adds 10.8 hours, raises evidence from 20 to 31, and drops missed flaws from 14 to 0 the same move brings the trace down from 221 to 90; pending candidates go from 7 to 10, because white box sees more flaws ``` The time column corrects the previous lesson's rough plan. There, a single 76-hour plan covered 20 items with no distinction by type; once the type is distinguished, the same 20 items range between 47.2 and 58.0 hours, and **all three fit inside the 70-hour net window**. So the choice is not a time constraint. The gap opens up when time is measured per evidenced finding: one evidenced finding takes 8.82 hours in black box and 1.87 hours in white box. Black box's time does not go toward validation — it goes toward eliminating 62 false positives. The trace column writes down the blue-team side. Black box leaves 468 records on the defense side, white box leaves 90. The ratio between them is 5.2, and the reason is not only the candidate count: black box has to test every candidate over the network, while white box closes off part of its candidates through documentation. Trace per evidenced finding drops from 78 to 2.9. This is the correct unit for measuring how much an assessment occupies the defense: not the total record count, but **records per evidenced finding**. The last two columns show white box's limit. 17 of the 31 evidenced findings are meaningful only from the inside: it has been shown that they exist in the configuration, but not that an external request can reach them. All 6 of black box's 6 evidenced findings, by contrast, fall in the externally-reachable class. The two types answer different questions — one asks "does this flaw exist," the other asks "can this flaw be reached from outside" — and in a remediation queue, the two cannot be given the same priority. This distinction turns the choice of type from a preference into a question-matching exercise. If the question asked is "what does someone looking from outside see, and where can they reach," black box is the right tool, and its 6 evidenced findings answer that question directly; the 468 records are separately valuable as a measure of whether the defense noticed this kind of attempt at all. If the question asked is "how many flaws does the inventory have, and which ones need to go into the remediation queue," white box is the right tool, because it is the only type that zeroes out the missed column. Evaluating both under the same contract with the same metric amounts to writing the answers to two different questions into the same table. The mix table shows the transition along a single axis. Every four items shifted to white box reduces the missed flaws and lowers the trace; in exchange, the time increases. The pending candidates do not move from seven across the first three transitions, then climb to eight and ten in the last two, because as more flaws become visible, more of the items whose evidence is forbidden also become visible. White box's real cost does not show up in the table: the organization has to open its configuration and source to the test team, and that is not a time line item but an organizational decision. This is also why the table's intermediate rows are useful — opening four or eight items instead of the whole inventory brings the missed flaws down from 14 to 10 and to 8, while keeping the amount of documentation opened at roughly a fifth of the total. Which items get opened, though, is a decision made on the business side, not the technical one. #### Summary - On the same 20 items, black box produces 78 candidates and leaves 6 evidenced findings, while white box turns 60 candidates into 31; the candidate count and the evidenced-finding count move in opposite directions. - The source of the gap is that in 19 of the 41 flaws, the proof level is higher than the visibility level: seeing a flaw and proving it require separate access. - 10 flaws cannot be evidenced by any type, because their evidence can only be obtained through an action that would affect the service and the rules of engagement forbid it; these items stand in the report not as evidenced findings but as candidates with their conditions written down. - All three types fit inside the 70-hour window; the distinction is time per evidenced finding — 8.82 hours versus 1.87 hours. - Black box leaves 468 records on the defense side, white box leaves 90; trace per evidenced finding drops from 78 to 2.9. - 17 of white box's 31 evidenced findings are meaningful only from the inside; all 6 of black box's evidenced findings fall in the externally-reachable class, and the two types answer different questions. #### Next Step All three types share one common residue: false positives. In black box, 62 candidates do not correspond to a real flaw; in gray box, 36; in white box, 19 — and in this lesson these numbers were produced by a single multiplier. In reality, which candidate gets eliminated is determined not by a multiplier but by what evidence the candidate rests on: a candidate produced by reading version information and a candidate produced by observing behavior do not carry the same weight. The next lesson runs the candidate list through validation rules and counts the eliminated false positives one by one: how many candidates rest only on version information, how many are confirmed by behavior, and how much risk the validation itself adds to the target. ### Vulnerability Validation Source: https://academia.sh/en/courses/penetration-testing/vulnerability-validation 240 raw candidates are run through four validation rules: deduplication removes 101 candidates without dropping a single finding, the version support rule removes 23, the impact boundary removes 9 candidates, validation eliminates 46 false positives, and 61 evidenced findings remain. 43 percent of the candidates come from version info, but only 16.3 percent of that class is real. When the validation order is tied to the chain, the validated candidates drop from 107 to 74, risk from 17.8 to 11.9, and the defense-side trace from 461 to 313 records; the cost is that 16 real flaws whose precondition is treated as closed stay unevidenced. The previous lesson compared the three assessment types and found the same residue in all of them: 62 candidates in black box, 36 in gray box, 19 in white box did not correspond to a real flaw. Those numbers were produced by a single noise multiplier. In reality, which candidate gets eliminated is determined not by a multiplier but by what signal the candidate rests on. This lesson runs a candidate list through validation rules and counts how many candidates each rule removes and how many real findings it takes down with them. A **false positive** is a candidate that does not correspond to a real flaw; **validation** is the step that turns a candidate into evidence. The validation step does not stay on paper — it actually touches the target: **taking this step without written authorization and a defined scope is both illegal and unprofessional.** - **TP13.** The candidate list is 240 raw items and rests on the 20 in-scope items from the previous two lessons. Both the list and the flaw set are **fictional**; no action is taken against any target. - **TP14.** A **slot** is one chain step of one item. 20 items and 4 steps give 80 slots; a portion of the slots actually have a flaw. This set is the model's ground truth, and the assessment does not know it. - **TP15.** Each candidate carries a **signal class**: version info, response difference, configuration, or behavior. Weak classes scatter most of their candidates onto clean slots. - **TP16.** A **finding** is a real flaw at a slot; more than one candidate can point to the same finding. **Lost** is the number of real findings with no remaining candidate. - **TP17.** Validation has three costs: time, the **risk score** added to the target, and the **trace** left on the defense side. All three vary per candidate and by signal class. - **TP18.** The chain model is taken as input from the previous course: if a step finds no flaw, the item's subsequent steps can be treated as unreachable. #### The Elimination Pipeline ```js // elimination.mjs — runs the candidate list through validation rules and counts false positives. // THIS IS A MODEL; the candidate list is fictional, no target is touched. export const SEED = 20260214, ITEM = 20; const makeRng = (t) => { let s = t >>> 0; return () => (s = (s * 1664525 + 1013904223) >>> 0) / 4294967296; }; const r = makeRng(SEED); // Signal class: what evidence the candidate rests on and that evidence's real-flaw rate. // Version info is the weakest signal, because a fix can be backported without a version bump. export const CLASS = [ { name: "version info", share: 0.46, real: 0.17, time: 0.15, risk: 0.0, trace: 2 }, { name: "response diff", share: 0.24, real: 0.44, time: 0.35, risk: 0.1, trace: 5 }, { name: "configuration", share: 0.18, real: 0.71, time: 0.50, risk: 0.0, trace: 1 }, { name: "behavior", share: 0.12, real: 0.88, time: 0.90, risk: 0.6, trace: 9 }, ]; const pickClass = (u) => { let t = 0; for (const s of CLASS) { t += s.share; if (u < t) return s; } return CLASS.at(-1); }; // Slot: one chain step of an item. 20 items x 4 steps = 80 slots; a portion of them // actually have a flaw. This set is the model's ground truth, the assessment does not know it. const SLOT = []; for (let k = 0; k < ITEM; k++) for (let z = 1; z <= 4; z++) SLOT.push({ k, z, flaw: r() < 0.35 }); const FLAW_SLOT = SLOT.filter((y) => y.flaw), CLEAN_SLOT = SLOT.filter((y) => !y.flaw); const pick = (l) => l[Math.floor(r() * l.length)]; // 240 raw candidates. Weak signal classes scatter most of the candidates onto clean slots. export const CANDIDATE = Array.from({ length: 240 }, (_, i) => { const s = pickClass(r()), y = r() < s.real ? pick(FLAW_SLOT) : pick(CLEAN_SLOT); return { id: i, item: y.k, step: y.z, class: s.name, real: y.flaw, impact: r() < 0.08 }; }); // A finding is a real flaw at one item's chain step; multiple candidates can point to the // same finding. Lost is the number of real findings with no remaining candidate. export const FINDING = new Set(CANDIDATE.filter((a) => a.real).map((a) => `${a.item}|${a.step}`)); const lost = (l) => FINDING.size - new Set(l.filter((a) => a.real).map((a) => `${a.item}|${a.step}`)).size; export function pipeline() { const stage = []; const pass = (name, list, keep) => { const remaining = [], eliminated = []; for (const a of list) (keep(a) ? remaining : eliminated).push(a); stage.push({ name, in: list.length, eliminated: eliminated.length, remaining: remaining.length, lost: lost(remaining) }); return remaining; }; let l = CANDIDATE; // K1: multiple candidates from the same signal, on the same item's same chain step, are deduplicated. const seen = new Set(); l = pass("deduplication", l, (a) => { const k = `${a.item}|${a.class}|${a.step}`; if (seen.has(k)) return false; seen.add(k); return true; }); // K2: a candidate that rests only on version info is eliminated unless the same item's // same chain step also has a behavior or response-diff signal. A version match is not evidence. const supported = new Set(l.filter((a) => a.class === "behavior" || a.class === "response diff") .map((a) => `${a.item}|${a.step}`)); l = pass("version support", l, (a) => a.class !== "version info" || supported.has(`${a.item}|${a.step}`)); // K3: a candidate whose evidence requires an action that would affect the service is not // validated, it stays pending. const pending = l.filter((a) => a.impact); l = pass("impact boundary", l, (a) => !a.impact); // K4: the remaining candidates are validated; the ones that are not real are eliminated false positives. const toValidate = l; l = pass("validation", l, (a) => a.real); return { stage, evidence: l, pending, toValidate }; } if (import.meta.url === `file://${process.argv[1]}`) { const gen = (B) => (h) => h.map((v, j) => (j ? String(v).padStart(B[j]) : String(v).padEnd(B[0]))).join(""); const { stage, evidence, pending } = pipeline(); console.log(`model (seed ${SEED}): ${CANDIDATE.length} raw candidates, ${ITEM} items, ` + `${CANDIDATE.filter((a) => a.real).length} of them real flaws`); const y1 = gen([16, 9, 9, 11, 13]); console.log("\n" + y1(["rule", "in", "elim.", "remaining", "lost finding"])); for (const a of stage) console.log(y1([a.name, a.in, a.eliminated, a.remaining, a.lost])); console.log(y1(["final result", "", `${pending.length} pending`, `${evidence.length} evidence`, ""])); const y2 = gen([16, 8, 9, 10, 9, 8]); console.log("\n" + y2(["signal class", "cand.", "real", "real %", "evid.", "elim."])); for (const s of CLASS) { const l = CANDIDATE.filter((a) => a.class === s.name); const k = evidence.filter((a) => a.class === s.name).length; console.log(y2([s.name, l.length, l.filter((a) => a.real).length, `${(100 * l.filter((a) => a.real).length / l.length).toFixed(1)}%`, k, l.length - k])); } console.log(`\nof ${CANDIDATE.length} candidates, ${CANDIDATE.length - evidence.length - pending.length} ` + `are eliminated, ${evidence.length} become evidence, ${pending.length} stay pending`); console.log(`of the ${FINDING.size} real findings, ${stage.at(-1).lost} are lost during elimination; ` + `the version support rule takes ${stage[1].lost - stage[0].lost}, the impact boundary rule takes ` + `${stage[2].lost - stage[1].lost}`); } ``` ``` model (seed 20260214): 240 raw candidates, 20 items, 114 of them real flaws rule in elim. remaining lost finding deduplication 240 101 139 0 version support 139 23 116 1 impact boundary 116 9 107 2 validation 107 46 61 2 final result 9 pending61 evidence signal class cand. real real % evid. elim. version info 104 17 16.3% 7 97 response diff 60 33 55.0% 18 42 configuration 37 30 81.1% 17 20 behavior 39 34 87.2% 19 20 of 240 candidates, 170 are eliminated, 61 become evidence, 9 stay pending of the 29 real findings, 2 are lost during elimination; the version support rule takes 1, the impact boundary rule takes 1 ``` The pipeline's four rules do four very different jobs. **Deduplication alone removes 101 candidates — 42 percent of the list — and drops no findings.** This is the pipeline's cheapest rule: duplicate candidates produced by the same signal, on the same item's same chain step, collapse into a single item. Nearly half of the "240 findings" written on a report is a counting error that this one operation alone could erase. The version support rule eliminates 23 candidates and, in exchange, takes down **1 real finding.** The impact boundary sets 9 candidates aside as pending, dropping one more finding. The actual validation step eliminates 46 candidates — these are the pipeline's real false positives. What remains is 61 evidenced findings: 25.4 percent of the 240 raw candidates. The fourth column writes down the pipeline's own error. **2 of the 29 real findings are lost during elimination** — that is, 6.9 percent. The value of an elimination pipeline cannot be measured only by how many candidates it removes; every rule is a threshold, and a threshold cuts both noise and signal. If these two numbers are not reported together, an aggressive elimination pipeline looks high-quality even while it narrows the scope. The second table shows where the candidates come from and explains why the pipeline is built in this order. Version info, with 104 candidates, is the largest source in the list, but only 16.3 percent of it is real and it leaves only 7 evidenced findings: 43 percent of the candidates, 11.5 percent of the evidence. The reason is not a measurement flaw — when a fix is backported, the flaw closes without the version string changing, so a version match does not carry evidence. The behavior class, meanwhile, with 39 candidates the second-smallest source, gives the largest contribution at 19 evidenced findings: observed behavior is measurement, not inference. #### Validation Order and Added Risk ```js // sequence.mjs — validation order: if the chain's precondition step is validated first, // candidates in later steps are decided without ever being run. import { CLASS, pipeline } from "./elimination.mjs"; const info = (name) => CLASS.find((s) => s.name === name); const LIST = pipeline().toValidate; // candidates that passed the elimination rules const total = (l) => l.reduce((o, a) => { const s = info(a.class); return { count: o.count + 1, hours: o.hours + s.time, risk: o.risk + s.risk, trace: o.trace + s.trace }; }, { count: 0, hours: 0, risk: 0, trace: 0 }); // Cut steps: if a chain step gets a negative result, the rest of the item is treated as // unreachable. Empty set = every candidate is validated one by one. function run(cutSteps) { const ran = [], skipped = []; for (const k of new Set(LIST.map((a) => a.item))) { const itemCandidates = LIST.filter((a) => a.item === k); let cut = false; for (let z = 1; z <= 4; z++) { const stepCandidates = itemCandidates.filter((a) => a.step === z); if (cut) { skipped.push(...stepCandidates); continue; } ran.push(...stepCandidates); if (cutSteps.has(z) && stepCandidates.length && !stepCandidates.some((a) => a.real)) cut = true; } } return { ...total(ran), evidence: ran.filter((a) => a.real).length, skipped, missed: skipped.filter((a) => a.real).length, ran }; } const PATH = [["unordered", new Set()], ["precondition only", new Set([1])], ["full chain", new Set([1, 2, 3, 4])]]; const gen = (B) => (h) => h.map((v, j) => (j ? String(v).padStart(B[j]) : String(v).padEnd(B[0]))).join(""); const yz = gen([18, 12, 8, 10, 16, 8, 8, 10]); console.log(`${LIST.length} candidates enter the validation pipeline; the risk score is a ` + `validation step's weight of affecting the service`); console.log(yz(["validation order", "validated", "evid.", "skipped", "skipped real", "hours", "risk", "trace"])); const result = {}; for (const [name, cutSteps] of PATH) { const o = run(cutSteps); result[name] = o; console.log(yz([name, o.count, o.evidence, o.skipped.length, o.missed, o.hours.toFixed(1), o.risk.toFixed(1), o.trace])); } const y2 = gen([16, 12, 14, 14, 12]); console.log("\n" + y2(["signal class", "risk/cand.", "in pipeline", "in full chain", "not run"])); for (const s of CLASS) { const a = LIST.filter((x) => x.class === s.name).length; const b = result["full chain"].ran.filter((x) => x.class === s.name).length; console.log(y2([s.name, s.risk.toFixed(1), a, b, a - b])); } const [d, o1, o2] = PATH.map(([name]) => result[name]); console.log(`\nthe precondition cut skips ${o1.skipped.length} candidates: it brings risk down ` + `from ${d.risk.toFixed(1)} to ${o1.risk.toFixed(1)}, trace from ${d.trace} to ${o1.trace}`); console.log(`the full chain cut skips ${o2.skipped.length} candidates and lowers risk by ` + `${(100 * (1 - o2.risk / d.risk)).toFixed(1)}%; the cost is that ${o2.missed} real flaws whose ` + `precondition is treated as closed stay unevidenced`); ``` ``` 107 candidates enter the validation pipeline; the risk score is a validation step's weight of affecting the service validation order validated evid. skipped skipped real hours risk trace unordered 107 61 0 0 48.9 17.8 461 precondition only 89 53 18 8 41.3 15.2 383 full chain 74 45 33 16 33.3 11.9 313 signal class risk/cand. in pipeline in full chain not run version info 0.0 26 19 7 response diff 0.1 34 23 11 configuration 0.0 23 16 7 behavior 0.6 24 16 8 the precondition cut skips 18 candidates: it brings risk down from 17.8 to 15.2, trace from 461 to 383 the full chain cut skips 33 candidates and lowers risk by 33.1%; the cost is that 16 real flaws whose precondition is treated as closed stay unevidenced ``` The three rows handle the same 107 candidates in three different orders, and each one buys something different. Unordered validation gives the most evidence — 61 — but it is the most expensive: 48.9 hours, a risk score of 17.8, and 461 records on the defense side. The full-chain order brings the number of validated candidates down to 74; risk drops 33.1 percent, trace falls to 313 records, and 15.6 hours are saved. Where the gain comes from is written in the second table. Almost all of the risk score sits in the behavior class; version and configuration candidates add zero risk to the target, because both are observation. The full-chain order never runs 8 of the 24 behavior candidates, and most of the risk drop comes from those eight runs not happening. The same logic works in the trace column: a behavior candidate produces 9 records, while a configuration candidate produces 1. The cost is in the fourth column: in the full-chain order, 33 candidates are never run at all, and 16 of them are real flaws. Whether these 16 items count as a loss depends on how accurate the chain model is. If the model is correct, the precondition leading to those flaws is closed and an attacker cannot reach them either — in that case, the 16 runs would have been unnecessary risk. If the model is incomplete, that is, if there is a second path to the same place, then 16 real flaws stay unevidenced. This is why the middle row exists: cutting only at the precondition step brings risk down from 17.8 to 15.2 while keeping the skipped real flaws at 8. Whichever order is chosen, one rule stays fixed on the reporting side. A candidate that was not run is not deleted from the list; it stays on record as "not validated because its precondition was treated as closed," with a note on which step produced which result. Otherwise, the report shows a scope wider than it actually covered. #### Summary - 240 raw candidates pass through four rules and come down to 61 evidenced findings; deduplication alone removes 101 candidates — 42 percent of the list — without dropping a single finding. - The elimination pipeline also produces its own error: 2 of the 29 real findings are lost during the rules, which is why the eliminated-candidate count and the dropped-finding count must be reported together. - 43 percent of the candidates come from version info, but only 16.3 percent of that class is real and it gives only 11.5 percent of the evidence; when a fix is backported, the version string does not change. - The behavior class is the second-smallest source, yet gives the most evidence — 19 out of 39 candidates — because observed behavior is measurement, not inference. - When the validation order is tied to the chain, the validated candidates drop from 107 to 74, risk from 17.8 to 11.9, and trace from 461 to 313 records; most of the drop comes from the 8 behavior candidates that are never run. - The cost of the full-chain order is that 16 real flaws whose precondition is treated as closed stay unevidenced; these items are not removed from the report, they are written down together with the reason they were not validated. #### Next Step What remains on hand is 61 evidenced findings, 9 pending candidates, and a list left unvalidated. If these three sets are going into a remediation queue, they have to be put in order, and the criterion for that order has not been written yet. Technical weight produces one order: ease of access, breadth of impact, difficulty of the fix. The same list, looked at from the business side, produces a different order, because the business carried by one item is not the same as another's. The next lesson scores the same set of findings in two separate business contexts and counts how many positions the two rankings diverge by. ### Impact Assessment Source: https://academia.sh/en/courses/penetration-testing/impact-assessment 61 evidenced findings are first scored by technical weight, then in two separate business contexts. Not one finding stays in the same position between the two business rankings, the mean shift is 22.23 positions, and the top-ten lists share no finding in common; 41.6 percent of pairwise comparisons invert. Under a 40-person-day budget, the technical ranking remediates 12 findings and closes 24.6 percent of the field context's impact while its own ranking closes 28.8 percent, and the ranking that divides business score by remediation day closes 39.4 percent. The previous lesson produced 61 evidenced findings, 9 pending candidates, and a list left unvalidated. If this set is going into a remediation queue, it has to be put in order. The order can be built from technical criteria: is the flaw reachable from outside or only from the inside, how many items does it affect, how much privilege does it grant. These three criteria give a single number, and that number produces a ranking. This lesson shows that this ranking is not the only ranking. The same finding carries a different weight depending on which business the asset it sits on carries, and the answer to "which business matters" changes from one organization to another. **Business impact** is the finding's technical weight multiplied by the business the asset carries. What is being measured is not which of the two scorings is correct, but **how many positions apart the two orderings put the same list.** - **TP19.** The finding set is the previous lesson's 61 evidenced findings, and each finding is tied to one of five asset classes. Both the set and the weights are **fictional**. - **TP20.** The technical score comes from three components: access level (weight 0.45), impact breadth (0.30), and privilege gained (0.25). The result is scaled to a range of 0 to 10. - **TP21.** **Business weight** is an asset class's value in that context. Two contexts are defined: revenue-and-subscriber-data priority and field-continuity priority. - **TP22.** The business score is the product of the technical score and the business weight. The same finding carries two separate business scores across the two contexts, while its technical score stays the same in both. - **TP23.** The difference between two rankings is written with four measures: the number of findings that stay in the same position, the mean rank shift, the top-ten list's shared members, and the rate of pairwise comparisons that invert. - **TP24.** The remediation budget is 40 person-days, and the entire set totals 286 person-days. The queue pulls from the ordering and skips any finding that does not fit the budget. #### One List, Three Rankings ```js // scoring.mjs — the same evidenced finding set is scored by technical weight and in two // separate business contexts. THIS IS A MODEL; the finding set and the weights are fictional. export const SEED = 20260222, N = 61; const makeRng = (t) => { let s = t >>> 0; return () => (s = (s * 1664525 + 1013904223) >>> 0) / 4294967296; }; const r = makeRng(SEED); // The previous lesson's 61 evidenced findings. Each finding is tied to an asset class. export const ASSET = ["subscriber data", "billing", "measurement accuracy", "field operations", "internal management"]; const ACCESS = { external: 1.0, credentialed: 0.6, internal: 0.3 }; export const FINDING = Array.from({ length: N }, (_, i) => { const e = ["external", "credentialed", "internal"][Math.floor(r() * 3)]; const breadth = Math.round(r() * 18) / 20, privilege = Math.round(r() * 20) / 20; return { id: i + 1, asset: ASSET[Math.floor(r() * ASSET.length)], access: e, breadth, privilege, days: 1 + Math.floor(r() * 8), // remediation cost (person-days) technical: +(10 * (0.45 * ACCESS[e] + 0.30 * breadth + 0.25 * privilege)).toFixed(2) }; }); // Business weight: an asset class's weight in that context. The same finding, with the same // technical score, carries a different business impact in each of the two contexts. export const CONTEXT = { "revenue and subscriber data": { "subscriber data": 1.0, "billing": 0.9, "measurement accuracy": 0.6, "field operations": 0.35, "internal management": 0.3 }, "field continuity": { "subscriber data": 0.4, "billing": 0.45, "measurement accuracy": 0.85, "field operations": 1.0, "internal management": 0.5 }, }; export const businessScore = (b, name) => +(b.technical * CONTEXT[name][b.asset]).toFixed(2); // Ranking: descending by score; the finding id breaks ties. export const rank = (key) => [...FINDING] .sort((a, b) => key(b) - key(a) || a.id - b.id).map((b) => b.id); export const position = (s) => new Map(s.map((id, i) => [id, i + 1])); // The difference between two rankings: how many stay in the same spot, the mean shift, the // largest shift, the top-ten overlap, and the inverted-pair rate. export function diff(s1, s2) { const p1 = position(s1), p2 = position(s2); const shift = s1.map((id) => Math.abs(p1.get(id) - p2.get(id))); const top10 = new Set(s1.slice(0, 10)); let inverted = 0, pairs = 0; for (let i = 0; i < s1.length; i++) for (let j = i + 1; j < s1.length; j++) { pairs++; if ((p1.get(s1[i]) - p1.get(s1[j])) * (p2.get(s1[i]) - p2.get(s1[j])) < 0) inverted++; } return { same: shift.filter((k) => k === 0).length, largest: Math.max(...shift), mean: shift.reduce((a, k) => a + k, 0) / shift.length, top10Common: s2.slice(0, 10).filter((id) => top10.has(id)).length, inverted: +(100 * inverted / pairs).toFixed(1) }; } if (import.meta.url === `file://${process.argv[1]}`) { const gen = (B) => (h) => h.map((v, j) => (j ? String(v).padStart(B[j]) : String(v).padEnd(B[0]))).join(""); const NAMES = Object.keys(CONTEXT); const y1 = gen([23, 8, 14, 30, 21]); console.log(`model (seed ${SEED}): ${N} evidenced findings, ${ASSET.length} asset classes`); console.log("\n" + y1(["asset class", "findings", "avg technical", ...NAMES])); for (const v of ASSET) { const l = FINDING.filter((b) => b.asset === v); const ot = l.reduce((a, b) => a + b.technical, 0) / l.length; console.log(y1([v, l.length, ot.toFixed(2), ...NAMES.map((name) => CONTEXT[name][v].toFixed(2))])); } const sT = rank((b) => b.technical), sA = rank((b) => businessScore(b, NAMES[0])); const sB = rank((b) => businessScore(b, NAMES[1])); const y2 = gen([29, 10, 12, 12, 14, 16]); console.log("\n" + y2(["comparison", "same rank", "mean shift", "largest", "top-10 common", "inverted pair %"])); for (const [ad, x, y] of [["technical - revenue context", sT, sA], ["technical - field context", sT, sB], ["revenue - field context", sA, sB]]) { const f = diff(x, y); console.log(y2([ad, f.same, f.mean.toFixed(2), f.largest, `${f.top10Common}/10`, f.inverted])); } const pT = position(sT), pA = position(sA), pB = position(sB); const y3 = gen([8, 18, 11, 8, 9, 9, 9]); console.log("\n" + y3(["finding", "asset", "technical", "rank T", "rank A", "rank B", "A-B diff"])); const focus = [...FINDING].sort((a, b) => Math.abs(pA.get(b.id) - pB.get(b.id)) - Math.abs(pA.get(a.id) - pB.get(a.id))).slice(0, 6); for (const b of focus) { console.log(y3([b.id, b.asset, b.technical.toFixed(2), pT.get(b.id), pA.get(b.id), pB.get(b.id), pA.get(b.id) - pB.get(b.id)])); } } ``` ``` model (seed 20260222): 61 evidenced findings, 5 asset classes asset class findings avg technical revenue and subscriber data field continuity subscriber data 16 5.06 1.00 0.40 billing 13 5.37 0.90 0.45 measurement accuracy 14 5.77 0.60 0.85 field operations 5 5.24 0.35 1.00 internal management 13 5.94 0.30 0.50 comparison same rank mean shift largest top-10 common inverted pair % technical - revenue context 2 14.79 39 5/10 31.3 technical - field context 1 10.00 30 2/10 20.6 revenue - field context 0 22.23 43 0/10 41.6 finding asset technical rank T rank A rank B A-B diff 6 field operations 6.40 21 47 4 43 17 field operations 7.05 17 42 2 40 16 field operations 3.60 51 58 21 37 33 field operations 7.50 10 38 1 37 34 subscriber data 5.22 37 13 46 -33 31 subscriber data 5.28 33 12 44 -32 ``` The last two columns of the first table are this lesson's actual subject. The same asset class carries a weight ranging from 0.30 to 1.00 across the two contexts, and the weights are not related to the average technical score at all. The most striking row is internal management: its average technical score, 5.94, is the highest on the list, yet it carries the list's lowest weight (0.30) in the revenue context and sits in the middle (0.50) in the field context. Items that rise to the top of the technical ranking can fall to the bottom on the business side. The second table is the numerical counterpart of this. When the two business contexts are compared, **none of the 61 findings stay in the same position**, the mean rank shift is 22.23, and the largest shift is 43 positions. The top-ten lists **share not a single finding in common**. 41.6 percent of the pairwise comparisons invert, meaning that when you take two findings and ask "which one matters more," the two contexts give a different answer to almost every other question. The technical ranking's distance to the two contexts is not equal, either: it shares 5 top-ten findings with the revenue context and 2 with the field context; the inverted-pair rate is 31.3 percent against 20.6 percent. **A ranking presented without a context is not actually context-free; it has silently picked one** — and the report does not say which one it picked. This raises the question of who is supposed to fill in the weight table. The test team can derive the technical score from the data it measures itself, but it cannot measure how valuable an asset class is to the organization; that number comes from the business side. The rule that follows from this is a formal one: the report should not be a single ranked list — **the technical score, the asset class, and the weight table must be given together.** If the weights are written down, the ranking can be reproduced; if they are not, the list is a decision whose reasoning does not show. The third table shows individual items one by one. Finding 33 sits in **first** place in the field context; it is 10th in the technical ranking and 38th in the revenue context. Finding 6 is 4th in the field context and 47th in the revenue context — a gap of 43 positions. These are not a scoring error; they are two different answers to two different questions. #### The Cost of Ranking Under a Budget ```js // budget.mjs — under a limited remediation budget, what each ranking closes. import { FINDING, CONTEXT, businessScore, rank } from "./scoring.mjs"; const BUDGET = 40; // remediation budget (person-days, fictional) const NAMES = Object.keys(CONTEXT); const record = new Map(FINDING.map((b) => [b.id, b])); const total = (name) => FINDING.reduce((a, b) => a + businessScore(b, name), 0); // Pull from the ranking until the budget runs out; a finding that does not fit is skipped. function queue(rankedIds) { const selected = []; let days = 0; for (const id of rankedIds) { const b = record.get(id); if (days + b.days <= BUDGET) { selected.push(b); days += b.days; } } return { selected, days }; } const PATH = [["technical weight", rank((b) => b.technical)], ["revenue context", rank((b) => businessScore(b, NAMES[0]))], ["field context", rank((b) => businessScore(b, NAMES[1]))], ["revenue efficiency", rank((b) => businessScore(b, NAMES[0]) / b.days)], ["field efficiency", rank((b) => businessScore(b, NAMES[1]) / b.days)]]; const gen = (B) => (h) => h.map((v, j) => (j ? String(v).padStart(B[j]) : String(v).padEnd(B[0]))).join(""); const yz = gen([21, 12, 7, 32, 20]); console.log(`budget ${BUDGET} person-days; all ${FINDING.length} findings together ` + `total ${FINDING.reduce((a, b) => a + b.days, 0)} person-days`); console.log(yz(["ranking", "remediated", "days", `${NAMES[0]} %`, `${NAMES[1]} %`])); const result = {}; for (const [name, s] of PATH) { const { selected, days } = queue(s); const closed = NAMES.map((x) => 100 * selected.reduce((a, b) => a + businessScore(b, x), 0) / total(x)); result[name] = closed; console.log(yz([name, selected.length, days, closed[0].toFixed(1), closed[1].toFixed(1)])); } // The cost of ranking on the wrong axis: for each context, the closed business impact gap // between that context's own ranking and the technical ranking. const y2 = gen([30, 14, 20, 14]); console.log("\n" + y2(["context", "by technical", "by own ranking", "lost points"])); for (const [i, name] of NAMES.entries()) { const t = result["technical weight"][i], k = result[i === 0 ? "revenue context" : "field context"][i]; console.log(y2([name, `${t.toFixed(1)}%`, `${k.toFixed(1)}%`, `${(k - t).toFixed(1)} points`])); } const topItem = [rank((b) => businessScore(b, NAMES[0]))[0], rank((b) => businessScore(b, NAMES[1]))[0]]; const posT = new Map(rank((b) => b.technical).map((id, i) => [id, i + 1])); for (const [i, id] of topItem.entries()) { const b = record.get(id); console.log(`the finding ranked first in the ${NAMES[i]} context is finding ${id} (${b.asset}, ` + `technical ${b.technical.toFixed(2)}) — rank ${posT.get(id)} in the technical ranking`); } ``` ``` budget 40 person-days; all 61 findings together total 286 person-days ranking remediated days revenue and subscriber data % field continuity % technical weight 12 40 27.0 24.6 revenue context 9 40 28.8 14.1 field context 9 40 14.5 28.8 revenue efficiency 18 40 40.8 31.3 field efficiency 20 40 33.9 39.4 context by technical by own ranking lost points revenue and subscriber data 27.0% 28.8% 1.8 points field continuity 24.6% 28.8% 4.2 points the finding ranked first in the revenue and subscriber data context is finding 10 (subscriber data, technical 9.57) — rank 1 in the technical ranking the finding ranked first in the field continuity context is finding 33 (field operations, technical 7.50) — rank 10 in the technical ranking ``` The budget covers only one-seventh of the 61 findings, and the table shows how much work the ranking does under this constraint. The first three rows spend the same 40 days in three different ways. The field context closes 28.8 percent of its own impact using its own ranking, but only 14.1 percent using the revenue ranking: **ranking on the wrong axis means spending the same budget at half the efficiency.** The cost of the technical ranking is not equal across the two contexts, either. It falls behind its own ranking by 1.8 points in the revenue context and by 4.2 points in the field context. The technical weight looks "neutral," but that neutrality costs one context nearly two and a half times as much as the other. The last two rows are more interesting on their own. The ranking that divides the business score by the remediation day closes 18 and 20 findings in the same 40 days, and **beats the other three rankings in both contexts** — 40.8 percent in the revenue context, 39.4 percent in the field context. The reason shows up in the table: ranking by pure business impact puts the budget into nine expensive items, while dividing by cost fits more than twice as many items into the same budget. A report's "critical"-tagged top ten makes this comparison impossible if the remediation cost is not written down. Another thing the last two rows show is that the rankings do not all start at the same place: the revenue context's top item is also first in the technical ranking, while the field context's top item is tenth in the technical ranking. When a report imposes a single order, this second item falling nine positions behind is not a decision — it is a side effect. #### Summary - When the same 61 findings are scored in two business contexts, not a single one stays in the same position; the mean rank shift is 22.23, the largest shift is 43 positions, and the top-ten lists share no common members. - 41.6 percent of the pairwise comparisons invert between the two contexts; the technical ranking's distance to the two contexts is not equal either (5/10 and 2/10 shared), meaning a context-free ranking has silently picked a context. - Business weight is not related to the technical score: the internal management class has the highest average technical score at 5.94, yet carries the list's lowest weight in the revenue context. - Under a 40-person-day budget, the field context closes 28.8 percent of its impact with its own ranking and only 14.1 percent with the revenue ranking; ranking on the wrong axis spends the same budget at half the efficiency. - The technical ranking falls behind by 1.8 points in the revenue context and by 4.2 points in the field context: the cost of the seemingly neutral axis varies twofold by context. - The ranking that divides the business score by the remediation day closes 18 to 20 findings on the same budget and beats the other rankings in both contexts; a list that does not record the remediation cost makes this comparison impossible. #### Next Step The findings are evidenced and put in order along two separate axes. What comes next goes to a remediation team, and that team will read the finding for the first time. Having evidence on hand does not mean that evidence can be reproduced on the other side: if the precondition, the input, and the environment it was observed under are not written down, verifying the fix is impossible too. The next lesson measures the report in terms of **reproducibility**: how many findings can be reproduced step by step, how many stay environment-dependent, and how many items of the remediation recommendation are directly actionable. ### Reporting Source: https://academia.sh/en/courses/penetration-testing/reporting 61 report records are audited in terms of reproducibility. Even though each of the five mandatory fields is individually filled at a high rate, the rate at which all five are filled together is 31.1 percent; a second reader can reproduce 8 findings, and 3 can be called closed after a retest round. Writing the missing fields takes 9.4 hours and raises the closeable count to 49; 12 findings carry an environment variable and cannot be reproduced step by step under any circumstance. The remediation suggestion is directly actionable for 29 findings, requires a separate decision for 22, and is entirely absent for 10. The previous lesson put the 61 evidenced findings in order along two separate axes and measured what a 40-person-day remediation budget closes. The next step is for the finding to go to a remediation team, and that team will read the finding for the first time. The test team having evidence on hand does not mean that evidence can be reproduced on the other side. This lesson tests the report with a single measure: **reproducibility**. A finding is reproducible if someone who did not produce it can observe the same result; if it cannot be observed, whether the remediation worked can never be verified either. **Reproduction steps** are therefore a report item, not a narrative. No example step is written in this lesson; what is measured is **what information a step must carry** and how many records actually contain that information. - **TP25.** The report set is the previous lesson's 61 findings. The records are **fictional** and contain no attack steps; what is counted is field names. - **TP26.** Eight fields are defined: precondition, input definition, observed result, timestamp, evidence id, environment info, expected behavior, and remediation suggestion. - **TP27.** The first five fields are **mandatory** for reproduction. The remaining three complete the record but do not enter the step set itself. - **TP28.** A record can carry an **environment variable**: even when environment info is written down, its value changes from run to run. These records cannot be reproduced step by step. - **TP29.** A second reader can reproduce a finding only if the step set is complete, environment info is written, and there is no environment variable. - **TP30.** A finding can be called "closed" in a retest round only if it is reproducible **and** its expected behavior is written. Field-writing durations are fictional. #### Field Audit ```js // report.mjs — reproducibility audit of report records. // THIS IS A MODEL; the records are fictional and contain no attack steps — only the // FIELD NAMES a step must carry are counted. export const SEED = 20260519, N = 61; const makeRng = (t) => { let s = t >>> 0; return () => (s = (s * 1664525 + 1013904223) >>> 0) / 4294967296; }; const r = makeRng(SEED); // A reproduction step is a report item. Fields are the information the step must carry; // the fill rate is fictional and reflects the field's difficulty. export const FIELD = [ { name: "precondition", fillRate: 0.82, mandatory: true, minutes: 4 }, { name: "input definition", fillRate: 0.74, mandatory: true, minutes: 6 }, { name: "observed result", fillRate: 0.91, mandatory: true, minutes: 5 }, { name: "timestamp", fillRate: 0.88, mandatory: true, minutes: 1 }, { name: "evidence id", fillRate: 0.63, mandatory: true, minutes: 3 }, { name: "environment info", fillRate: 0.57, mandatory: false, minutes: 4 }, { name: "expected behavior", fillRate: 0.49, mandatory: false, minutes: 7 }, { name: "remediation suggestion", fillRate: 0.86, mandatory: false, minutes: 12 }, ]; // Remediation suggestion class: directly actionable, decision-required, structural. const SUGGESTION = [["direct", 0.41], ["decision-required", 0.34], ["structural", 0.25]]; const pickSuggestion = (u) => { let t = 0; for (const [name, p] of SUGGESTION) { t += p; if (u < t) return name; } return SUGGESTION.at(-1)[0]; }; export const RECORD = Array.from({ length: N }, (_, i) => { const filled = {}; for (const f of FIELD) filled[f.name] = r() < f.fillRate; return { no: i + 1, filled, suggestion: filled["remediation suggestion"] ? pickSuggestion(r()) : "none", // Environment-dependent: even when the record carries environment info, its value // changes from run to run. variable: r() < 0.23 }; }); const MANDATORY = FIELD.filter((f) => f.mandatory).map((f) => f.name); export const stepsComplete = (k) => MANDATORY.every((name) => k.filled[name]); // A second reader can reproduce a finding only if the step set is complete, environment // info is present, and there is no environment variable. export const isReproducible = (k) => stepsComplete(k) && k.filled["environment info"] && !k.variable; if (import.meta.url === `file://${process.argv[1]}`) { const gen = (B) => (h) => h.map((v, j) => (j ? String(v).padStart(B[j]) : String(v).padEnd(B[0]))).join(""); const yz = gen([23, 10, 9, 10, 12, 13]); console.log(`model (seed ${SEED}): ${N} report records, ${FIELD.length} fields, ` + `${MANDATORY.length} mandatory for reproduction`); console.log("\n" + yz(["field", "mandatory", "filled", "missing", "sole gap", "min/finding"])); for (const f of FIELD) { const filled = RECORD.filter((k) => k.filled[f.name]).length; // Sole gap in: records that would complete their step set if only this field were filled. const sole = RECORD.filter((k) => !k.filled[f.name] && f.mandatory && MANDATORY.filter((z) => !k.filled[z]).length === 1).length; console.log(yz([f.name, f.mandatory ? "yes" : "no", filled, N - filled, sole, f.minutes])); } const complete = RECORD.filter(stepsComplete), reproducible = RECORD.filter(isReproducible); const noEnv = complete.filter((k) => !k.filled["environment info"]).length; const variable = complete.filter((k) => k.filled["environment info"] && k.variable).length; const y2 = gen([48, 9, 9]); console.log("\n" + y2(["record class", "findings", "percent"])); for (const [name, s] of [["step set complete", complete.length], ["step set incomplete", N - complete.length], ["complete but no environment info", noEnv], ["complete, environment written, value variable", variable], ["second reader can reproduce", reproducible.length]]) { console.log(y2([name, s, `${(100 * s / N).toFixed(1)}`])); } const missingMost = MANDATORY.map((z) => [z, RECORD.filter((k) => !k.filled[z]).length]) .sort((a, b) => b[1] - a[1])[0]; console.log(`\nthe mandatory field missing most often is "${missingMost[0]}" ` + `(missing from ${missingMost[1]} records); writing all fields for one record ` + `takes ${FIELD.reduce((a, x) => a + x.minutes, 0)} minutes`); } ``` ``` model (seed 20260519): 61 report records, 8 fields, 5 mandatory for reproduction field mandatory filled missing sole gap min/finding precondition yes 55 6 1 4 input definition yes 41 20 9 6 observed result yes 56 5 2 5 timestamp yes 54 7 1 1 evidence id yes 36 25 11 3 environment info no 37 24 0 4 expected behavior no 30 31 0 7 remediation suggestion no 51 10 0 12 record class findings percent step set complete 19 31.1 step set incomplete 42 68.9 complete but no environment info 7 11.5 complete, environment written, value variable 4 6.6 second reader can reproduce 8 13.1 the mandatory field missing most often is "evidence id" (missing from 25 records); writing all fields for one record takes 42 minutes ``` Looked at field by field, the first table makes the report look good: observed result is written in 56 of 61 records, precondition in 55, timestamp in 54. No single field is a disaster on its own. The second table, though, leaves **the count of records where all five mandatory fields are filled together at 19** — 31.1 percent. Independent gaps do not add, they multiply: a team that fills each field at a high individual rate can still end up at roughly one-third on the combined measure. Report quality must be measured **per record**, not per field. The "sole gap" column shows where the cheapest gain lies. Evidence id is the sole missing field in 11 records: filling only that field raises the count of records with a complete step set from 19 to 30. The field costs 3 minutes per record, and comes to 75 minutes across the 25 records where it is missing — a 57 percent increase for under an hour and a half of work. Input definition is the sole gap in 9 records, and the 20 records where it is missing come to 120 minutes. An empty column carries information too: environment info, expected behavior, and remediation suggestion are never the sole gap in any record, because they are not part of the mandatory field set and cannot complete the step set on their own. The third number is the real warning. Of the 19 records with a complete step set, only **8** can be reproduced by a second reader. Of the rest, 7 have no environment info written at all, and 4 have environment info written but its value changes from run to run. Reproducibility does not end with the steps being written down; it also requires that the steps be executable **under the same conditions**. #### The Cost of Completion and the Retest ```js // closure.mjs — the cost of completing missing fields and how many findings can be // called "closed" in a retest round. import { RECORD, FIELD, stepsComplete, isReproducible } from "./report.mjs"; const N = RECORD.length; const minutesOf = (name) => FIELD.find((f) => f.name === name).minutes; const MANDATORY = FIELD.filter((f) => f.mandatory).map((f) => f.name); const missingMinutes = (fields) => RECORD.reduce((a, k) => a + fields.filter((name) => !k.filled[name]).reduce((s, name) => s + minutesOf(name), 0), 0); // A finding can be called "closed" in a retest round only if it is reproducible AND its // expected behavior is written. const closes = (k) => isReproducible(k) && k.filled["expected behavior"]; const gen = (B) => (h) => h.map((v, j) => (j ? String(v).padStart(B[j]) : String(v).padEnd(B[0]))).join(""); const yz = gen([32, 15, 13, 18, 16]); const STEP = [ ["today's report", [], (k) => isReproducible(k), closes], ["mandatory fields completed", MANDATORY, (k) => k.filled["environment info"] && !k.variable, (k) => k.filled["environment info"] && !k.variable && k.filled["expected behavior"]], ["environment info also written", [...MANDATORY, "environment info"], (k) => !k.variable, (k) => !k.variable && k.filled["expected behavior"]], ["expected behavior also written", [...MANDATORY, "environment info", "expected behavior"], (k) => !k.variable, (k) => !k.variable], ]; console.log(`${N} findings; writing a missing field adds overhead to a record equal to ` + `that field's duration`); console.log(yz(["completion step", "added minutes", "added hours", "reproducible", "closeable"])); for (const [name, fields, u, kap] of STEP) { const d = missingMinutes(fields); console.log(yz([name, d, (d / 60).toFixed(1), RECORD.filter(u).length, RECORD.filter(kap).length])); } const variable = RECORD.filter((k) => k.variable).length; console.log(`even if every field were written, ${variable} findings carry an environment ` + `variable and cannot be reproduced step by step; for these, the evidence record itself ` + `is the only basis`); // Remediation suggestion class: how many items are directly actionable. const CLASS = ["direct", "decision-required", "structural", "none"]; const y2 = gen([20, 9, 18, 16, 18]); console.log("\n" + y2(["suggestion class", "findings", "reproducible", "closeable", "needs decision"])); for (const s of CLASS) { const l = RECORD.filter((k) => k.suggestion === s); console.log(y2([s, l.length, l.filter(isReproducible).length, l.filter(closes).length, s === "direct" || s === "none" ? 0 : l.length])); } console.log(`\nof ${N} findings, ${RECORD.filter((k) => k.suggestion === "direct").length} have a ` + `directly actionable suggestion; ${RECORD.filter((k) => k.suggestion === "decision-required" || k.suggestion === "structural").length} require a separate decision before remediation, ` + `${RECORD.filter((k) => k.suggestion === "none").length} have no suggestion at all`); console.log(`findings that can be called closed in a retest round: ${RECORD.filter(closes).length}; ` + `even among the ${RECORD.filter(stepsComplete).length} findings with a complete step set, ` + `a portion falls short of this bar`); ``` ``` 61 findings; writing a missing field adds overhead to a record equal to that field's duration completion step added minutes added hours reproducible closeable today's report 0 0.0 8 3 mandatory fields completed 251 4.2 27 14 environment info also written 347 5.8 49 25 expected behavior also written 564 9.4 49 49 even if every field were written, 12 findings carry an environment variable and cannot be reproduced step by step; for these, the evidence record itself is the only basis suggestion class findings reproducible closeable needs decision direct 29 5 1 0 decision-required 12 1 1 12 structural 10 1 1 10 none 10 1 0 0 of 61 findings, 29 have a directly actionable suggestion; 22 require a separate decision before remediation, 10 have no suggestion at all findings that can be called closed in a retest round: 3; even among the 19 findings with a complete step set, a portion falls short of this bar ``` The first row gives this course's sharpest number: in today's report, **3 of the 61 findings** are in a state that can be called closed after a retest round. For the remaining 58, there is no basis to answer the question "did it work" once remediation is applied; the fix gets made, resources get spent, and the result cannot be measured. The rest of the ladder writes the price of this gap. Completing the mandatory fields takes 251 minutes, or 4.2 hours, and raises the closeable count to 14. When environment info is also written, 25 findings are reached at a total of 5.8 hours; when expected behavior is also written, **49 findings** are reached at 9.4 hours. The previous lesson's remediation budget was 40 person-days; 9.4 hours is roughly 3 percent of that budget. When report writing is seen as overhead, it is the first item to get cut — but what actually gets cut is the measurability of the remediation budget. 12 findings fall outside this ladder. These carry an environment variable, and even if every field were written, step-by-step reproduction would not give the same result. For these records, the only basis is **the evidence record itself**, taken at the moment of observation: under what identifier it is stored, when it was captured, and how its integrity is demonstrated. A finding that cannot be reproduced is not deleted from the report; it stays with an "environment-dependent" tag and its evidence id. This also explains why the evidence id field, despite being one of the cheapest fields, is the mandatory field missing most often: it takes three minutes to write, but when it is missing it both breaks the step set and removes the only basis for environment-dependent findings. Only an identifier string sits in the record; where the evidence that identifier points to is stored and how it is verified is a separate item. The second table separates the two axes from each other. The remediation suggestion is directly actionable for 29 findings, but only 5 of those 29 are reproducible and only 1 can be called closed. **Suggestion quality and evidence quality are two independent axes**: a well-written suggestion attached to an unverifiable finding gets implemented, but its result cannot be measured. The reverse also happens — 10 findings have no suggestion at all, and these stop at the first step once they enter the remediation queue. The remaining 22 findings require a separate decision before remediation: for 12, the suggestion offers a choice; for 10, a structural change is needed. This distinction being written into the report is necessary for the remediation team to order the queue correctly; a "decision-required" item cannot be put into the same sprint as a directly actionable one. #### Summary - Even though each of the five mandatory fields is individually filled at a high rate, all five together are filled in only 19 records — 31.1 percent: independent gaps do not add, they multiply. - Evidence id is the sole missing field in 11 records, and completing it raises the count of records with a complete step set from 19 to 30; its cost is 75 minutes for the 25 records where it is missing. - Of the 19 records with a complete step set, only 8 can be reproduced by a second reader; 7 lack environment info, and 4 have environment info written but with a variable value. - In today's report, 3 of the 61 findings can be called closed in a retest round; writing the missing fields takes 9.4 hours and raises this number to 49 — roughly 3 percent of the remediation budget. - 12 findings cannot be reproduced step by step under any circumstance because they carry an environment variable; for these, the only basis is the evidence record's identifier, timestamp, and integrity. - Suggestion quality and evidence quality are independent: of the 29 findings with a directly actionable suggestion, only 1 can be called closed, while 22 findings require a separate decision before remediation. #### Next Step The report is written, and which findings are verifiable has been counted. All of this measurement rested on a single assumption: where the report is going is known, and a remediation team is there. That assumption falls away when the finding comes from outside the organization's own contracted assessment rather than from it. Then who it gets reported to, through which channel, and within how much time; how long remediation should be given; and when the finding can be published become the subject of a separate process. The next lesson measures responsible disclosure in terms of time and steps, and closes out the course. ### Responsible Disclosure Source: https://academia.sh/en/courses/penetration-testing/responsible-disclosure The time from notification to publication is measured across six stages: single-party findings close in an average of 12.7 days, four-party findings in 96.0 days, and the longest stage is remediation. Comparing three publication policies, the fixed 90-day window publishes 13 findings unfixed, the fix-linked window publishes none unfixed but pushes the latest publication to day 226, and the extendable window stays at 3. The lesson closes the course and gathers the candidate, evidence, and trace counts of eighteen lessons into a single table. The previous lesson measured the report in terms of reproducibility, and it rested on a single assumption: where the report is going is known, a remediation team is there, and that team is expecting the finding. In a contracted assessment, this assumption holds. It falls away when the finding comes from outside — from an independent research effort, from an item spotted in a third-party component. **Responsible disclosure** is the process that determines, in that case, who to notify, through which channel, within how much time, and when the finding can be published. This lesson measures it in terms of two numbers: **how many days** and **how many parties**. The measurement is a model, and all durations are fictional; no real organization, real component, or real vulnerability identifier is written. - **TP31.** The finding set is the previous lesson's 61 records. Each finding falls into one of four notification classes, and the class determines the number of parties that enter the process. - **TP32.** The process has six stages: finding the notification channel, acknowledgment, technical verification, remediation, deployment, and a verification pass. Durations are in days. - **TP33.** Remediation time deviates from the class baseline by a factor of 0.6 to 1.6; for a portion of findings, remediation **gets stuck** and the duration quadruples. - **TP34.** The **ready day** is the day the fix is deployed and verified. A fix being written and a fix being applied are two separate dates. - **TP35.** Three publication policies are compared: a fixed publication 90 days after notification; publication 14 days after the fix is ready; and 90 days, with a one-time 30-day extension if remediation is still underway. - **TP36.** If a finding is not ready on its publication day, it counts as **published unfixed**, and the number of days in between is the period it stays open after publication. #### From Notification to Publication ```js // disclosure.mjs — responsible disclosure process: days and parties from notification to publication. // THIS IS A MODEL; the finding set, durations, and parties are fictional. export const SEED = 20260714, N = 61; const makeRng = (t) => { let s = t >>> 0; return () => (s = (s * 1664525 + 1013904223) >>> 0) / 4294967296; }; const r = makeRng(SEED); // Stages: from finding the notification channel to publication. Durations are in days // and vary by class; each class involves a different number of parties. export const STAGE = ["channel discovery", "acknowledgment", "technical verification", "remediation", "deployment", "verification pass"]; const SHORT = ["channel", "ack", "review", "remediation", "deploy", "pass"]; export const CLASS = [ { name: "org configuration", share: 0.34, parties: ["org"], days: [1, 1, 2, 3, 2, 2] }, { name: "org software", share: 0.28, parties: ["org", "dev team"], days: [1, 2, 4, 14, 7, 3] }, { name: "third-party component", share: 0.23, parties: ["org", "dev team", "component vendor"], days: [5, 7, 10, 45, 21, 5] }, { name: "shared infrastructure", share: 0.15, parties: ["org", "dev team", "infrastructure operator", "regulator"], days: [3, 5, 8, 30, 30, 7] }, ]; const pickClass = (u) => { let t = 0; for (const c of CLASS) { t += c.share; if (u < t) return c; } return CLASS.at(-1); }; // Each finding falls into a class; remediation time deviates from the class baseline, // and gets stuck for a portion of findings. export const FINDING = Array.from({ length: N }, (_, i) => { const c = pickClass(r()), multiplier = 0.6 + r(), stuck = r() < 0.12; const days = c.days.map((d, j) => (j === 3 ? Math.round(d * multiplier * (stuck ? 4 : 1)) : d)); return { no: i + 1, class: c.name, parties: c.parties, days, stuck, ready: days.reduce((a, d) => a + d, 0) }; }); if (import.meta.url === `file://${process.argv[1]}`) { const gen = (B) => (h) => h.map((v, j) => (j ? String(v).padStart(B[j]) : String(v).padEnd(B[0]))).join(""); const yz = gen([22, 9, 9, 6, 8, 13, 9, 7, 10, 9]); console.log(`model (seed ${SEED}): ${N} findings, ${CLASS.length} notification classes`); console.log("\n" + yz(["finding class", "finding", ...SHORT, "avg days", "parties"])); for (const c of CLASS) { const l = FINDING.filter((f) => f.class === c.name); const avg = l.reduce((a, f) => a + f.ready, 0) / l.length; console.log(yz([c.name, l.length, ...c.days, avg.toFixed(1), c.parties.length])); } const parties = new Set(CLASS.flatMap((c) => c.parties)); const longest = STAGE.map((a, j) => [a, CLASS.reduce((x, c) => x + c.days[j], 0)]) .sort((a, b) => b[1] - a[1])[0]; console.log(`\nthe four classes involve ${parties.size} distinct parties in total; ` + `the longest stage is "${longest[0]}" (${longest[1]} days total across the four classes)`); const deploy = CLASS.reduce((a, c) => a + c.days[4], 0); console.log(`once remediation is written, deployment alone takes ${deploy} days: ` + `a fix existing and a fix being applied are two separate dates`); } ``` ``` model (seed 20260714): 61 findings, 4 notification classes finding class finding channel ack review remediation deploy pass avg days parties org configuration 26 1 1 2 3 2 2 12.7 1 org software 15 1 2 4 14 7 3 40.3 2 third-party component 11 5 7 10 45 21 5 113.2 3 shared infrastructure 9 3 5 8 30 30 7 96.0 4 the four classes involve 5 distinct parties in total; the longest stage is "remediation" (92 days total across the four classes) once remediation is written, deployment alone takes 60 days: a fix existing and a fix being applied are two separate dates ``` Read together, the last two columns do not move in perfect lockstep, but the ends are clear: the average duration grows from 12.7 days to 113.2 days — nearly ninefold — and only the four-party class, at 96.0 days, falls below the three-party class. A single-party configuration finding closes in 12.7 days; a three-party component finding closes in 113.2 days. The source is not remediation alone: channel discovery grows from 1 day to 5, acknowledgment from 1 day to 7, because each additional party adds its own notification channel, acknowledgment, and technical review. The longest stage is remediation, taking 92 days across the four classes. But looking at remediation alone is misleading: **deployment alone takes 60 days.** A fix being written and a fix being applied are two separate dates, and the gap between them is still an open exposure. In the shared infrastructure class, deployment takes as long as remediation itself, because the change has to be applied not in one place but across every operated item. The channel discovery stage is small, but it says something: whoever is making the notification first has to search for **who to notify**. An organization without a published notification address adds days to its own remediation time from the very start. #### Publication Window and Party Count ```js // publication.mjs — publication policy: fixed window, fix-linked window, extendable window. import { FINDING, CLASS } from "./disclosure.mjs"; const N = FINDING.length, WINDOW = 90, EXTENSION = 30, WAIT = 14; // Each policy returns a publication day for a finding. const POLICY = [ ["fixed window", () => WINDOW], ["fix-linked", (f) => f.ready + WAIT], ["extendable window", (f) => (f.ready <= WINDOW ? WINDOW : WINDOW + EXTENSION)], ]; const gen = (B) => (h) => h.map((v, j) => (j ? String(v).padStart(B[j]) : String(v).padEnd(B[0]))).join(""); const yz = gen([20, 11, 11, 13, 16, 20]); console.log(`${N} findings; window ${WINDOW} days, extension ${EXTENSION} days, ` + `post-fix wait ${WAIT} days (fictional)`); console.log(yz(["policy", "avg publish", "latest", "fixed", "unfixed", "open after publish"])); for (const [name, f] of POLICY) { const days = FINDING.map(f), fixedBefore = FINDING.filter((k) => k.ready <= f(k)).length; console.log(yz([name, (days.reduce((a, d) => a + d, 0) / N).toFixed(1), Math.max(...days), fixedBefore, N - fixedBefore, (FINDING.reduce((a, k) => a + Math.max(0, k.ready - f(k)), 0) / N).toFixed(1)])); } // The relationship between party count and exceeding the window. const y2 = gen([22, 8, 10, 14, 16, 14]); console.log("\n" + y2(["finding class", "parties", "findings", "over 90 days", "over 120 days", "stuck"])); for (const c of CLASS) { const l = FINDING.filter((k) => k.class === c.name); console.log(y2([c.name, c.parties.length, l.length, l.filter((k) => k.ready > WINDOW).length, l.filter((k) => k.ready > WINDOW + EXTENSION).length, l.filter((k) => k.stuck).length])); } const over = FINDING.filter((k) => k.ready > WINDOW); const manyParties = FINDING.filter((k) => k.parties.length >= 3); console.log(`\n${over.length} of ${N} findings exceed the ${WINDOW}-day window; ` + `${over.filter((k) => k.parties.length >= 3).length} of them come from classes ` + `involving three or more parties`); console.log(`${(100 * manyParties.filter((k) => k.ready > WINDOW).length / manyParties.length) .toFixed(1)}% of the ${manyParties.length} findings with three or more parties exceed the ` + `window; ${FINDING.filter((k) => k.parties.length === 1 && k.ready > WINDOW).length} of the ` + `${FINDING.filter((k) => k.parties.length === 1).length} single-party findings exceed it`); const stuck = FINDING.filter((k) => k.stuck); console.log(`the ${stuck.length} findings whose remediation gets stuck wait an average of ` + `${(stuck.reduce((a, k) => a + k.ready + WAIT, 0) / stuck.length).toFixed(1)} days unpublished ` + `under the fix-linked policy; under the fixed window they publish on day ${WINDOW}`); ``` ``` 61 findings; window 90 days, extension 30 days, post-fix wait 14 days (fictional) policy avg publish latest fixed unfixed open after publish fixed window 90.0 90 48 13 5.9 fix-linked 63.9 226 61 0 0.0 extendable window 96.4 120 58 3 3.1 finding class parties findings over 90 days over 120 days stuck org configuration 1 26 0 0 3 org software 2 15 0 0 3 third-party component 3 11 9 2 2 shared infrastructure 4 9 4 1 1 13 of 61 findings exceed the 90-day window; 13 of them come from classes involving three or more parties 65.0% of the 20 findings with three or more parties exceed the window; 0 of the 26 single-party findings exceed it the 9 findings whose remediation gets stuck wait an average of 106.9 days unpublished under the fix-linked policy; under the fixed window they publish on day 90 ``` Three policies protect three different things, and none subsumes the other. The fixed window gives a firm date and holds the organization to a deadline, but it publishes 13 findings before they are fixed, and those stay both open and public for an average of 5.9 days. The fix-linked window never publishes a finding before it is fixed; its average publish day, 63.9, is the earliest, because most findings close quickly. Its cost is not in the last column, but in the "latest" column: **day 226.** When remediation gets stuck, publication can be deferred indefinitely, and the decision belongs to the party that received the notification. The extendable window sits between the two: it brings the number published before being fixed down from 13 to 3, and the post-publication open period down from 5.9 days to 3.1 days, while keeping the latest publication at 120 days. What it gains is a firm upper bound; what it adds is the decision of when to grant the extension. The second table shows who exceeds the window, and the result collapses onto a single axis: **all 13 findings that exceed 90 days come from classes involving three or more parties.** None of the 26 single-party findings exceed the window; 65.0 percent of the 20 findings with three or more parties do. What determines the duration is not the technical weight of the exposure, it is how many organizations have to line up on the same calendar. The 9 findings whose remediation gets stuck are spread across this table, and under the fix-linked policy they wait an average of 106.9 days before publication. A practical rule follows from this: the window cannot be written as a single number. The party count is known on the first day of notification, and the window length can be tied to that count. For a single-party finding, 90 days is more than enough; for a four-party finding, the same 90 days means publishing before coordination is complete. #### Summary - The time from notification to the ready day depends on the party count: single-party findings take an average of 12.7 days, two-party 40.3 days, three-party 113.2 days, four-party 96.0 days. - The increase does not come from remediation alone; channel discovery grows from 1 day to 5, acknowledgment from 1 day to 7, because each party adds its own channel, acknowledgment, and review. - Deployment takes a total of 60 days across the four classes: a fix being written and a fix being applied are two separate dates, and the gap between them is still an open exposure. - The fixed 90-day window publishes 13 findings before they are fixed and leaves them open for an average of 5.9 days; the fix-linked window publishes none of them unfixed but pushes the latest publication to day 226. - The extendable window brings the number published unfixed down to 3 and the remaining open period down to 3.1 days, while keeping the upper bound at 120 days. - All 13 findings that exceed 90 days come from classes involving three or more parties; none of the 26 single-party findings exceed it. Window length should be tied to party count, not technical weight. #### Course Wrap-Up | Lesson | Candidate | Turned into evidence | Trace left on defense | |---|---|---|---| | Open-Source Intelligence | 29 items, 20 candidates | 9 evidenced findings, 6 false positives | 12 access log entries, 0 alerts | | Domain Name and Registration Lookup | 20 candidates, 4 criteria | 8 evidenced findings, 0 false positives | 26 log entries, 0 alerts | | Subdomain and Asset Discovery | 24 zone records | 21 found, 7 dangling | 8.57 records per name, 5 alerts | | Port Scanning | 96 points, 21 ambiguous | 18 evidenced findings, 3 false positives | 72–90 log entries, 2 alerts | | Service and Version Detection | 21 services, 4 levels | 16 confirmed, 0 wrong version | 168 log entries, 6 alerts | | Packet Capture and Analysis | 85 claim candidates | 50 evidenced findings, 28 inferences | 0 log entries, 1 configuration confirmation | | Protocol Analyzers | 91 candidates, 30 fields | 43 evidenced findings, 48 inferences | 0 log entries, 1 alert per flow | | Request Manipulation | 18 candidates, 40 requests | 8 evidenced findings, 6 eliminated | 35 log entries, 16 alerts | | Sandbox Analysis | 14 items, 27 behaviors | 12 resolved, 1 wrong | 0 log entries, 5 external connections | | File and Link Reputation Services | 24 items, 8 submittable | 86% precision, 50% recall at threshold 7 | 0 log entries, 14 service queries | | Exploitation Frameworks | 16 candidates, 9 real | 3 evidenced findings, 3 evidence of absence | 31 log entries, 17 alerts | | Test Distributions and Labs | 18 skill items | 13 covered, 5 open | 0 log entries, 0 real risk | | Rules of Engagement | 64 targets, 11 borderline | 20 confirmed in scope | 0 log entries, 6 parties, 3 stop-work | | Test Types | 78 / 63 / 60 candidates | 6 / 20 / 31 evidenced findings | 468 / 221 / 90 log entries | | Vulnerability Validation | 240 raw candidates | 61 evidenced findings, 9 pending | 461 log entries, 313 chained | | Impact Assessment | 61 findings, 2 contexts | 0 common rank in the top ten | 0 log entries, 40-person-day queue | | Reporting | 61 records, 8 fields | 8 reproducible, 3 closed | 0 log entries, 9.4 hours writing | | Responsible Disclosure | 61 findings, 4 classes | 48 fixed-before-publish | 0 log entries, 5 parties, 13 over window | The table's second and third columns are never equal in any row, and this course's rule follows from that: **the output of a penetration test is not a count of findings, it is evidenced, reproducible findings; when an unevidenced candidate makes it into the report, it wastes the remediation budget.** Nine rows read "0 log entries" in the fourth column, and the two lessons that produce the most evidenced findings are among them: the step that leaves the most trace is not the step that produces the most evidence. Across the eighteen lessons, the candidate-to-evidence gap ran in the same direction every time. Reconnaissance produced candidates in abundance and evidence sparingly; the analysis tools separated evidence from inference; validation cut the list to a quarter and wrote down its own error rate as well; the process lessons showed that evidence alone is not enough. One column stayed one-sided throughout this entire measurement. How many log entries and how many alerts each step left on the defense side was counted every time: from 12 access log entries to 468, from zero alerts to 17. But **who** reads those log entries, **in how much time**, and **at what threshold** was never asked. Does an alert actually reach anyone, how quickly is it looked at, which log entries below which threshold are never seen at all, and is it written down who does what in a real incident? The **M24/K08 Security Operations and Incident Response** course begins with exactly this question.