Test Distributions and Labs

The previous lesson measured validation attempts by risk class, evidence strength, and the meaning of a negative result. All of these measures must have been learned somewhere; a step tried for the first time during validation puts both the target and the assessment itself at risk. A real institutional network is not the place to practice. An attempt pointed at someone else’s system without written permission is unauthorized access even if done for learning, and it carries legal consequences.

This lesson measures legal practice labs in terms of scope, and what it asks is not what they teach, it is what they do not teach. The skill items produced by the topic’s six lessons are listed, how many each environment type covers is counted, and the items none of them cover are written down separately.

Scope Is a Set Difference

Four environment types are considered. A self-built isolated lab is an environment set up on your own hardware that does not reach an outside network: you build the topology yourself, so you also choose where capture happens. A prebuilt vulnerable image set consists of pre-prepared targets; setup is cheap, the topology is given. A scenario-based practice lab is one run by a provider, with scope and rules defined in advance; the terms of use are a contract, and reading it is part of the exercise. A security-focused test distribution is a toolset; it shortens setup time.

The fifth row is the measure’s reference: an authorized real assessment. By definition it covers every item, requires no setup, and requires a written contract and a defined scope.

• AT42. The skill list, environment types, and scope sets are a model; no real platform, distribution, or provider name is written.
• AT43. The skill items were drawn from this topic’s six lessons; the four items sourced from process were not measured in any lesson and were put on the list as a limit.
• AT44. An environment covering a skill means that skill can be worked through there along with its result; being able to run the tool is not enough.
• AT45. Setup times were written by hand per environment type.
• AT46. The union of labs covers three practice types; the real assessment does not join the union, it is the comparison side.
• AT47. The real record produced and the real risk carried in a lab are zero.

// lab/scope.mjs — this is a SCOPE model of legal practice labs. No real platform,
// distribution, or product name is written; environments are referred to only by type.
// The skill list was drawn from this topic's six lessons; the ones sourced from 'process'
// were not measured in any lesson and were put here as a limit.
export const SKILL = `
  b01 capture-point-selection          01
  b02 field-class-separation           01
  b03 analyzer-label-validation        02
  b04 metadata-extraction-limit        02
  b05 validation-plan-building         03
  b06 request-risk-class-assignment    03
  b07 scope-check-running              03
  b08 sandbox-prep                     04
  b09 hidden-behavior-reading          04
  b10 majority-threshold-selection     05
  b11 privacy-class-decision           05
  b12 validation-option-selection      06
  b13 negative-result-interpretation   06
  b14 target-identity-verification     06
  b15 scope-negotiation           process
  b16 client-communication        process
  b17 production-impact-carrying  process
  b18 finding-reporting           process
`.trim().split('\n').map((r) => {
  const [name, skill, source] = r.trim().split(/\s+/);
  return { name, skill, source };
});

// Environment type: which skills it covers, how many hours to set up, what its legal condition is.
// 'security-focused-test-distribution' is a toolset, not a practice lab: it shortens
// setup time and covers no skill on its own.
export const ENVIRONMENT = {
  'self-built-isolated-lab': { setupHours: 24, legal: 'your-own-hardware',
    covers: ['b01', 'b02', 'b03', 'b04', 'b05', 'b06', 'b08', 'b09', 'b12', 'b13'] },
  'prebuilt-vulnerable-image-set': { setupHours: 3, legal: 'your-own-hardware',
    covers: ['b03', 'b04', 'b05', 'b06', 'b12', 'b13'] },
  'scenario-based-practice-environment': { setupHours: 0, legal: 'provider-terms-of-use',
    covers: ['b03', 'b04', 'b05', 'b06', 'b07', 'b09', 'b10', 'b12', 'b13', 'b14'] },
  'security-focused-test-distribution': { setupHours: 1, legal: 'your-own-hardware', covers: [] },
  'authorized-real-assessment': { setupHours: 0, legal: 'written-contract-and-scope',
    covers: SKILL.map((b) => b.name) },
};

export const LAB = Object.keys(ENVIRONMENT).filter((o) => o !== 'authorized-real-assessment');

export function union(environments) {
  const k = new Set();
  for (const o of environments) for (const b of ENVIRONMENT[o].covers) k.add(b);
  return k;
}

export function perLesson(covered) {
  const m = new Map();
  for (const b of SKILL) {
    if (!m.has(b.source)) m.set(b.source, { total: 0, covered: 0 });
    const s = m.get(b.source);
    s.total += 1;
    if (covered.has(b.name)) s.covered += 1;
  }
  return m;
}

Eighteen Skills, Four Environments

// lab/measure.mjs — four environment types and a real assessment, across eighteen skills.
import { SKILL, ENVIRONMENT, LAB, union, perLesson } from './scope.mjs';

const y = (g, ...s) => console.log(s.map((v, i) =>
  (g[i] < 0 ? String(v).padEnd(-g[i]) : String(v).padStart(g[i]))).join('').trimEnd());

console.log(`${SKILL.length} skill items, ${Object.keys(ENVIRONMENT).length} environment types; ` +
  `skills were drawn from this topic's six lessons and from process`);

const A = [-38, 10, 11, 28];
console.log('\n1. coverage per environment');
y(A, 'environment', 'covered', 'setup h', 'legal condition');
for (const [o, v] of Object.entries(ENVIRONMENT))
  y(A, o, `${v.covers.length}/${SKILL.length}`, v.setupHours, v.legal);

const lab = union(LAB);
const open = SKILL.filter((b) => !lab.has(b.name));

const B = [-10, 9, 11, 12];
console.log('\n2. coverage per source lesson (union of three lab types)');
y(B, 'source', 'skill', 'covered', 'not covered');
for (const [k, s] of perLesson(lab)) y(B, k, s.total, s.covered, s.total - s.covered);

const C = [-8, -32, 8, 17];
console.log('\n3. skill covered by no lab');
y(C, 'item', 'skill', 'source', 'where it exists');
for (const b of open) y(C, b.name, b.skill, b.source, 'real only');

console.log('\n4. lab vs. real assessment difference');
console.log(`  union of three lab types             ${lab.size} / ${SKILL.length}`);
console.log(`  difference                           ${open.length} items`);
console.log(`  broadest single lab type              ` +
  `${Math.max(...LAB.map((o) => ENVIRONMENT[o].covers.length))} / ${SKILL.length}`);
console.log(`  distribution's coverage on its own    ${ENVIRONMENT['security-focused-test-distribution'].covers.length}`);
console.log(`  total setup time (four environments)  ` +
  `${LAB.reduce((t, o) => t + ENVIRONMENT[o].setupHours, 0)} hours`);

console.log('\n5. trace and result');
console.log(`  real record produced in the lab       0`);
console.log(`  real risk carried in the lab          0`);
console.log(`  authorization condition in a real assessment  ` +
  `${ENVIRONMENT['authorized-real-assessment'].legal}`);
console.log(`  source of the ${open.length} uncovered items: ` +
  `${[...new Set(open.map((b) => b.source))].join(', ')}`);

18 skill items, 5 environment types; skills were drawn from this topic's six lessons and from process

1. coverage per environment
environment                              covered    setup h             legal condition
self-built-isolated-lab                    10/18         24           your-own-hardware
prebuilt-vulnerable-image-set               6/18          3           your-own-hardware
scenario-based-practice-environment        10/18          0       provider-terms-of-use
security-focused-test-distribution          0/18          1           your-own-hardware
authorized-real-assessment                 18/18          0  written-contract-and-scope

2. coverage per source lesson (union of three lab types)
source        skill    covered not covered
01                2          2           0
02                2          2           0
03                3          3           0
04                2          2           0
05                2          1           1
06                3          3           0
process           4          0           4

3. skill covered by no lab
item    skill                             source  where it exists
b11     privacy-class-decision                05        real only
b15     scope-negotiation                process        real only
b16     client-communication             process        real only
b17     production-impact-carrying       process        real only
b18     finding-reporting                process        real only

4. lab vs. real assessment difference
  union of three lab types             13 / 18
  difference                           5 items
  broadest single lab type              10 / 18
  distribution's coverage on its own    0
  total setup time (four environments)  28 hours

5. trace and result
  real record produced in the lab       0
  real risk carried in the lab          0
  authorization condition in a real assessment  written-contract-and-scope
  source of the 5 uncovered items: 05, process

A Distribution Is Not an Environment

The first table’s clearest row is its fourth: a security-focused test distribution covers zero of eighteen skill items. This is not a shortfall, it is a matter of definition. A distribution is a toolset; it cuts the time to install the tools down to an hour and stops there. Having installed the tools is not a skill; the skill is being able to separate the candidate a tool produces from evidence, and that requires a target, a scope, and a result.

The remaining three rows show an unexpected relationship between setup and coverage. Your own lab, which needs twenty-four hours of setup, covers 10 items; the scenario-based environment, which needs zero setup, also covers 10 items. But the items they cover are not the same. In your own lab, because you build the topology yourself, capture-point selection and field-class separation can be worked through; in a prebuilt scenario the topology is given, and those two items drop out. In return, because a scenario carries a scope definition, scope-check running, target-identity verification, and majority-threshold selection can be worked through; your own lab has no concept of scope, because everything in it is yours.

When the two are used together, the union rises to 13 items. The prebuilt vulnerable image set adds no new item to the union; its contribution is not coverage, it is cheapening repetition through a three-hour setup. The total setup time of twenty-eight hours is the combined cost of the three lab types and the distribution, and it is a one-time cost.

It is worth saying here that the coverage count does not measure a skill. The table says an item can be worked through, not how well it was worked through. Building a validation plan is an item covered in all three environments; having built one plan is not the same as having built twenty and having seen which ones got shorter. This is exactly where the prebuilt image set’s contribution lies: it adds nothing to the coverage column, it cheapens repetition. Coverage is a precondition, not sufficiency.

The second table groups skills by source lesson and shows that almost all of the technical side can be worked through in a lab: in five of six lessons, there is no uncovered item. The one exception is the fifth lesson, and even there the uncovered item is not technical. This distribution confirms that the lab’s limit has nothing to do with tool capability.

The Five Items the Lab Cannot Teach

The third table is this lesson’s result. Five items are covered by no lab, and four of them come straight from process: scope negotiation, client communication, production-impact carrying, finding reporting. The fifth comes from a technical lesson, and stands out for exactly that reason: privacy-class decision.

A lab has no subscriber data. Deciding not to send a file outside is a free choice there; what makes the decision hard is what happens if you send it, and in a lab, nothing happens. The same structure shows up in the other four items. Scope negotiation only means something when there is an organization on the other side. Carrying production impact is only learned when there is a service that actually goes down. Reporting a finding is only learned when someone reading the report has their budget shift.

The fifth block gives this in a single line: the real record produced in the lab is zero, the real risk carried is zero. What the lab cannot teach is the reason it is a lab. Zero risk makes the exercise legal and repeatable; that same zero leaves consequential decisions off the list.

The practical conclusion here is not about environment choice. Even if all three lab types are used, five items stay open, and these items come not from the next environment but from a process. The authorized real assessment row’s coverage is 18, but its condition is a written contract and a defined scope; doing the same work without that condition is not practice, it is a crime.

What the Lab Gives on Its Own

The difference list should not be read one-sidedly. The lab has an advantage the real assessment does not, and it connects directly to this topic’s measurement axis: in a lab, the actual state is known.

Every model in this topic carried an actual field — whether the item was open, whether it was bad. The model carried that field only so it could compute the result; a real assessment has no such field. When a candidate is not proven, it is unknown whether the vulnerability exists and just could not be shown, or never existed at all. In a lab, the target’s state is known in advance, so it is possible to measure your own false-positive and false-negative rate.

This is the only measurable form of practice. How many of ten targets got the right decision, how many candidates got needlessly validated, which step got skipped how many times — all countable in a lab; in a real assessment it is only understood years later, when an incident is read back retroactively. The 13-item section of the coverage table is therefore worked through not as “less” but as “measurable.”

A rule of use follows from this. The lab’s job is not to see a technique, it is to learn your own error rate at that technique. Running the same target set twice and reaching the same decision both times teaches nothing; changing the target set and comparing the decision distribution does.

Legal Limit and Terms of Use

The three lab types do not share a legal condition. In an environment set up on your own hardware, the limit is the limit of your own hardware; even a single outbound connection crosses it, and that is the reason isolation is built. In a scenario-based environment the limit is drawn by the provider’s terms of use: which targets can be touched, which techniques are forbidden, and how a found vulnerability is to be reported are written there. Reading that text is part of the exercise itself — because its counterpart in a real assessment is the rules of engagement, and not reading it produces the same consequence.

Failing to build isolation in your own lab is not just a legal problem; it also breaks the measurement. The fourth lesson showed that an environment hint cannot be hidden under any configuration, because hiding it required the item under examination to reach a real target. The same limit stands the same way in a practice lab: letting one item reach outside removes both the isolation and the legal ground at the same time. Controlling the lab’s outbound connection is part of the exercise itself, and its counterpart in a real assessment is the scope document.

This draws the topic’s seven lessons together into a single line. Tools produced candidates, validation steps turned some of those candidates into evidence, every step left a trace on the defense side, and at every stage a scope limit wrote down how many items it left off the list. The scope limit behaved the same way in all seven lessons: it measured not a tool’s capability, but the permission to use that capability.

Summary

• Thirteen of eighteen skill items are covered by the union of three lab types; no single type exceeds 10, and the difference is five items.
• A security-focused test distribution covers zero skills on its own: a toolset shortens setup time, it does not substitute for a practice lab.
• Your own lab and the scenario-based environment cover the same number of items but different ones: the first gives you the topology, the second gives you a scope definition.
• Four of the five uncovered items come from process, one from a privacy decision; what they share is requiring a decision with a consequence.
• Real record and real risk are zero in a lab; that zero makes the exercise legal and repeatable, and the same zero leaves five items off the list.
• The legal limit varies by environment type: the limit of your own hardware, or a provider’s terms of use. Reading the terms of use is the counterpart of the rules of engagement in a real assessment.

Next Step

This topic counted what seven tool classes produce and what they cannot prove. Packet capture split claims into three classes, the protocol analyzer dropped to metadata in an encrypted flow, the intercepting proxy measured validation in terms of request count and risk class, the sandbox counted suppressed behavior, the reputation service showed the limit of the majority decision, the exploitation framework wrote down what it does not prove, and this lesson measured where all of it can be learned, in terms of scope.

The same column stayed missing in every lesson. The scope limit kicked in again and again — the request to an out-of-scope target was held at zero, two flows were left unresolved, ten items were never queried, two targets were never touched — but who drew that limit, with what authority, and with what document was never written down. Nor was it written down in what form each measured permission, each alert, and each piece of evidence is to be reported.

The next topic, Testing Process, fills this gap: turning the rules of engagement into measurable items, the difference between black-box, gray-box, and white-box assessments, eliminating false positives, expressing a finding in terms of business impact, writing reproduction steps, and the timeline of responsible disclosure.