Protocol Analyzers

The previous lesson classified fields by their producer and passed over encrypted flows in a single line: the payload is invisible, the server-name field is visible. The busiest flows on the fictional measurement network are exactly the encrypted ones. It is not true that analysis comes up empty-handed on these flows; its hands are full in a different way, and this lesson’s subject is how much that “different way” is worth.

A protocol analyzer splits captured bytes into application-layer fields. It does two things: it assumes a format and names the fields according to that format. Neither produces evidence; the first is a guess, the second reads a field if the guess is correct. What is measured is how many flows these two steps hold up on.

Visibility Is Two Columns

A flow has two separate sets of fields. Application fields are readable only if encryption is off and the format is known correctly. Metadata fields are produced under every condition: endpoint addresses, transport number, direction sequence, packet-size sequence, timing interval, flow duration. Encrypted sessions add handshake parameters and the server-name field to this — encryption does not close off these two fields.

Metadata answers its own questions at evidence level: how long the flow lasted, what its volume was, whether it was evenly spaced. These are the capture point’s own observations. The problem starts when metadata is used as if it answered other questions. Looking at the size sequence and saying “the transaction succeeded,” looking at the direction sequence and saying “this was a write operation,” is a surrogate inference: the field’s answer is guessed without reading the field. These inferences are often correct, and they are not evidence.

• AT8. The format catalog, flow list, and claim list are a model; there is no real capture.
• AT9. The analyzer assumes the format from the transport number. If the assumption is wrong, it cannot read the fields and mislabels the flow. Real analyzers also look at content signatures; this model looks only at the number and gives the mislabel count as an upper bound.
• AT10. When a text-based format is mislabeled, a raw text fragment is counted as visible; in a binary format nothing is visible.
• AT11. In an encrypted flow, no application field is read; metadata fields are produced in full.
• AT12. A claim is evidence if its basis comes from a read application field; meta-type claims are also evidence when satisfied from a metadata field. A field-type claim satisfied from a surrogate field is inference.
• AT13. Access to key material is written as end-agent if it is provided by an agent on the organization’s own end, and none if it cannot be provided.
• AT14. Passive analysis produces no record on the target; resolving by intercepting is an active step.

// prot/resolve.mjs — this is a MODEL of protocol-analyzer visibility. There is no real capture,
// real address, or real organization. An analyzer ASSUMES a format from transport information;
// if the assumption is wrong, it cannot read the fields and mislabels the flow.
export const FORMAT = {
  'text-request':          { text: true,  field: ['request-line', 'header-fields', 'body', 'status-code', 'session-id', 'user-field'] },
  'measurement-protocol':  { text: false, field: ['function-code', 'record-address', 'value-field', 'unit-id'] },
  'file-transfer':         { text: true,  field: ['command-name', 'file-name', 'user-field', 'size-field'] },
  'name-resolution':       { text: true,  field: ['query-name', 'record-type', 'response-code'] },
  'binary-sync':           { text: false, field: ['operation-code', 'record-count', 'timestamp'] },
  'encrypted-session':     { text: false, field: [] },
};

// The format the analyzer assumes from the transport number.
export const ASSUMPTION = { 80: 'text-request', 443: 'encrypted-session', 502: 'measurement-protocol',
  21: 'file-transfer', 53: 'name-resolution', 8080: 'text-request' };

// Fields that remain even if encryption is stripped: the analyzer produces these in every flow.
export const META = ['endpoint-addresses', 'transport-number', 'direction-sequence', 'packet-size-sequence',
  'timing-interval', 'flow-duration'];
export const ENCRYPTED_EXTRA = ['server-name-field', 'handshake-parameters'];

// row: flow | transport number | real format | encrypted | access to key material
export const FLOW = `
  s1-end>collector       502 measurement-protocol yes end-agent
  s2-end>collector       502 measurement-protocol yes end-agent
  collector>validator    443 encrypted-session     yes end-agent
  sub-end>portal         443 encrypted-session     yes none
  field-end>field-svc   8080 binary-sync           no  none
  admin>collector          80 text-request         no  none
  admin>validator          80 text-request         no  none
  billing>ext-end         443 encrypted-session    yes none
  collector>billing       502 binary-sync          no  none
  field-end>file-svc       21 file-transfer        no  none
`.trim().split('\n').map((r) => {
  const [name, port, format, encrypted, key] = r.trim().split(/\s+/);
  return { name, port: +port, format, encrypted: encrypted === 'yes', key };
});

// Resolution result: read application field, produced metadata field, mislabel.
export function resolve(a) {
  const assumed = ASSUMPTION[a.port];
  const meta = [...META, ...(a.encrypted ? ENCRYPTED_EXTRA : [])];
  if (a.encrypted) return { read: [], meta, label: assumed, mislabeled: false, depth: 'meta-only' };
  if (assumed === a.format)
    return { read: FORMAT[a.format].field, meta, label: assumed, mislabeled: false, depth: 'full' };
  const partial = FORMAT[a.format].text ? ['raw-text-fragment'] : [];
  return { read: partial, meta, label: assumed, mislabeled: true,
    depth: partial.length ? 'partial' : 'none' };
}

// Claim: a proposition drawn from the analyzer's output.
// 'meta'-type claims are satisfied directly from a metadata field and count as evidence (a point
// observation). 'field'-type claims are proven only from an application field; surrogate is a
// guess made from a metadata field and counts as INFERENCE.
export const CLAIM = [
  { name: 'flow-duration-and-volume', type: 'meta',  basis: ['flow-duration'], surrogate: [] },
  { name: 'regular-interval-present', type: 'meta',  basis: ['timing-interval'], surrogate: [] },
  { name: 'counterparty-name',        type: 'meta',  basis: ['server-name-field'], surrogate: [] },
  { name: 'session-parameters',       type: 'meta',  basis: ['handshake-parameters'], surrogate: [] },
  { name: 'format-spoken',     type: 'field', basis: ['request-line', 'function-code', 'command-name', 'operation-code', 'query-name'], surrogate: ['transport-number'] },
  { name: 'requested-resource', type: 'field', basis: ['request-line', 'query-name', 'file-name'], surrogate: ['server-name-field'] },
  { name: 'transaction-result', type: 'field', basis: ['status-code', 'response-code'], surrogate: ['packet-size-sequence'] },
  { name: 'write-or-read',      type: 'field', basis: ['function-code', 'operation-code', 'command-name'], surrogate: ['direction-sequence'] },
  { name: 'transferred-file',   type: 'field', basis: ['file-name', 'size-field'], surrogate: ['packet-size-sequence'] },
  { name: 'record-count',      type: 'field', basis: ['record-count', 'record-address'], surrogate: ['packet-size-sequence'] },
  { name: 'carried-identity-field',    type: 'field', basis: ['user-field', 'session-id'], surrogate: [] },
  { name: 'written-measurement-value', type: 'field', basis: ['value-field'], surrogate: [] },
];

// A claim's status for a given flow: evidence | inference | unmet
export function status(claim, res) {
  if (claim.basis.some((g) => res.read.includes(g))) return 'evidence';
  if (claim.type === 'meta' && claim.basis.some((g) => res.meta.includes(g))) return 'evidence';
  if (claim.surrogate.some((g) => res.meta.includes(g))) return 'inference';
  return 'unmet';
}

export function measure() {
  const s = { defined: 0, read: 0, meta: 0, mislabeled: 0, candidate: 0, evidence: 0, inference: 0 };
  for (const a of FLOW) {
    const c = resolve(a);
    s.defined += FORMAT[a.format].field.length;
    s.read += c.read.length;
    s.meta += c.meta.length;
    if (c.mislabeled) s.mislabeled += 1;
    for (const claim of CLAIM) {
      const d = status(claim, c);
      if (d !== 'unmet') { s.candidate += 1; s[d] += 1; }
    }
  }
  return s;
}

Ten Flows, Two Columns

// prot/visibility.mjs — ten flows are run through the analyzer; field and meta are counted separately.
import { FLOW, FORMAT, CLAIM, resolve, status, measure } from './resolve.mjs';

const y = (g, ...s) => console.log(s.map((v, i) =>
  (g[i] < 0 ? String(v).padEnd(-g[i]) : String(v).padStart(g[i]))).join('').trimEnd());

const A = [-21, -22, 11, 10, 8, 6, 22, 11];
console.log('1. resolution depth per flow');
y(A, 'flow', 'real format', 'transport', 'encrypted', 'defined', 'read', 'label', 'depth');
for (const a of FLOW) {
  const c = resolve(a);
  y(A, a.name, a.format, a.port, a.encrypted ? 'yes' : 'no', FORMAT[a.format].field.length,
    c.read.length, c.label, c.depth);
}

const B = [-27, 6, 10, 11, 7];
console.log('\n2. satisfaction per claim type (10 flows)');
y(B, 'claim', 'type', 'evidence', 'inference', 'unmet');
for (const claim of CLAIM) {
  const d = FLOW.map((a) => status(claim, resolve(a)));
  y(B, claim.name, claim.type, d.filter((x) => x === 'evidence').length,
    d.filter((x) => x === 'inference').length, d.filter((x) => x === 'unmet').length);
}

const s = measure();
console.log('\n3. totals');
console.log(`  defined application fields    ${s.defined}`);
console.log(`  read application fields       ${s.read}`);
console.log(`  produced metadata fields      ${s.meta}`);
console.log(`  mislabeled flows              ${s.mislabeled} / ${FLOW.length}`);
console.log(`  produced claim candidates     ${s.candidate} / ${CLAIM.length * FLOW.length}`);
console.log(`    evidence                    ${s.evidence}`);
console.log(`    inference                   ${s.inference}`);
console.log(`  flows with a wrong format guess ${s.mislabeled}`);

const encrypted = FLOW.filter((a) => a.encrypted);
console.log('\n4. options for seeing inside an encrypted flow');
console.log(`  encrypted flows                       ${encrypted.length} / ${FLOW.length}`);
console.log(`  key access via an end-agent           ${encrypted.filter((a) => a.key !== 'none').length}`);
console.log(`  unresolvable without interception     ${encrypted.filter((a) => a.key === 'none').length}`);
console.log(`  trace passive analysis leaves on target    0 records`);
console.log(`  trace interception leaves on target        1 alert per flow (active step)`);

1. resolution depth per flow
flow                 real format             transport encrypted defined  read                 label      depth
s1-end>collector     measurement-protocol          502       yes       4     0  measurement-protocol  meta-only
s2-end>collector     measurement-protocol          502       yes       4     0  measurement-protocol  meta-only
collector>validator  encrypted-session             443       yes       0     0     encrypted-session  meta-only
sub-end>portal       encrypted-session             443       yes       0     0     encrypted-session  meta-only
field-end>field-svc  binary-sync                  8080        no       3     0          text-request       none
admin>collector      text-request                   80        no       6     6          text-request       full
admin>validator      text-request                   80        no       6     6          text-request       full
billing>ext-end      encrypted-session             443       yes       0     0     encrypted-session  meta-only
collector>billing    binary-sync                   502        no       3     0  measurement-protocol       none
field-end>file-svc   file-transfer                  21        no       4     4         file-transfer       full

2. satisfaction per claim type (10 flows)
claim                        type  evidence  inference  unmet
flow-duration-and-volume     meta        10          0      0
regular-interval-present     meta        10          0      0
counterparty-name            meta         5          0      5
session-parameters           meta         5          0      5
format-spoken               field         3          7      0
requested-resource          field         3          5      2
transaction-result          field         2          8      0
write-or-read               field         1          9      0
transferred-file            field         1          9      0
record-count                field         0         10      0
carried-identity-field      field         3          0      7
written-measurement-value   field         0          0     10

3. totals
  defined application fields    30
  read application fields       16
  produced metadata fields      70
  mislabeled flows              2 / 10
  produced claim candidates     91 / 120
    evidence                    43
    inference                   48
  flows with a wrong format guess 2

4. options for seeing inside an encrypted flow
  encrypted flows                       5 / 10
  key access via an end-agent           3
  unresolvable without interception     2
  trace passive analysis leaves on target    0 records
  trace interception leaves on target        1 alert per flow (active step)

The Analyzer’s Own Interpretation

The last two columns of the first table are this lesson’s quietest finding. Of the 30 defined application fields, only 16 are read. The loss comes from two sources: five flows are encrypted, and two flows are mislabeled.

The mislabel is not a flaw in the tool; it is the result of an assumption. Field synchronization speaks a binary format but listens near the customary text-request number; the analyzer assumes a text request and cannot read any of the three defined fields. The flow from collector to billing runs a binary sync over the measurement number; the analyzer stamps a measurement-protocol label on it. In both cases the output is not empty — the output is wrong. If the line “this flow speaks the measurement protocol” goes into the report, that line is a false positive, and its source is the analyzer itself, not the system on the target.

In the second table, the format-spoken row gives this as a number: in three flows the format is confirmed from an actually-read field, in seven it is inferred only from the transport number, and two of those seven are wrong. The way to turn an inference into evidence is to actually read a field that gives the format away.

These two flows have a second reading, and it is written to the defense side. A service speaking outside its customary number is easy to miss in an inventory, and it misleads every control that classifies by looking at the number in the same way. What misleads the assessment’s analyzer also misleads the defense’s classifier; the finding is not written as “the analyzer was wrong,” but as “these two flows do not fit any number-based classification.”

What Metadata Says and Does not Say

Ninety-one candidates were produced; 43 are evidence, 48 are inference. The inference column is nearly half, and its content shows a clear pattern. record-count is inference in ten of ten flows, evidence in zero: looking at the packet-size sequence and saying “this many records were transferred” is possible in every flow and evidence in none. transaction-result is inference in eight flows — reading success from the response size is an old habit, and it looks like the only option in encrypted flows.

By contrast, two rows produce no inference at all. carried-identity-field is evidence in three flows and unmet in seven; written-measurement-value is unmet in ten of ten flows. These two rows are where encryption really holds: the content of the identity field and the written measurement value cannot be derived from metadata — they have no surrogates. Writing “unmet” in the assessment report is not a shortcoming, it is the correct answer; writing an inference as evidence is the shortcoming.

The four meta-type claims produce 30 pieces of evidence in total, and 10 of these come only from encrypted flows — the server-name field and handshake parameters. While encryption closes off the application fields, it leaves these two fields open and provides a surrogate for the requested-resource claim in five flows. Encrypted traffic does not close off the whole question; it keeps saying who was talked to and for how long.

The Cost of Seeing Inside the Encryption

In three of the five encrypted flows, access to key material sits on the organization’s own end: the measurement ends and the collector are devices the organization manages, and an agent on the end makes analysis possible within scope. The remaining two are the subscriber-portal session and the external-notification flow. The first has a subscriber on the other end, the second has another organization. The only way to see inside these two flows is to intercept; intercepting means stopping being passive, producing records and alerts on the target, and touching a third party’s traffic. If written authorization and defined scope do not cover this, the step is both illegal and unprofessional, and the two flows are left unresolved.

The trace left on the defense side also splits in two here. Passive analysis produces zero records, as in the previous lesson. Resolving by intercepting produces at least one alert per flow — the established trust chain differing from what is expected is the first thing the control on the end will see. This is the first non-zero trace column in the course, and the reason is not that the tools changed, but that the step has become active.

Summary

• Of the 30 application fields defined across ten flows, 16 are read; five flows’ worth of the loss comes from encryption, two flows’ worth from the format assumption not holding.
• The analyzer assumes the format from transport information; two flows are mislabeled and the output is not empty but wrong. This is a source of false positives that comes from the tool, not the target.
• Of 91 candidates, 43 are evidence and 48 are surrogate inference drawn from metadata; record-count is inference in ten of ten flows and evidence in zero.
• The content of the identity field and the written measurement value can be derived from metadata in no flow; these two rows are where encryption really holds.
• Three of the five encrypted flows have in-org access to the key; two are left unresolved because they involve a subscriber and a third party.
• Passive analysis leaves zero records; resolving by intercepting produces at least one alert per flow and moves the step into the active class.

Next Step

So far, both tools were only watching tools: they recorded the traffic that passed and read the fields they could read. Both share the same limit: seeing that a field passed with a particular value in a flow does not say what the system will do when a different value is written to that field. Showing that a candidate is really open requires changing the request and seeing the response.

The next lesson treats the intercepting proxy as a validation step and asks a single question: how many requests does it take to show that a candidate is open. Requests are split into three risk classes — observation-only, state-changing on the target, and carrying a risk of service disruption — and each candidate’s validation plan is counted in terms of these classes. Scope checking also comes into play for the first time: some requests cannot be sent at all because they touch a party outside scope, and those candidates remain unturned into evidence.