Subdomain and Asset Discovery

The previous two lessons worked on a closed list: twenty candidates came out of publicly available items and were classified by record lookup. Under an organization’s domain name, though, sit names that appear in no document and have a link on no page. This lesson searches for those names and counts two things at once while doing it: the methods’ coverage, and records that point to a target that no longer exists.

That a zone file is kept as an inventory and that orphaned records are cleaned up was covered from the zone owner’s side in the Endpoint and System Hardening course. Here the same zone is viewed from the outside: an item the owner has not cleaned up because they do not know about it sits as a candidate on the assessing party’s list.

Dangling Record and the Forgotten Item

RE7: each name carries three things — whether its target responds, whether it is in the inventory, and its responsible team. A record whose target does not respond is dangling: the record stands, but what it pointed to has fallen away. A record whose target responds but has no owner is forgotten: it works, but no one is responsible for it. RE8: a method produces a candidate set; if a candidate has no counterpart in the zone, it is a false positive. RE9: a method’s trace is the record it leaves on the target and the number of alerts it triggers; an attempt count can produce trace even when it produces no candidate.

All of these steps are limited to the domain name in the scope definition and cannot be done in a name space that falls outside written authorization.

// recon/zone.mjs — a fictional measurement network's subdomain records and three discovery
// methods' findings (a model): no real domain name, no real name server, no real target. No
// randomness.

// RE7: each row is a name — whether its target responds, whether it is in the inventory, its
// responsible team, and the methods that found it. A record whose target does not respond counts
// as dangling; one whose target responds but has no owner counts as forgotten. In the last
// column p is passive record compilation, s is dictionary attempt, b is linked-item inference;
// "-" means that method did not find it.
export const NAME = `
  subscriber      live      1 portal        psb
  field           live      1 field         psb
  vpn             live      1 network       ps-
  collector       live      1 measurement   p--
  report          live      1 data          psb
  test            live      1 portal        psb
  shared          live      1 portal        p-b
  work-order      live      1 field         p-b
  meter           live      1 measurement   p--
  old-subscriber  live      0 -             p--
  marketing       live      0 marketing     -sb
  training        live      0 -             -s-
  demo            dead      0 -             ps-
  old-report      dead      0 -             p--
  campaign-2      dead      0 -             p--
  archive         dead      0 -             ps-
  staging         live      0 data          ps-
  backup          dead      1 network       -s-
  monitoring      live      1 network       ps-
  log             live      1 data          -s-
  survey          dead      0 -             -s-
  old-field       dead      0 -             ---
  branch-1        live      1 network       ---
  branch-2        live      1 network       ---
  campaign-1      none      0 -             p--`
  .trim().split("\n").map((r) => {
    const [name, target, inv, owner, found] = r.trim().split(/\s+/);
    return { name, target, inv: +inv, owner, found };
  });

// RE8: a method produces a candidate set; if a candidate has no counterpart in the zone, it is a
// false positive. RE9: a method's trace is the record it leaves on the target and the alerts it
// triggers. An attempt count can produce trace even when it produces no candidate: the zone
// transfer request is refused by all four name servers and returns no names at all.
export const METHOD = [
  { code: "p", name: "passive compilation", attempts: 0, records: 0, alerts: 0 },
  { code: "s", name: "dictionary attempt", attempts: 120, records: 120, alerts: 1 },
  { code: "b", name: "linked-item inference", attempts: 12, records: 12, alerts: 0 },
  { code: "z", name: "zone transfer request", attempts: 4, records: 4, alerts: 4 },
];

export const RECORD = NAME.filter((a) => a.target !== "none");
export const finds = (y, a) => a.found.includes(y.code);
export const dangling = (a) => a.target === "dead";
export const forgotten = (a) => a.target === "live" && a.owner === "-";

// recon/asset.mjs — three discovery methods over the same name space: how many candidates, how
// many dangling items, how much trace.
import { NAME, METHOD, RECORD, finds, dangling, forgotten } from "./zone.mjs";

const write = (g, ...s) => console.log(s.map((v, i) =>
  (g[i] < 0 ? String(v).padEnd(-g[i]) : String(v).padStart(g[i]))).join(""));
const DANGLING = RECORD.filter(dangling).length;

console.log(`${RECORD.length} zone records: ${RECORD.filter((a) => a.target === "live").length} ` +
  `targets respond, ${DANGLING} dangling, ${RECORD.filter(forgotten).length} forgotten, ` +
  `${RECORD.filter((a) => !a.inv && a.target === "live").length} outside inventory`);

const A = [-23, 9, 7, 8, 9, 10, 10];
console.log("\n1. candidates and dangling items per method");
write(A, "method", "attempts", "cand", "true", "falseP", "dangling", "cover %");
for (const y of METHOD) {
  const l = NAME.filter((a) => finds(y, a));
  const d = l.filter((a) => a.target !== "none");
  write(A, y.name, y.attempts, l.length, d.length, l.length - d.length, d.filter(dangling).length,
    ((d.length / RECORD.length) * 100).toFixed(1));
}

const B = [-23, 8, 8, 11, 15, 14];
console.log("\n2. the trace a method leaves");
write(B, "method", "records", "alerts", "rec/cand", "rec/dangling", "cand/attempt");
for (const y of METHOD) {
  const d = NAME.filter((a) => finds(y, a) && a.target !== "none");
  const sz = d.filter(dangling).length;
  write(B, y.name, y.records, y.alerts, d.length ? (y.records / d.length).toFixed(2) : "-",
    sz ? (y.records / sz).toFixed(2) : "-",
    y.attempts ? (d.length / y.attempts).toFixed(2) : "-");
}
const TRACE = METHOD.reduce((t, y) => t + y.records, 0);
const ALERTS = METHOD.reduce((t, y) => t + y.alerts, 0);

const C = [-23, 9, 9, 11, 11];
console.log("\n3. combination and overlap");
write(C, "set", "cand", "true", "dangling", "cover %");
const set = [];
for (const y of METHOD) {
  set.push(y);
  const l = NAME.filter((a) => set.some((x) => finds(x, a)));
  const d = l.filter((a) => a.target !== "none");
  write(C, `+ ${y.name}`, l.length, d.length, d.filter(dangling).length,
    ((d.length / RECORD.length) * 100).toFixed(1));
}
const howMany = (a) => METHOD.filter((y) => finds(y, a)).length;
const D = [-23, 9, 11, 12];
console.log("\n4. how many methods find it");
write(D, "method count", "records", "dangling", "forgotten");
for (const n of [0, 1, 2, 3]) {
  const l = RECORD.filter((a) => howMany(a) === n);
  write(D, `${n} methods`, l.length, l.filter(dangling).length, l.filter(forgotten).length);
}
const found = RECORD.filter((a) => howMany(a) > 0);
console.log(`record no method found: ` +
  RECORD.filter((a) => !howMany(a)).map((a) => a.name).join(", "));
console.log(`combination finds ${found.length}/${RECORD.length} records (` +
  `${((found.length / RECORD.length) * 100).toFixed(1)}%) and ` +
  `${found.filter(dangling).length}/${DANGLING} dangling items; ` +
  `total trace ${TRACE} records, ${ALERTS} alerts`);

24 zone records: 17 targets respond, 7 dangling, 2 forgotten, 4 outside inventory

1. candidates and dangling items per method
method                  attempts   cand    true   falseP  dangling   cover %
passive compilation            0     17      16        1         4      66.7
dictionary attempt           120     14      14        0         4      58.3
linked-item inference         12      7       7        0         0      29.2
zone transfer request          4      0       0        0         0       0.0

2. the trace a method leaves
method                  records  alerts   rec/cand   rec/dangling  cand/attempt
passive compilation           0       0       0.00           0.00             -
dictionary attempt          120       1       8.57          30.00          0.12
linked-item inference        12       0       1.71              -          0.58
zone transfer request         4       4          -              -          0.00

3. combination and overlap
set                         cand     true   dangling    cover %
+ passive compilation         17       16          4       66.7
+ dictionary attempt          22       21          6       87.5
+ linked-item inference       22       21          6       87.5
+ zone transfer request       22       21          6       87.5

4. how many methods find it
method count             records   dangling   forgotten
0 methods                      3          1           0
1 methods                      9          4           2
2 methods                      8          2           0
3 methods                      4          0           0
record no method found: old-field, branch-1, branch-2
combination finds 21/24 records (87.5%) and 6/7 dangling items; total trace 136 records, 5 alerts

The numbers belong to the measurement class; their inputs are RE7–RE9.

Seven Dangling Records

Twenty-four zone records — 7 of them are dangling: the record stands, but the target it points to does not respond. The rate is 29.2% — one in every three records in the zone points at nothing. Alongside these there are 2 forgotten items whose target responds but has no owner, and 4 items that never appear in the inventory at all yet still run.

These three classes do not carry the same weight for assessment purposes. The four items outside the inventory are a scope problem: they run, they serve the organization, but their names never came up in the scope conversation. The two forgotten items are a responsibility gap: if a finding comes out of them, who the report gets written to stays unclear. The seven dangling records are the most attention-grabbing, though, because a record is a promise binding a name to a target, and when the target disappears, that promise stays open to whichever other party is ready to claim it. The party keeping the record is the organization; the party providing the target is no one anymore.

Three Methods, Three Different Coverages

The first table runs all three methods over the same name space. Passive record compilation produces 17 candidates, 16 of them have a counterpart in the zone, and its coverage is 66.7%. Dictionary attempt produces 14 candidates and covers 58.3%. Linked-item inference produces only 7 candidates — 29.2%.

The difference does not come from the methods’ quality, it comes from what they look at. Linked-item inference follows links running from one page to another; a name is found only if someone linked to it. Dangling items are exactly the items no one links to, which is why this method finds none of the seven dangling records. In the same column, passive compilation produces 4 and dictionary attempt produces 4 dangling items. A method is eliminated not because it is “the one producing the fewest candidates,” but because it is structurally blind to the class it is looking for.

The false-positive column is filled by a single item. campaign-1, produced by passive compilation, does not exist in the zone at all: it was once entered into a certificate record, and the name was later removed from the zone. This property of passive sources is the same as the archived copy from the previous lesson — something that was true in the past does not have to be true today.

One Hundred Twenty Attempts, Fourteen Names

The second table ties this course’s trace column to a method choice for the first time. Passive record compilation leaves zero records on the target and is the method that finds the most candidates. Dictionary attempt makes 120 attempts, finds 14 names, and leaves 120 records on the target; 8.57 records per name, 30.00 records per dangling item found. Linked-item inference leaves 12 records and produces 1.71 trace per name.

Dictionary attempt’s hit rate shows up in the cand/attempt column: 0.12. A hundred and six of the hundred twenty attempts go to waste, and the wasted attempts leave records too. Read from the defense side: the blue team notices this method not by looking at queries for names that exist, but by looking at the density of queries for names that do not exist. The single alert produced across this lesson is triggered by exactly that density.

The fourth row is the lesson’s sharpest comparison. The zone transfer request is refused by four name servers: 0 candidates, 4 records, 4 alerts. Four of the total five alerts come from this single step that returns no names at all. Had it been allowed, it would have handed over the entire zone in one step; when it is not, all that is left is trace. A step attempted because its expected return is high becomes pure cost once its realized return turns out to be zero.

Where the Combination Stops

The third table combines the methods. The first two together find 21 records and coverage climbs from 66.7% to 87.5%; dangling items rise from 4 to 6. When the third and fourth methods are added, all three numbers do not change at all. All seven names linked-item inference finds have already come out of the first two methods.

The fourth table shows this item by item. Of the twenty-four records, 4 come from all three methods, 8 from two, 9 from only one, and 3 from none. Of the nine records that come from only one method, 4 are dangling and 2 are forgotten: the items the organization is least aware of are the items the fewest methods see. None of the four records that come from all three methods is dangling or forgotten; they are the zone’s most visible, best-maintained names.

The three records no method finds mark the list’s boundary. branch-1 and branch-2 are numbered names with no dictionary counterpart; old-field is both dangling and has left no trace in any passive record. So six of the seven dangling items are found, one is not — and in this run, that it is not found is only known because the referee inventory is on hand. In a real assessment, this record’s existence would never be known at all; the only thing known would be the 21 names.

A Dangling Record Is not Evidence Yet

Finding six dangling records means, by this course’s measure, six candidates, not six findings. A record’s target might not respond because the target is temporarily down, because the name only resolves from inside the organization’s network, or because the space the target sits in is available for another party to reclaim. Only the third case is a finding that belongs in a report; the first two are false positives, and none of the discovery steps can tell the three apart.

The cost of making that distinction is two items. The first is an extra step: for every dangling record, whether the service the target sits in is open to reclaiming has to be tested. The second, and more restrictive, is scope: that test is run not on the organization’s infrastructure but on the third party’s. None of the targets the six dangling records point to is an asset named in the scope definition.

This is the second encounter with the rule written for the lead class in the previous lesson, and it is sharper here: finding an item is unrestricted, confirming it requires separate authorization. How many of the six candidates turn into evidence is not settled in this lesson — it is settled after the scope conversation.

Summary

• Of twenty-four zone records, 7 are dangling (29.2%): the record stands, its target does not respond. Alongside them sit 2 forgotten and 4 outside-inventory items.
• The three methods’ coverage is 66.7%, 58.3%, and 29.2%; linked-item inference finds none of the seven dangling records, because a dangling item is exactly one no one links to.
• Passive record compilation leaves 0 records on the target and finds the most candidates; its single false positive is a name already removed from the zone.
• Dictionary attempt finds 14 names with 120 attempts (0.12 hit rate), leaves 8.57 records per name and 30.00 per dangling item found; one of this course’s five alerts comes from its density.
• The zone transfer request brings back 0 candidates and leaves 4 records and 4 alerts: 80% of all alerts come from this single step that produces no names.
• The combination plateaus at 87.5% after two methods; of the 9 records from a single method, 4 are dangling and 2 are forgotten, while none of the 4 records from all three methods are.

Next Step

At the end of three lessons the externally visible name list is ready: twenty-one names, six of them dangling. A name is not an asset, though. The name subscriber resolves to a target, but what runs on that target has never been asked until now. The machine a record points to might accept no connections at all; it might serve three services at once; or it might keep a service reachable only from the organization’s network closed off from the outside. All three look identical in the same record.

The step that tells these apart is testing which ports the target responds on, and this step differs from the previous three: it sends requests directly and in large numbers to the target. The next lesson compares scan types on a single axis — how many candidates each type finds, how many it misses, how many records and alerts it produces on the defense side. This lesson accumulated 136 records and 5 alerts; in that lesson both numbers move into a different order of magnitude.