Secrets and Data Management

The previous lesson tied the production-like environment to a cost but did not ask what goes into that environment. The data side was measured in the Test Data Management lesson: masking looks at the content of personal fields, and what property it preserves was counted there. Access credentials, identity, signing keys, and external-service tokens were left outside that measurement. These are secrets, and they cannot be masked — a masked version of a secret does not work, so it either goes in or it does not.

The resource this lesson distributes is authorization: how many secrets, with how much scope, for how long, enter the test environment. The distribution has two returns — every secret that does not enter misses a defect class, every secret that does enter opens a leak surface.

Leak Surface

Leak surface is not an estimate, it is a countable quantity: how many of the artifacts a run leaves behind carry the secret itself. The run below actually calls the notification boundary and triggers the defect measured in the previous course — the unknown recipient. Two token forms are tried: an indefinitely valid persistent token and a token issued only for that run.

// leak.mjs -- how many places a token appears in run artifacts, and its value after the run
import { createServer } from 'node:http';

const valid = new Set(['B-persistent-9f21', 'B-run-4a7c']);
const members = new Set(['U-17']);
const server = createServer((req, res) => {
  const token = req.headers['x-token'] ?? '';
  const recipient = new URL(req.url, 'http://y').searchParams.get('recipient');
  const [status, body] = !valid.has(token)
    ? [403, `{"error":"invalid token"}`]
    : members.has(recipient) ? [202, '{"status":"queued"}']
      // the diagnostic body echoes the request's context back: this is the third leak point
      : [404, `{"error":"unknown recipient: ${recipient}","token":"${token}"}`];
  res.writeHead(status, { 'content-type': 'application/json' });
  res.end(body);
});
server.listen(8941, '127.0.0.1');

const BASE = 'http://127.0.0.1:8941/notify';
const call = (token, recipient) => fetch(`${BASE}?recipient=${recipient}`, { headers: { 'x-token': token } });

// The run triggers the defect measured in the Test Environment Management lesson: unknown recipient.
async function run(token) {
  const artifact = [`STEP loan-flow x-token=${token}`];        // 1: run record
  const res = await call(token, 'U-99');
  artifact.push(`ERROR ${await res.text()}`);                  // 2: the other side's diagnostic body
  artifact.push(`RETRY 1/1 x-token=${token} outcome=${res.status}`);   // 3: retry summary
  return { artifact, leaked: artifact.filter((s) => s.includes(token)).length };
}

const persistent = await run('B-persistent-9f21');
const runScoped = await run('B-run-4a7c');
valid.delete('B-run-4a7c');                                    // validity window closed
const after = async (token) => (await call(token, 'U-17')).status;
const persistentAfter = await after('B-persistent-9f21');
const runAfter = await after('B-run-4a7c');
server.close();

console.log(`persistent token: ${persistent.artifact.length} artifacts, value present in ${persistent.leaked}`);
console.log(`run-scoped token: ${runScoped.artifact.length} artifacts, value present in ${runScoped.leaked}`);
console.log(`leak point count is the same in both forms: ${persistent.leaked === runScoped.leaked}`);
console.log(`post-run request, persistent token: ${persistentAfter}`);
console.log(`post-run request, run-scoped token: ${runAfter}`);
console.log(`artifact not in the team's code: ${runScoped.artifact[1]}`);

persistent token: 3 artifacts, value present in 3
run-scoped token: 3 artifacts, value present in 3
leak point count is the same in both forms: true
post-run request, persistent token: 202
post-run request, run-scoped token: 403
artifact not in the team's code: ERROR {"error":"unknown recipient: U-99","token":"B-run-4a7c"}

Three results are worth reading. First, a single failed call leaves three artifacts, and all three carry the token itself. Second, only two of these three points are in the team’s own code; the third is the other side’s diagnostic body, and no scrubbing rule the team writes can anticipate that body. Third, the two token forms leak exactly the same number of points — the leak point count depends on the shape of the error path, not the nature of the token.

The only thing that changes is the value after the run: the persistent token still gets 202, the run-scoped token 403. The lever is not concealment, it is scope and duration.

Secret Catalog

The test environment wants five secrets, and each opens specific defect classes. The numbers come from the previous lesson’s defect set: all thirty-two defects caught at the mid stage are tied to a secret, four at the nightly stage, one at the release stage.

TP14 — the access counts under production scope are an assumption; the ones under narrow scope are a measurement. Narrow scope is the subset the secret actually touches, and it was read from the test set the Test Data Management lesson measured: 772 rows, 200 members. The numbers on the production side are assumptions about the same system’s production scale and carry an order of magnitude.

TP15 — in the time-limited arrangement, the validity window is 44 minutes, and the setup’s manual steps are counted per secret. The window is the sum of the mid stage’s 41-minute run and the ephemeral environment’s 3-minute setup; it cannot be shorter than the run, because authorization has to end when the run ends. The manual steps are three items: opening one account per secret, writing one scope list under narrow scope, setting up one mechanism for automated issuance. These are paid once per period, while a per-run manual step is paid again on every run.

// secrets.mjs -- the secret catalog and authorization arrangements
// Per-stage feedback time and the caught defect count not tied to a secret; both read from
// the previous lesson's "production-like heavy" row.
export const stages = {
  fast: { feedback: 8, free: 28 },
  mid: { feedback: 49, free: 0 },
  nightly: { feedback: 425, free: 5 },
  release: { feedback: 2357, free: 5 },
};
export const MISSED_FLOOR = 5;     // two classes no test sees
export const MANUAL = 15;          // TP6: one manual step is 15 minutes
export const RUNS = { mid: 100, nightly: 20, release: 4 };

// [name, stage, defect count opened, is write required, prod access, narrow-scope access]
export const secrets = [
  ['catalog ID', 'mid', 13, true, 240000, 772],
  ['notification token', 'mid', 10, false, 18000, 200],
  ['payment key', 'mid', 9, true, 9000, 41],
  ['performance environment ID', 'nightly', 4, true, 240000, 772],
  ['metrics store ID', 'release', 1, true, 4800, 24],
].map(([name, stage, count, write, prod, narrow]) => ({ name, stage, count, write, prod, narrow }));

// An arrangement gives, for each secret: does it enter the environment, how many records can it
// access, can it write; validity window (minutes), setup manual steps, per-run manual steps.
export const arrangements = [
  { name: 'production identity', enters: 1, access: (g) => g.prod, writes: 1, window: Infinity, setup: 0, perRunManual: 0 },
  { name: 'separate account', enters: 1, access: () => 772, writes: 1, window: Infinity, setup: 5, perRunManual: 0 },
  { name: 'read only', enters: 1, access: () => 772, writes: 0, window: Infinity, setup: 5, perRunManual: 0 },
  { name: 'narrow scope', enters: 1, access: (g) => g.narrow, writes: 1, window: Infinity, setup: 10, perRunManual: 0 },
  { name: 'narrow scope, time-limited', enters: 1, access: (g) => g.narrow, writes: 1, window: 44, setup: 10, perRunManual: 1 },
  { name: 'time-limited, automated issuance', enters: 1, access: (g) => g.narrow, writes: 1, window: 44, setup: 11, perRunManual: 0 },
  { name: 'mock', enters: 0, access: () => 0, writes: 0, window: 0, setup: 0, perRunManual: 0 },
];

Seven Authorization Arrangements

The least privilege principle is tied to a number here: exposure surface is the total number of records the leaked secrets can reach. If a secret does not get the write permission it requires, the defect classes it opens escape; the same happens if it does not enter the environment at all.

// scope.mjs -- seven authorization arrangements: leak surface, exposure, manual steps, escaped defects
import { stages, secrets, arrangements, MISSED_FLOOR, MANUAL, RUNS } from './secrets.mjs';

const LEAK_POINTS = 3;           // measured value: the error path produces three artifacts

function measure(d) {
  const entering = d.enters === 1 ? secrets : [];
  const exposed = entering.reduce((a, g) => a + d.access(g), 0);
  const manualRuns = d.perRunManual * Object.values(RUNS).reduce((a, x) => a + x, 0);

  let extra = 0;
  const feedback = {};
  for (const [name, s] of Object.entries(stages)) {
    if (name !== 'fast') extra += d.perRunManual * MANUAL;
    feedback[name] = s.feedback + extra;
  }

  let missed = MISSED_FLOOR, weight = 0, total = 0;
  for (const [name, s] of Object.entries(stages)) {
    weight += s.free;
    total += feedback[name] * s.free;
  }
  for (const g of secrets) {
    if (d.enters === 0 || (g.write && d.writes === 0)) { missed += g.count; continue; }
    weight += g.count;
    total += feedback[g.stage] * g.count;
  }
  return {
    entering: entering.length, leak: entering.length * LEAK_POINTS, exposed,
    window: d.window, manual: d.setup + manualRuns, feedback: total / weight, missed,
  };
}

const s = (x, n) => String(x).padStart(n);
console.log(`${'arrangement'.padEnd(35)}${s('enters', 8)}${s('leak', 7)}${s('exposure', 12)}`
  + `${s('window', 11)}${s('manual steps', 14)}${s('avg feedback', 14)}${s('missed/100', 12)}`);
for (const d of arrangements) {
  const r = measure(d);
  console.log(`${d.name.padEnd(35)}${s(r.entering, 8)}${s(r.leak, 7)}${s(r.exposed, 12)}`
    + `${s(r.window === Infinity ? 'unlimited' : r.window, 11)}${s(r.manual, 14)}`
    + `${s(r.feedback.toFixed(1), 14)}${s(r.missed, 12)}`);
}

const prod = secrets.reduce((a, g) => a + g.prod, 0);
const narrow = secrets.reduce((a, g) => a + g.narrow, 0);
const writers = secrets.filter((g) => g.write);
console.log(`\nsecrets ${secrets.length}, requiring write ${writers.length}, `
  + `defects they open ${writers.reduce((a, g) => a + g.count, 0)}`);
console.log(`prod scope ${prod} records, narrow scope ${narrow} records, `
  + `narrowing ${(prod / narrow).toFixed(0)}x`);
console.log(`run count ${Object.values(RUNS).reduce((a, x) => a + x, 0)}; `
  + `one manual step per run costs ${Object.values(RUNS).reduce((a, x) => a + x, 0) * MANUAL} minutes`);

arrangement                          enters   leak    exposure     window  manual steps  avg feedback  missed/100
production identity                       5     15      511800  unlimited             0         263.5           5
separate account                          5     15        3860  unlimited             5         263.5           5
read only                                 5     15        3860  unlimited             5         304.7          32
narrow scope                              5     15        1809  unlimited            10         263.5           5
narrow scope, time-limited                5     15        1809         44           134         277.1           5
time-limited, automated issuance          5     15        1809         44            11         263.5           5
mock                                      0      0           0          0             0         371.9          42

secrets 5, requiring write 4, defects they open 27
prod scope 511800 records, narrow scope 1809 records, narrowing 283x
run count 124; one manual step per run costs 1860 minutes

Reading the Table

The first and fourth rows are this lesson’s central measurement. Between production identity and narrow scope, escaped defects (five) and average feedback (263.5 minutes) are identical to the digit; the only place they diverge is exposure surface — 511,800 records against 1,809, two hundred eighty-three times over. Least privilege’s cost in escaped defects is zero. What is paid is the ten manual setup steps, and these are paid once per period. What makes narrowing scope hard is not a measurable trade-off but the deferral of a one-time task.

This number has a condition, and it ties to the second half of this lesson’s title. The narrow scope’s 1,809 records assumes the data in the environment is the masked test set. If a copy of production was loaded into the environment, then no matter how narrow the secret’s scope is, exposure surface equals the environment’s own size: 240,000 records sit there, and every token that can reach that environment reaches them. The two decisions cannot be made separately for this reason — unmasked data defeats least privilege, and narrow authorization does not stand in for masking either.

The third row is narrowing done on the wrong axis. Read-only authority sounds safe, but four of the five secrets require write: the catalog ID changes schema in the migration test, the payment key updates card status, the metrics IDs write results. The twenty-seven defects these four open escape, and escaped defects rise from five to thirty-two. Exposure surface does not shrink at all — it stays at 3,860, because the value still enters the environment and still leaks from three points. This row loses on both axes at once.

Here the average feedback column shows the opposite of the trap in the previous two lessons. Under read-only, the average rises from 263.5 to 304.7, and to 371.9 under mock. The reason is that the missed defects are the cheap ones caught: once the mid stage’s forty-nine- minute defects drop out of the denominator, only the expensive ones caught remain, and the average rises. The same column can lie by improving in one lesson and worsening in another; in both, what exposes the lie is the escaped defects column.

The last row is the only arrangement that truly zeroes the leak surface. No secret enters, the leak point count is zero, exposure surface is zero — and escaped defects are forty-two. Mocking the external service was built as a test form in the Mocking External Services lesson; here it shows up as an authorization decision, and its cost is thirty-seven defects. An arrangement that wants to zero out the leak surface ends up closing off the test itself.

The fifth and sixth rows isolate time-limited authorization’s cost. The window dropping from unlimited to 44 minutes changes neither the leak point count (fifteen) nor exposure surface (1,809); the only thing that changes is how useful the leaked value is after the run — that is what the 403 in the run above is. Issuing the token manually adds one step to each of the hundred twenty-four runs: the manual step count rises from ten to a hundred thirty-four, average feedback from 263.5 to 277.1 minutes. Automating issuance gives the same window with eleven manual steps and without changing feedback at all. The difference is a single setup step, and it removes 1,860 minutes of manual work.

The Return on the Decision

When a secret is seen to have leaked, the decision made depends on the window column in the table. If the window is unlimited, the decision is a revoke-and-reissue operation; every system using that value has to be found, and the work falls to the team holding the production side. If the window is 44 minutes, the decision is limited to opening a record, because the value is already invalid by the time the record opens. Same event, same leak point, two different decisions.

The second return is in the pipeline itself. An authorization rejection (403) and the defect itself (404) look like the same red to a stage. Unless they are separated, an expired token looks like a flaky test and gets quarantined; and a quarantined test, as the previous lesson measured, drops a defect class from the pipeline. A classification that tells an authorization error apart from a defect error is, for this reason, not a security measure but a gate decision.

Summary

• A secret cannot be masked: it either enters the test environment or it does not; every value that enters produces a leak surface, every value that does not produces an escaped defect class.
• A single failed call left three artifacts, and all three carried the token; one of the three was not in the team’s code, but in the other side’s diagnostic body.
• The leak point count does not depend on the token’s form; the persistent and time-limited tokens leaked the same number of points, and after the run one got 202, the other 403.
• Narrowing scope dropped exposure surface from 511,800 records to 1,809 (283 times over) and did not change escaped defects at all; its cost is the ten manual setup steps.
• Pulling back to read-only left four of the five secrets non-functional: escaped defects rose from five to thirty-two, and exposure surface did not shrink at all.
• Issuing time-limited authorization manually added one step to a hundred twenty-four runs (134 manual steps, 277.1-minute feedback); automating issuance gave the same window with 11 steps and 263.5 minutes.

Next Step

Three lessons counted three distributions: run minutes, environment-minutes, and authorization. The numbers all three produce sit in the same place — inside a run. No one reads the three artifacts above; no one watches the gate arrangement’s average feedback, the environment queue, and the exposure surface on a screen. For a measurement to change a decision, it first has to be readable, and readability has a budget too: how many lines, to whom, how often. The next lesson takes on this distribution — it counts which of the hundreds of lines a run leaves behind reach the team, and which decision the ones that do not reach anyone silently cancel.