---
title: 'Secrets and Data Management'
source: 'https://academia.sh/en/courses/testing-process/secrets-and-data-management'
course: 'The Testing Process and Automation Infrastructure'
language: en
updated: '2026-08-23T14:25:19+00:00'
license: 'CC BY-SA 4.0'
---

# Secrets and Data Management

The distribution of secrets entering the test environment: leak points measured in run artifacts, least privilege's effect on exposure surface, the manual step count a time-limited privilege leaves behind, and the escaped defects seven authorization arrangements produce on the same defect set.

The previous lesson tied the production-like environment to a cost but did not ask what goes
into that environment. The data side was measured in the Test Data Management lesson: masking
looks at the content of personal fields, and what property it preserves was counted there.
Access credentials, identity, signing keys, and external-service tokens were left outside that
measurement. These are **secrets**, and they cannot be masked — a masked version of a secret
does not work, so it either goes in or it does not.

The resource this lesson distributes is **authorization**: how many secrets, with how much
scope, for how long, enter the test environment. The distribution has two returns — every
secret that does not enter misses a defect class, every secret that does enter opens a **leak
surface**.

## Leak Surface

Leak surface is not an estimate, it is a countable quantity: how many of the artifacts a run
leaves behind carry the secret itself. The run below actually calls the notification boundary
and triggers the defect measured in the previous course — the unknown recipient. Two token
forms are tried: an indefinitely valid persistent token and a token issued only for that run.

```js
// leak.mjs -- how many places a token appears in run artifacts, and its value after the run
import { createServer } from 'node:http';

const valid = new Set(['B-persistent-9f21', 'B-run-4a7c']);
const members = new Set(['U-17']);
const server = createServer((req, res) => {
  const token = req.headers['x-token'] ?? '';
  const recipient = new URL(req.url, 'http://y').searchParams.get('recipient');
  const [status, body] = !valid.has(token)
    ? [403, `{"error":"invalid token"}`]
    : members.has(recipient) ? [202, '{"status":"queued"}']
      // the diagnostic body echoes the request's context back: this is the third leak point
      : [404, `{"error":"unknown recipient: ${recipient}","token":"${token}"}`];
  res.writeHead(status, { 'content-type': 'application/json' });
  res.end(body);
});
server.listen(8941, '127.0.0.1');

const BASE = 'http://127.0.0.1:8941/notify';
const call = (token, recipient) => fetch(`${BASE}?recipient=${recipient}`, { headers: { 'x-token': token } });

// The run triggers the defect measured in the Test Environment Management lesson: unknown recipient.
async function run(token) {
  const artifact = [`STEP loan-flow x-token=${token}`];        // 1: run record
  const res = await call(token, 'U-99');
  artifact.push(`ERROR ${await res.text()}`);                  // 2: the other side's diagnostic body
  artifact.push(`RETRY 1/1 x-token=${token} outcome=${res.status}`);   // 3: retry summary
  return { artifact, leaked: artifact.filter((s) => s.includes(token)).length };
}

const persistent = await run('B-persistent-9f21');
const runScoped = await run('B-run-4a7c');
valid.delete('B-run-4a7c');                                    // validity window closed
const after = async (token) => (await call(token, 'U-17')).status;
const persistentAfter = await after('B-persistent-9f21');
const runAfter = await after('B-run-4a7c');
server.close();

console.log(`persistent token: ${persistent.artifact.length} artifacts, value present in ${persistent.leaked}`);
console.log(`run-scoped token: ${runScoped.artifact.length} artifacts, value present in ${runScoped.leaked}`);
console.log(`leak point count is the same in both forms: ${persistent.leaked === runScoped.leaked}`);
console.log(`post-run request, persistent token: ${persistentAfter}`);
console.log(`post-run request, run-scoped token: ${runAfter}`);
console.log(`artifact not in the team's code: ${runScoped.artifact[1]}`);
```

```
persistent token: 3 artifacts, value present in 3
run-scoped token: 3 artifacts, value present in 3
leak point count is the same in both forms: true
post-run request, persistent token: 202
post-run request, run-scoped token: 403
artifact not in the team's code: ERROR {"error":"unknown recipient: U-99","token":"B-run-4a7c"}
```

Three results are worth reading. First, a single failed call leaves three artifacts, and all
three carry the token itself. Second, only two of these three points are in the team's own
code; the third is the other side's diagnostic body, and no scrubbing rule the team writes can
anticipate that body. Third, the two token forms leak **exactly** the same number of points —
the leak point count depends on the shape of the error path, not the nature of the token.

The only thing that changes is the value after the run: the persistent token still gets 202,
the run-scoped token 403. The lever is not concealment, it is **scope** and **duration**.

## Secret Catalog

The test environment wants five secrets, and each opens specific defect classes. The numbers
come from the previous lesson's defect set: all thirty-two defects caught at the mid stage are
tied to a secret, four at the nightly stage, one at the release stage.

**TP14 — the access counts under production scope are an assumption; the ones under narrow
scope are a measurement.** Narrow scope is the subset the secret actually touches, and it was
read from the test set the Test Data Management lesson measured: 772 rows, 200 members. The
numbers on the production side are assumptions about the same system's production scale and
carry an order of magnitude.

**TP15 — in the time-limited arrangement, the validity window is 44 minutes, and the setup's
manual steps are counted per secret.** The window is the sum of the mid stage's 41-minute run
and the ephemeral environment's 3-minute setup; it cannot be shorter than the run, because
authorization has to end when the run ends. The manual steps are three items: opening one
account per secret, writing one scope list under narrow scope, setting up one mechanism for
automated issuance. These are paid once per period, while a per-run manual step is paid again
on every run.

```js
// secrets.mjs -- the secret catalog and authorization arrangements
// Per-stage feedback time and the caught defect count not tied to a secret; both read from
// the previous lesson's "production-like heavy" row.
export const stages = {
  fast: { feedback: 8, free: 28 },
  mid: { feedback: 49, free: 0 },
  nightly: { feedback: 425, free: 5 },
  release: { feedback: 2357, free: 5 },
};
export const MISSED_FLOOR = 5;     // two classes no test sees
export const MANUAL = 15;          // TP6: one manual step is 15 minutes
export const RUNS = { mid: 100, nightly: 20, release: 4 };

// [name, stage, defect count opened, is write required, prod access, narrow-scope access]
export const secrets = [
  ['catalog ID', 'mid', 13, true, 240000, 772],
  ['notification token', 'mid', 10, false, 18000, 200],
  ['payment key', 'mid', 9, true, 9000, 41],
  ['performance environment ID', 'nightly', 4, true, 240000, 772],
  ['metrics store ID', 'release', 1, true, 4800, 24],
].map(([name, stage, count, write, prod, narrow]) => ({ name, stage, count, write, prod, narrow }));

// An arrangement gives, for each secret: does it enter the environment, how many records can it
// access, can it write; validity window (minutes), setup manual steps, per-run manual steps.
export const arrangements = [
  { name: 'production identity', enters: 1, access: (g) => g.prod, writes: 1, window: Infinity, setup: 0, perRunManual: 0 },
  { name: 'separate account', enters: 1, access: () => 772, writes: 1, window: Infinity, setup: 5, perRunManual: 0 },
  { name: 'read only', enters: 1, access: () => 772, writes: 0, window: Infinity, setup: 5, perRunManual: 0 },
  { name: 'narrow scope', enters: 1, access: (g) => g.narrow, writes: 1, window: Infinity, setup: 10, perRunManual: 0 },
  { name: 'narrow scope, time-limited', enters: 1, access: (g) => g.narrow, writes: 1, window: 44, setup: 10, perRunManual: 1 },
  { name: 'time-limited, automated issuance', enters: 1, access: (g) => g.narrow, writes: 1, window: 44, setup: 11, perRunManual: 0 },
  { name: 'mock', enters: 0, access: () => 0, writes: 0, window: 0, setup: 0, perRunManual: 0 },
];
```

## Seven Authorization Arrangements

The **least privilege** principle is tied to a number here: **exposure surface** is the total
number of records the leaked secrets can reach. If a secret does not get the write permission
it requires, the defect classes it opens escape; the same happens if it does not enter the
environment at all.

```js
// scope.mjs -- seven authorization arrangements: leak surface, exposure, manual steps, escaped defects
import { stages, secrets, arrangements, MISSED_FLOOR, MANUAL, RUNS } from './secrets.mjs';

const LEAK_POINTS = 3;           // measured value: the error path produces three artifacts

function measure(d) {
  const entering = d.enters === 1 ? secrets : [];
  const exposed = entering.reduce((a, g) => a + d.access(g), 0);
  const manualRuns = d.perRunManual * Object.values(RUNS).reduce((a, x) => a + x, 0);

  let extra = 0;
  const feedback = {};
  for (const [name, s] of Object.entries(stages)) {
    if (name !== 'fast') extra += d.perRunManual * MANUAL;
    feedback[name] = s.feedback + extra;
  }

  let missed = MISSED_FLOOR, weight = 0, total = 0;
  for (const [name, s] of Object.entries(stages)) {
    weight += s.free;
    total += feedback[name] * s.free;
  }
  for (const g of secrets) {
    if (d.enters === 0 || (g.write && d.writes === 0)) { missed += g.count; continue; }
    weight += g.count;
    total += feedback[g.stage] * g.count;
  }
  return {
    entering: entering.length, leak: entering.length * LEAK_POINTS, exposed,
    window: d.window, manual: d.setup + manualRuns, feedback: total / weight, missed,
  };
}

const s = (x, n) => String(x).padStart(n);
console.log(`${'arrangement'.padEnd(35)}${s('enters', 8)}${s('leak', 7)}${s('exposure', 12)}`
  + `${s('window', 11)}${s('manual steps', 14)}${s('avg feedback', 14)}${s('missed/100', 12)}`);
for (const d of arrangements) {
  const r = measure(d);
  console.log(`${d.name.padEnd(35)}${s(r.entering, 8)}${s(r.leak, 7)}${s(r.exposed, 12)}`
    + `${s(r.window === Infinity ? 'unlimited' : r.window, 11)}${s(r.manual, 14)}`
    + `${s(r.feedback.toFixed(1), 14)}${s(r.missed, 12)}`);
}

const prod = secrets.reduce((a, g) => a + g.prod, 0);
const narrow = secrets.reduce((a, g) => a + g.narrow, 0);
const writers = secrets.filter((g) => g.write);
console.log(`\nsecrets ${secrets.length}, requiring write ${writers.length}, `
  + `defects they open ${writers.reduce((a, g) => a + g.count, 0)}`);
console.log(`prod scope ${prod} records, narrow scope ${narrow} records, `
  + `narrowing ${(prod / narrow).toFixed(0)}x`);
console.log(`run count ${Object.values(RUNS).reduce((a, x) => a + x, 0)}; `
  + `one manual step per run costs ${Object.values(RUNS).reduce((a, x) => a + x, 0) * MANUAL} minutes`);
```

```
arrangement                          enters   leak    exposure     window  manual steps  avg feedback  missed/100
production identity                       5     15      511800  unlimited             0         263.5           5
separate account                          5     15        3860  unlimited             5         263.5           5
read only                                 5     15        3860  unlimited             5         304.7          32
narrow scope                              5     15        1809  unlimited            10         263.5           5
narrow scope, time-limited                5     15        1809         44           134         277.1           5
time-limited, automated issuance          5     15        1809         44            11         263.5           5
mock                                      0      0           0          0             0         371.9          42

secrets 5, requiring write 4, defects they open 27
prod scope 511800 records, narrow scope 1809 records, narrowing 283x
run count 124; one manual step per run costs 1860 minutes
```

## Reading the Table

The first and fourth rows are this lesson's central measurement. Between production identity
and narrow scope, escaped defects (five) and average feedback (263.5 minutes) are identical to
the digit; the only place they diverge is exposure surface — 511,800 records against 1,809, two
hundred eighty-three times over. **Least privilege's cost in escaped defects is zero.** What is
paid is the ten manual setup steps, and these are paid once per period. What makes narrowing
scope hard is not a measurable trade-off but the deferral of a one-time task.

This number has a condition, and it ties to the second half of this lesson's title. The narrow
scope's 1,809 records assumes the data in the environment is the **masked** test set. If a copy
of production was loaded into the environment, then no matter how narrow the secret's scope is,
exposure surface equals the environment's own size: 240,000 records sit there, and every token
that can reach that environment reaches them. The two decisions cannot be made separately for
this reason — unmasked data defeats least privilege, and narrow authorization does not stand in
for masking either.

The third row is narrowing done on the wrong axis. Read-only authority sounds safe, but four of
the five secrets require write: the catalog ID changes schema in the migration test, the
payment key updates card status, the metrics IDs write results. The twenty-seven defects these
four open escape, and escaped defects rise from five to thirty-two. Exposure surface does not
shrink at all — it stays at 3,860, because the value still enters the environment and still
leaks from three points. This row loses on both axes at once.

Here the average feedback column shows the **opposite** of the trap in the previous two
lessons. Under read-only, the average rises from 263.5 to 304.7, and to 371.9 under mock. The
reason is that the missed defects are the cheap ones caught: once the mid stage's forty-nine-
minute defects drop out of the denominator, only the expensive ones caught remain, and the
average rises. The same column can lie by improving in one lesson and worsening in another; in
both, what exposes the lie is the escaped defects column.

The last row is the only arrangement that truly zeroes the leak surface. No secret enters, the
leak point count is zero, exposure surface is zero — and escaped defects are forty-two. Mocking
the external service was built as a test form in the Mocking External Services lesson; here it
shows up as an **authorization decision**, and its cost is thirty-seven defects. An arrangement
that wants to zero out the leak surface ends up closing off the test itself.

The fifth and sixth rows isolate time-limited authorization's cost. The window dropping from
unlimited to 44 minutes changes neither the leak point count (fifteen) nor exposure surface
(1,809); the only thing that changes is how useful the leaked value is after the run — that is
what the 403 in the run above is. Issuing the token manually adds one step to each of the
hundred twenty-four runs: the manual step count rises from ten to a hundred thirty-four, average
feedback from 263.5 to 277.1 minutes. Automating issuance gives the same window with eleven
manual steps and without changing feedback at all. The difference is a single setup step, and
it removes 1,860 minutes of manual work.

## The Return on the Decision

When a secret is seen to have leaked, the decision made depends on the window column in the
table. If the window is unlimited, the decision is a revoke-and-reissue operation; every system
using that value has to be found, and the work falls to the team holding the production side.
If the window is 44 minutes, the decision is limited to opening a record, because the value is
already invalid by the time the record opens. Same event, same leak point, two different
decisions.

The second return is in the pipeline itself. An authorization rejection (403) and the defect
itself (404) look like the same red to a stage. Unless they are separated, an expired token
looks like a **flaky test** and gets quarantined; and a quarantined test, as the previous lesson
measured, drops a defect class from the pipeline. A classification that tells an authorization
error apart from a defect error is, for this reason, not a security measure but a gate decision.

## Summary

- A secret cannot be masked: it either enters the test environment or it does not; every value
  that enters produces a leak surface, every value that does not produces an escaped defect
  class.
- A single failed call left three artifacts, and all three carried the token; one of the three
  was not in the team's code, but in the other side's diagnostic body.
- The leak point count does not depend on the token's form; the persistent and time-limited
  tokens leaked the same number of points, and after the run one got 202, the other 403.
- Narrowing scope dropped exposure surface from 511,800 records to 1,809 (283 times over) and
  did not change escaped defects at all; its cost is the ten manual setup steps.
- Pulling back to read-only left four of the five secrets non-functional: escaped defects rose
  from five to thirty-two, and exposure surface did not shrink at all.
- Issuing time-limited authorization manually added one step to a hundred twenty-four runs (134
  manual steps, 277.1-minute feedback); automating issuance gave the same window with 11 steps
  and 263.5 minutes.

## Next Step

Three lessons counted three distributions: run minutes, environment-minutes, and authorization.
The numbers all three produce sit in the same place — inside a run. No one reads the three
artifacts above; no one watches the gate arrangement's average feedback, the environment queue,
and the exposure surface on a screen. For a measurement to change a decision, it first has to be
readable, and readability has a budget too: how many lines, to whom, how often. The next lesson
takes on this distribution — it counts which of the hundreds of lines a run leaves behind reach
the team, and which decision the ones that do not reach anyone silently cancel.