Configuration and Secret Objects

The previous lesson brought the request to the pod’s door. The process running inside, though, has needs beyond the request arriving at the door: which record store to connect to, which threshold to validate with, which key to sign invoices with. If these values are written into the image, three environments mean three separate images — and then the image’s portability, the property measured in M22/K02, disappears. If they are kept outside, they mean a new object declared to the cluster.

There are two of these objects: the configuration object, which carries open values, and the secret object, which carries values that must stay hidden. Both do the same job; what separates them is an expectation about the visibility of the values they carry, and the second half of this lesson counts how well that expectation holds. M22/K01’s leak-path scan is not repeated here; the question there was where a value leaks to, the question here is how many subjects in the cluster are authorized to read it. What is measured is a model; the values are fictional.

CO25 — the verifier reads fourteen setting keys, four of which must be kept secret. CO26 — a setting passed as an environment variable is read once when the process starts; even if the object changes, the running process keeps seeing the old value. CO27 — a setting mounted as a file reaches the running pod three rounds later, and because the process re-reads the file, no restart is needed. CO28 — a restart takes three rounds, and the pod serves nothing during those three rounds. CO29 — a pod that binds an object, the node agent on that pod’s node, and two components of the control plane can read the object; the last two cannot be closed off by an authorization rule. CO30 — three changes occur over a twenty-four-round run: an open setting in round five, a secret setting in round twelve, an open setting again in round nineteen.

The Same Setting, Two Forms

The run below passes the same fourteen keys and the same three changes through four arrangements. The measured quantity is stale key-rounds: every key, for every round, where the value the running pod sees differs from the value in the object counts as one.

// config.mjs — the same setting set passed in two forms (MODEL). Values are fictional.
export const ROUNDS = 24, PROPAGATION = 3, RESTART = 2;   // CO27, CO28

export const KEYS = [
  ['collector-address', false], ['verification-threshold', false], ['log-level', false],
  ['retry-count', false], ['timeout-rounds', false], ['batch-size', false],
  ['region-code', false], ['measurement-unit', false], ['report-interval', false],
  ['cache-lifetime', false], ['record-store-password', true], ['billing-signing-key', true],
  ['work-order-token', true], ['fault-notification-key', true],
];                                                        // CO25: 14 keys, 4 secret

// Change schedule: which key's value changes in which round.
const CHANGE = { 5: 'verification-threshold', 12: 'billing-signing-key', 19: 'timeout-rounds' };

// CO26: an env variable is read once when the process starts; a file reflects PROPAGATION rounds later.
export function run(format, restartOnChange) {
  const object = {}, visible = {};
  for (const [a] of KEYS) { object[a] = 'mock-value-0'; visible[a] = 'mock-value-0'; }
  const pending = [];
  let stale = 0, restarts = 0, outage = 0, outageEnd = -1;
  for (let t = 0; t < ROUNDS; t++) {
    const key = CHANGE[t];
    if (key) {
      object[key] = `mock-value-${t}`;
      const f = format(key);
      if (f === 'file') pending.push({ key, at: t + PROPAGATION });
      else if (restartOnChange) {                         // env format: only reflected by a restart
        restarts++; outageEnd = t + RESTART;
        for (const [x] of KEYS) pending.push({ key: x, at: t + RESTART });
      }
    }
    for (let i = pending.length - 1; i >= 0; i--) {       // a reflected declaration leaves the queue
      if (pending[i].at <= t) { visible[pending[i].key] = object[pending[i].key]; pending.splice(i, 1); }
    }
    stale += KEYS.filter(([a]) => visible[a] !== object[a]).length;
    if (t <= outageEnd) outage++;
  }
  const remaining = KEYS.filter(([a]) => visible[a] !== object[a]).length;
  return { stale, restarts, outage, remaining };
}

if (import.meta.url === `file://${process.argv[1]}`) {
  const s = (x, n) => String(x).padStart(n);
  const secret = KEYS.filter(([, g]) => g).length;
  console.log(`${KEYS.length} keys (${secret} secret), ${ROUNDS} rounds, ${Object.keys(CHANGE).length}`
    + ` changes; propagation ${PROPAGATION} rounds, restart ${RESTART} rounds`);
  const FORMATS = [
    ['all env, no restart', () => 'env', false, `${KEYS.length}/0`],
    ['all env, with restart', () => 'env', true, `${KEYS.length}/0`],
    ['all file', () => 'file', false, `0/${KEYS.length}`],
    ['mixed', (a) => (KEYS.find(([x]) => x === a)[1] ? 'file' : 'env'), true,
      `${KEYS.length - secret}/${secret}`],
  ];
  console.log(`${'format'.padEnd(22)}${s('env/file', 13)}${s('stale key-rounds', 19)}`
    + `${s('restarts', 18)}${s('outage rounds', 15)}${s('stale at end', 14)}`);
  for (const [ad, f, yb, split] of FORMATS) {
    const r = run(f, yb);
    console.log(`${ad.padEnd(22)}${s(split, 13)}${s(r.stale, 19)}${s(r.restarts, 18)}`
      + `${s(r.outage, 15)}${s(r.remaining, 14)}`);
  }
}

14 keys (4 secret), 24 rounds, 3 changes; propagation 3 rounds, restart 2 rounds
format                     env/file   stale key-rounds          restarts  outage rounds  stale at end
all env, no restart            14/0                 36                 0              0             3
all env, with restart          14/0                  6                 3              9             0
all file                       0/14                  9                 0              0             0
mixed                          10/4                  7                 2              6             0

The first row is the environment-variable format’s silent result. Nothing fails, no outage occurs — and at the end of the run three keys are still at the wrong value. Thirty-six stale key-rounds are a drift nobody notices: the object changed, the declaration was updated, and the pod kept running with the old value. This is the most dangerous case in a declarative system, because the gap between desired state and actual state is not where the reconciler is looking; the replica count is correct, the pods are up, only the value inside them has gone stale.

The second row closes this gap and presents the bill. If the change triggers a restart, stale key-rounds drop from thirty-six to six, and stale-at-end is zero — but three restarts and nine rounds of outage appear. Nine of twenty-four rounds, nearly forty percent, are out of service because of a one-line setting change. The source of this cost is CO26: an environment variable is given to the process once, at the start, and there is no way to update it within the process’s lifetime.

The third row lowers both measures at once: nine stale key-rounds, zero restarts, zero outage. A setting mounted as a file reaches the running pod with a three-round propagation delay, and the process stays up because it re-reads the file. Three changes each produced three rounds of staleness; nine in total. That is three more than the six in the second row — but in exchange, the nine rounds of outage vanish.

The fourth row measures a common split: the four secret keys as a file, the ten open keys as environment variables. The result sits between the two — seven stale key-rounds, two restarts, six rounds of outage — and it is exactly worse than the third row. The mixed arrangement pays the cost of both formats: outage for the open settings, propagation delay for the secret ones. The reason for this split is not reflection speed, it is the visibility measured in the next section; but looking at this table alone, nobody can say the mixed arrangement is inherently better.

The lesson common to all four rows concerns where the drift is measured. The replica set controller compares the desired count against the actual count and closes the gap; setting values sit outside that comparison. The gap between the value in the object and the value in the running process is nowhere any controller is looking, which is why nobody notices anything across twenty-four rounds in the first row. In a declarative system, every field that is not reconciled is a field that can drift silently.

The Secret Object’s Visibility

The reason for keeping secret keys in a separate object type is visibility. This section turns that reason into a number: with the same authorization schema, in the same cluster, how many subjects can read a secret object.

// visibility.mjs — how many subjects in the cluster can read a secret object (MODEL).
import { KEYS } from './config.mjs';

// Our own small authorization schema: a rule grants a subject a verb over an object type and name pattern.
export const AUTH_SCHEMA = { subject: 'string', type: 'string|*', name: 'string|*', verb: 'list' };

const PODS = {                                            // which pod runs on which node
  'verifier-pod': 'node-1', 'billing-pod': 'node-2',
  'collector-pod': 'node-3',
};
const BOUND_BY = {                                        // which pods bind which object
  'billing-signing-key': ['verifier-pod', 'billing-pod'],
  'verification-threshold': ['verifier-pod', 'billing-pod', 'collector-pod'],
};
const CONTROL_PLANE = ['state-store', 'api-server'];       // both carry both objects

const ROLES = {
  broad: [
    { subject: 'operate-role', type: '*', name: '*', verb: ['read', 'write'] },
    { subject: 'observe-role', type: '*', name: '*', verb: ['read'] },
    { subject: 'release-subject', type: '*', name: '*', verb: ['read', 'write'] },
    { subject: 'backup-subject', type: '*', name: '*', verb: ['read'] },
    { subject: 'audit-subject', type: '*', name: '*', verb: ['read'] },
  ],
  least: [
    { subject: 'operate-role', type: 'config', name: '*', verb: ['read', 'write'] },
    { subject: 'observe-role', type: 'config', name: '*', verb: ['read'] },
    { subject: 'release-subject', type: 'secret', name: 'billing-signing-key', verb: ['write'] },
    { subject: 'backup-subject', type: 'config', name: '*', verb: ['read'] },
    { subject: 'audit-subject', type: 'config', name: '*', verb: ['read'] },
  ],
};

export function canRead(object, type, policy) {
  const pods = BOUND_BY[object] ?? [];
  const agents = [...new Set(pods.map((k) => `agent-${PODS[k]}`))];
  const roles = ROLES[policy]
    .filter((r) => (r.type === '*' || r.type === type) && (r.name === '*' || r.name === object)
      && r.verb.includes('read'))
    .map((r) => r.subject);
  return { pods, roles, agents, plane: CONTROL_PLANE };
}

if (import.meta.url === `file://${process.argv[1]}`) {
  const s = (x, n) => String(x).padStart(n);
  const secret = KEYS.filter(([, g]) => g).length;
  console.log(`${secret} of ${KEYS.length} keys are in the secret object, ${KEYS.length - secret}`
    + ` are in the config object; authorization schema: ${Object.keys(AUTH_SCHEMA).join(', ')}`);
  console.log(`${'object'.padEnd(24)}${'type'.padEnd(14)}${'policy'.padEnd(9)}${s('pods', 8)}`
    + `${s('roles', 7)}${s('node agents', 14)}${s('control plane', 16)}${s('total', 8)}`);
  for (const [object, type] of [['billing-signing-key', 'secret'], ['verification-threshold', 'config']]) {
    for (const p of ['broad', 'least']) {
      const r = canRead(object, type, p);
      const total = r.pods.length + r.roles.length + r.agents.length + r.plane.length;
      console.log(`${object.padEnd(24)}${type.padEnd(14)}${p.padEnd(9)}${s(r.pods.length, 8)}`
        + `${s(r.roles.length, 7)}${s(r.agents.length, 14)}${s(r.plane.length, 16)}${s(total, 8)}`);
    }
  }
  const least = canRead('billing-signing-key', 'secret', 'least');
  console.log(`\nunder least privilege, those who can read the secret object: ${[...least.pods, ...least.roles,
    ...least.agents, ...least.plane].join(', ')}`);
  console.log(`${least.agents.length + least.plane.length} of them cannot be removed by an authorization rule`);
}

4 of 14 keys are in the secret object, 10 are in the config object; authorization schema: subject, type, name, verb
object                  type          policy       pods  roles   node agents   control plane   total
billing-signing-key     secret        broad           2      5             2               2      11
billing-signing-key     secret        least           2      0             2               2       6
verification-threshold  config        broad           3      5             3               2      13
verification-threshold  config        least           3      4             3               2      12

under least privilege, those who can read the secret object: verifier-pod, billing-pod, agent-node-1, agent-node-2, state-store, api-server
4 of them cannot be removed by an authorization rule

The first and third rows should be read together. Under the broad authorization policy, eleven subjects can read the secret object and thirteen can read the open configuration object. The gap of two comes from the secret being bound by one fewer pod — the role count is five in both. No role narrowed because the object’s type is secret. This is this section’s first result: putting a value into a secret object does not, by itself, change who can read it; what changes it is the authorization rule, and that rule has to be written separately.

The second row shows what happens once the rule is written. Under the least-privilege policy, the role count that can read the secret object drops from five to zero; the total falls from eleven to six. Under the same policy the configuration object stays at twelve, because reading the open settings is the job of four roles. This is where the real difference between the two object types is born — not in the type, but in the rule written for that type.

The last two rows say where the narrowing stops. Even under least privilege there are six readers, and four of them cannot be removed by an authorization rule: the node agents on the two nodes where the pods binding the secret run, and two components of the control plane. The node agent has to see the value because it sets up the pod’s filesystem; the state store and the API server see it because they store and distribute the object. The moment a secret object is written to the cluster, these four readers are born, and no role definition reduces them. The count does not drop to zero, it drops to four.

The Devolved Decision and What It Does Not Know

The devolved decision here is how the setting reaches the pod. This decision used to belong to whoever produced the image: the value was written into the image, and a separate image was built for each environment. Once the decision moves to the cluster, three information items the system needs to know are born: which key travels in which form, how quickly a change needs to be reflected, and which key counts as secret. All three are declarations and need no measurement; their cost is not in writing them, it is that nothing warns you when they are written wrong.

What the system does not know is when the process reads the setting. The model turned this into an assumption with CO26 and CO27; in reality the only place that knows this is the process’s own code, and the cluster cannot look inside it. The cost of this ignorance sits in the first table’s first row: thirty-six stale key-rounds and three wrong values at the end, without producing a single error record. The restart decision is born from the same ignorance too — the nine rounds of outage that would not be needed if the process re-read the file are paid because whether it does is unknown.

The second table shows a case where devolving does not pay off. Moving the secret value into a separate object type brought visibility down from thirteen to eleven; the real reduction was done by the authorization rule, and that rule was written by a human. The system does not know which role genuinely needs which secret; knowing this would require observing every subject’s access to every object and dropping what goes unused. Because it is not known, the default stays broad, and five roles carry read authorization they do not need.

Summary

• The same fourteen keys were passed in four arrangements. In the environment-variable format, if no restart happens, thirty-six stale key-rounds appear and three keys are still at the wrong value at the end of the run; no error record is produced.
• Restarting on change brings stale key-rounds down to six but produces three restarts and outage in nine of twenty-four rounds.
• Mounting as a file gives nine stale key-rounds, zero restarts, and zero outage; the propagation delay is three rounds per change.
• The mixed arrangement, secret as file and open as environment variable, produces seven stale key-rounds and six rounds of outage — it pays the cost of both formats and is worse than the file format alone.
• Putting a value into a secret object does not narrow visibility by itself: under the broad policy five roles can read both object types; what narrows it is the authorization rule written separately.
• Under the least-privilege policy, the secret object’s readers drop from eleven to six, but not to zero: the node agents on two nodes and two components of the control plane cannot be removed by an authorization rule.

Next Step

Configuration and secret objects reached the pod through a filesystem area, and that area was born with the pod and died with the pod. This is the correct behavior for settings: the value stays in the object, and nothing is lost when the pod goes away. For data the application produces, though, the same behavior means data loss. When the record store’s pod restarts, the meter readings must not go with it; yet everything measured in this topic so far — the pods the replica set deletes, the new version installed in a rollout, the endpoints that come and go with churn — assumed that a pod’s lifetime is short. The next lesson takes up the place where that assumption breaks: how an area of data that outlives the pod is declared to the cluster, and the new information items that declaration carries.