Key and Secret Management

The previous lesson counted an open file’s contents as readable. The records were encrypted on disk, but that was no obstacle for a subject with read permission — decryption happened on the provider’s side, as part of the read request. The question that was not asked is this: does the radius change depending on who holds the decryption key.

This lesson measures only that. The Cloud Governance, Security and Cost course measured encryption at rest and in transit, envelope encryption, and the rotation window; none of that encryption mechanics is repeated here. The only thing measured is key ownership’s effect on the number of readable items. The second half takes up the secret store, and this lesson’s second finding comes out of it: in the walk, the secret store is not an item — it is an edge source.

• CM23. There are three ownership modes. provider: the key sits with the provider, and decryption happens as part of the read request. managed: the key belongs to the customer and sits in the key management service; decryption requires a separate key-use permission. held: the key is outside the leased infrastructure, and a leaked cloud identity cannot reach it.
• CM24. Key-use permission is a set separate from access permission. An identity can have access to an item while being unable to use its key.
• CM25. Each item records the strongest mode it supports. A managed service that decrypts server-side cannot support held mode; this is a modeling decision, and it follows from the service type.
• CM26. When the secret store can be read, each secret inside it becomes an identity, and the walk continues from there. The secrets’ contents are not modeled; only which identity they correspond to is counted.
• CM27. The walk runs to a fixed point: as the identity set grows, key permission and readable items grow; as readable items grow, the identity set grows.

Three Modes, Three Numbers

// cloud/key.mjs — ownership's effect on the radius is a MODEL. Encryption mechanics, envelope
// encryption, and the rotation window are not in this lesson; only readable items are counted.

const ACCESS = {
  'account-collector': 'measurement-store measurement-queue',
  'account-validator': 'measurement-store measurement-queue raw-archive',
  'account-portal': 'subscriber-database portal-asset-store work-order-queue',
  'account-report': 'report-store billing-output-store log-store',
  'role-billing': 'billing-database billing-output-store mail-dispatch subscriber-database',
  'role-archive': 'raw-archive backup-store',
  'role-backup': 'backup-store billing-database subscriber-database measurement-store',
  'role-distribution': 'distribution-configuration template-store instance-management',
  'role-log': 'log-store audit-log',
  'role-management': 'instance-management key-management secret-store',
};
const ASSUME = [['account-collector', 'role-archive'], ['account-validator', 'role-archive'],
  ['role-archive', 'role-backup'], ['role-backup', 'role-management'], ['account-portal', 'role-billing'],
  ['role-billing', 'role-log'], ['account-report', 'role-log'], ['role-distribution', 'role-management']];
// An identity's key-use permission: a set separate from its access permission.
export const KEY_PERMISSION = {
  'account-collector': 'k-measurement', 'account-validator': 'k-measurement', 'account-portal': 'k-subscriber',
  'account-report': 'k-billing k-log', 'role-billing': 'k-billing k-subscriber', 'role-archive': 'k-measurement',
  'role-backup': 'k-backup', 'role-distribution': 'k-secret', 'role-log': 'k-log',
  'role-management': 'k-secret',
};
// Item: whether it is encrypted, with which key, and the strongest ownership mode it supports.
// Modes: provider < managed < held. A managed key sits in the key management service; a held
// key is outside the leased infrastructure, and a leaked identity cannot reach it.
export const MODE = ['provider', 'managed', 'held'];
export const ITEM = `
  measurement-store           k-measurement   held
  raw-archive                 k-measurement   held
  backup-store                k-backup        held
  billing-output-store        k-billing       held
  report-store                k-billing       held
  distribution-configuration  k-secret        held
  subscriber-database         k-subscriber    managed
  billing-database            k-billing       managed
  log-store                   k-log           managed
  secret-store                k-secret        managed
  audit-log                   k-log           provider
  measurement-queue           -               -
  portal-asset-store          -               -
  work-order-queue            -               -
  mail-dispatch               -               -
  key-management              -               -
  instance-management         -               -
  template-store               -              -`
  .trim().split('\n').map((s) => {
    const [name, key, support] = s.trim().split(/\s+/);
    return { name, key: key === '-' ? null : key, support };
  });
export const find = (name) => ITEM.find((k) => k.name === name);
// If the secret store is read, the secrets inside it are each an identity; the walk continues from there.
export const SECRETS = ['account-portal', 'role-billing', 'account-report'];

// How many legitimate operations a day use which key: [identity, key, uses a day]
export const LEGITIMATE_KEY = [['account-collector', 'k-measurement', 1440],
  ['account-validator', 'k-measurement', 720], ['account-portal', 'k-subscriber', 3100],
  ['account-report', 'k-billing', 40], ['account-report', 'k-log', 12],
  ['role-billing', 'k-billing', 260], ['role-billing', 'k-subscriber', 260],
  ['role-archive', 'k-measurement', 96], ['role-backup', 'k-backup', 4],
  ['role-distribution', 'k-secret', 6], ['role-log', 'k-log', 88], ['role-management', 'k-secret', 1]];

// The items readable from a leaked identity: a fixed point. If the secret store is read, the identity set grows.
export function readable(target, excluded = new Set()) {
  const identity = new Set(['account-collector']);
  const close = () => {                            // assumption closure
    let n = -1;
    while (identity.size !== n) {
      n = identity.size;
      for (const k of [...identity]) for (const [a, b] of ASSUME) if (a === k) identity.add(b);
    }
  };
  let previous = -1, item = new Set(), permission = new Set();
  while (identity.size !== previous) {
    previous = identity.size;
    close();
    permission = new Set([...identity].flatMap((k) => KEY_PERMISSION[k].split(' '))
      .filter((a) => !excluded.has(a)));
    item = new Set([...identity].flatMap((k) => ACCESS[k].split(' ')).filter((r) => {
      const x = find(r);
      if (x.key === null) return true;                            // unencrypted item
      const mode = MODE.indexOf(target) < MODE.indexOf(x.support) ? target : x.support;
      return mode === 'provider' || (mode === 'managed' && permission.has(x.key));
    }));
    if (item.has('secret-store')) for (const s of SECRETS) identity.add(s);
  }
  return { identity, item, permission };
}

// cloud/run1.mjs — the items a leaked identity can read in three ownership modes
import { ITEM, MODE, find, readable } from './key.mjs';

const print = (g, ...s) => console.log(s.map((v, i) =>
  g[i] < 0 ? String(v).padEnd(-g[i]) : String(v).padStart(g[i])).join(''));
const A = [-23, -14, -11, 12, 12, 10];
print(A, 'item', 'key', 'support', ...MODE);
const R = Object.fromEntries(MODE.map((k) => [k, readable(k)]));
const seen = [...new Set(MODE.flatMap((k) => [...R[k].item]))].sort();
for (const r of seen.filter((x) => find(x).key))
  print(A, r, find(r).key, find(r).support, ...MODE.map((k) => R[k].item.has(r) ? 'readable' : '-'));
print(A, `+ ${seen.filter((x) => !find(x).key).length} unencrypted item`, '', '',
  ...MODE.map(() => 'readable'));
print(A, 'items read', '', '', ...MODE.map((k) => R[k].item.size));
console.log(`\nidentity entering the walk: ` + MODE.map((k) => `${k} ${R[k].identity.size}`).join(', '));
console.log("key permission from the leaked identity's chain: " +
  [...R.held.permission].sort().join(', '));
console.log(`unencrypted item ${ITEM.filter((k) => k.key === null).length}/${ITEM.length}`);

item                   key           support        provider     managed      held
audit-log              k-log         provider       readable    readable  readable
backup-store           k-backup      held           readable    readable         -
billing-database       k-billing     managed        readable    readable  readable
billing-output-store   k-billing     held           readable    readable         -
log-store              k-log         managed        readable    readable  readable
measurement-store      k-measurement held           readable    readable         -
raw-archive            k-measurement held           readable    readable         -
report-store           k-billing     held           readable    readable         -
secret-store           k-secret      managed        readable    readable  readable
subscriber-database    k-subscriber  managed        readable    readable  readable
+ 6 unencrypted item                                readable    readable  readable
items read                                                16          16        11

identity entering the walk: provider 8, managed 8, held 8
key permission from the leaked identity's chain: k-backup, k-billing, k-log, k-measurement, k-secret, k-subscriber
unencrypted item 7/18

The first two columns are the same: sixteen. The number of readable items does not change at all when the key moves to the customer. The reason is the row before last — the leaked identity’s chain carries all six key permissions. Ownership mode ties decryption to a permission; if that permission is on the same chain, tying it to something does not help.

The third column gives eleven. The five items that drop are the ones that support held mode: the measurement store, the raw archive, the backup store, the billing output store, and the report store. The remaining five cannot support it, because they are managed services whose decryption happens server-side.

The row above the second-to-last is worth noting too. The number of identities entering the walk is eight in all three modes — even though the access graph itself would end at four identities. The four identities in between come from the secret store, and the secret store is readable in all three modes.

Ownership Alone Is not Enough

Ownership mode and key-use permission are two separate levers. They are run both separately and together.

// cloud/run2.mjs — the combined effect of ownership mode and key permission; the cost of each
import { ITEM, MODE, LEGITIMATE_KEY, SECRETS, readable } from './key.mjs';

const print = (g, ...s) => console.log(s.map((v, i) =>
  g[i] < 0 ? String(v).padEnd(-g[i]) : String(v).padStart(g[i])).join(''));
const A = [-12, 16, 15, 13, 12];
print(A, 'ownership', 'k-secret in chain', 'k-secret excluded', 'walked', 'walked');
print(A, 'mode', 'items read', 'items read', 'identity', 'identity');
for (const k of MODE) {
  const a = readable(k), b = readable(k, new Set(['k-secret']));
  print(A, k, a.item.size, b.item.size, a.identity.size, b.identity.size);
}
const secretOp = LEGITIMATE_KEY.filter(([k, a]) => a === 'k-secret' && k === 'role-management')
  .reduce((x, [, , n]) => x + n, 0);
console.log(`\ncost of taking the k-secret permission from the role in the chain: ${secretOp} legitimate operation a day`);
console.log(`secrets in the secret store ${SECRETS.length}; additional identity entering the walk if read ` +
  `${readable('provider').identity.size - 4}`);
const encrypted = ITEM.filter((k) => k.key);
const unsupported = encrypted.filter((k) => k.support !== 'held');
console.log(`\n${unsupported.length} of ${encrypted.length} encrypted items do not support held mode:`);
console.log('  ' + unsupported.map((k) => k.name).join(', '));
const external = LEGITIMATE_KEY.filter(([, a]) =>
  encrypted.some((k) => k.key === a && k.support === 'held'));
console.log(`(workload, key) pairs needing external key delivery ${external.length}; ` +
  `${external.reduce((x, [, , n]) => x + n, 0)} operations a day go through that path`);

ownership   k-secret in chaink-secret excluded       walked      walked
mode              items read     items read     identity    identity
provider                  16             16            8           8
managed                   16              6            8           4
held                      11              3            8           4

cost of taking the k-secret permission from the role in the chain: 1 legitimate operation a day
secrets in the secret store 3; additional identity entering the walk if read 4

5 of 11 encrypted items do not support held mode:
  subscriber-database, billing-database, log-store, secret-store, audit-log
(workload, key) pairs needing external key delivery 8; 2567 operations a day go through that path

The table has six cells, and three of them carry the decision. In the provider row, two columns are the same: sixteen and sixteen. Changing the key permission changes nothing for the provider key, because decryption never asks for permission — read access means plaintext.

The managed row gives this lesson’s sharpest comparison. Moving the key to the customer on its own leaves the radius at sixteen. In the same mode, taking a single key permission — the secret store’s key — out of the leaked identity’s chain brings the radius down to six. Same ownership mode, same graph, more than a two-and-a-half-fold difference. Ownership does not lower the radius; who the owner grants permission to does.

The held row reads the same way: eleven down to three. But the gap between eleven and sixteen is five items, and the cost is heavy — five encrypted items do not support this mode at all, and eight (workload, key) pairs require external key delivery; two thousand five hundred sixty-seven operations a day go through that path. The cost of removing a single permission, by contrast, is one legitimate operation a day.

Six Keys, One by One

The secret store’s key was picked as the example. Running all six keys through the same measure shows why it was the one chosen.

// cloud/run3.mjs — key permissions one by one: how much taking each one out of the chain lowers the radius
import { KEY_PERMISSION, LEGITIMATE_KEY, readable } from './key.mjs';

const KEY = [...new Set(Object.values(KEY_PERMISSION).flatMap((s) => s.split(' ')))].sort();
const print = (g, ...s) => console.log(s.map((v, i) =>
  g[i] < 0 ? String(v).padEnd(-g[i]) : String(v).padStart(g[i])).join(''));
const A = [-14, 11, 11, 12, 13];
const base = readable('managed').item.size;
print(A, 'key', 'managed', 'held', 'radius', 'stopped op');
print(A, 'from chain', 'mode', 'mode', 'drop', 'a day');
for (const a of KEY) {
  const c = new Set([a]);
  print(A, a, readable('managed', c).item.size, readable('held', c).item.size,
    base - readable('managed', c).item.size,
    LEGITIMATE_KEY.filter(([, x]) => x === a).reduce((s, [, , n]) => s + n, 0));
}
const all = new Set(KEY);
console.log(`\nif all are excluded: managed ${readable('managed', all).item.size}, ` +
  `held ${readable('held', all).item.size}; ` +
  `${LEGITIMATE_KEY.reduce((s, [, , n]) => s + n, 0)} operations a day stop`);

key               managed       held      radius   stopped op
from chain           mode       mode        drop        a day
k-backup               15         11           1            4
k-billing              13         10           3          300
k-log                  15         10           1          100
k-measurement          14         11           2         2256
k-secret                6          3          10            7
k-subscriber           15         10           1         3360

if all are excluded: managed 3, held 3; 6027 operations a day stop

Five of the keys drop the radius by somewhere between one and three; the sixth drops it by ten. k-subscriber carries three thousand three hundred sixty operations a day and, if removed, drops the radius by a single item. k-secret carries seven operations a day and drops the radius by ten items. The cost-per-item ratio between them differs by more than three thousand times.

The seven here should not be confused with the number from the previous section: here the permission is removed from every identity, and the distribution role’s six operations a day stop as well. Taking it from just the one role that is on the leaked identity’s chain is enough, and that costs one operation a day.

The last row draws the boundary: if every key permission is removed, the radius drops to three and six thousand twenty-seven operations a day stop. So the way to bring the radius down from ten to three is not to remove every permission; removing one of the six gives most of the difference.

The Secret Store Is not an Item

The two right-hand columns show why the difference is this large. With k-secret in the chain, eight identities enter the walk; with it removed, four. When the secret store is read, the three secrets inside it correspond to three identities, and with the assumption closure, four new identities join the walk.

This changes the secret store’s place in the graph. The other items are the walk’s endpoint: they are read and that is where it ends. When the secret store is read, new edges of the walk are born — meaning the secret store is not an item, it is an edge source. Looking at it as an item in the radius measurement shows nine instead of sixteen.

For the same reason, the measurement is a fixed-point calculation. As the identity set grows, key permission grows; as key permission grows, readable items grow; if the secret store is among the readable items, the identity set grows again. The loop only stops once no new identity arrives. A radius computed in a single pass comes up short in this model.

Combined with the previous lesson’s finding, the pattern becomes clear: an identity reference sitting in a configuration file grew the radius by five items; the secret store being readable grows it by seven. Both are small items, and in both cases the damage is not in their own contents — it is in the edges they open.

Summary

• Key ownership alone does not lower the radius: 16 items are read under the provider key, and still 16 under the customer-managed key. The leaked identity’s chain carries all six key permissions.
• The customer-held key brings 16 down to 11; the five items that drop are the ones that support this mode. Of the 10 encrypted items the walk reaches, 5 decrypt server-side and so do not support this mode at all.
• Taking a single key permission out of the chain drops managed mode from 16 to 6 and held mode from 11 to 3. More than a two-and-a-half-fold difference within the same ownership mode; the cost is 1 legitimate operation a day.
• Held mode’s cost is heavy: 8 (workload, key) pairs require external key delivery, and 2,567 operations a day go through that path.
• The secret store is not an item in the walk, it is an edge source: when read, 3 secrets correspond to 3 identities, and with the closure, 4 identities enter the walk, taking the identity count from 4 to 8.
• The measurement is a fixed-point calculation; a radius computed in a single pass comes out as 9 in this model, versus 16 at the fixed point.

Next Step

Over five lessons, misconfigured items were found one at a time: an overly broad trust list, an unclosed public endpoint, a forgotten object-level permission, a key permission sitting on the chain. All of them were found in a running system, and each was already in place by the time it was found. The question that was not asked is this: how many of these could have been seen in the template before they were ever put into the system. The next lesson runs the template set through a rule set, and separates the item caught before deployment from the item that can only be seen while running.