Security Debt

The previous lesson computed two metrics and showed that both could be improved without touching the thing being measured. The cheapest improvement was converting the old items to accepted risk: the open finding age fell from 59.8 days to 23.4 days, and the number of items actually fixed stayed at zero. Those seven items went somewhere, and where they went is the subject of this lesson.

Accepted risk and open debt together make up security debt. The Software Architecture curriculum measured technical debt; that measurement is not repeated here, and there is a single field that separates security debt from it: an acceptance has a validity. Technical debt does not pay itself off, but it does not expire either; an accepted risk whose validity was never written stops being debt and becomes a record.

The Debt Ledger

AC21. The debt ledger has 34 items: the 21 open items that remained unclosed in the previous lesson and 13 items that were converted to acceptance earlier. The set is a model of the subscriber portal team of the fictional regional measurement network; the generator is written in this block and the seed is visible. AC22. Each item has an effort (person-days) and a risk score; both are fictional data. AC23. An accepted item can carry three fields: a named owner, a validity period, and a reassessment point. AC24. The distribution of the fields is given as data: the first five acceptances have a validity written, the first two a reassessment point, and the first six an owner.

// debt/ledger.mjs — the security debt ledger is a MODEL: fictional regional measurement network, planning time.
// The generator is written here and the seed is visible; the ledger comes out the same on every run.
export const gen = (B) => (...h) => h.map((v, j) =>
  (j ? String(v).padStart(B[j]) : String(v).padEnd(B[0]))).join("");

export const SEED = 41100, N_OPEN = 21, N_ACCEPTED = 13;
let s = SEED;
const rnd = () => (s = (Math.imul(s, 1664525) + 1013904223) >>> 0) / 4294967296;

// Of the accepted items, the first five have a validity period, the first two a reassessment
// point, and the first six a named owner; the rest have the field empty. This distribution is data.
const VALIDITY = [90, 120, 180, 180, 365];

export const LEDGER = [];
for (let i = 0; i < N_OPEN + N_ACCEPTED; i++) {
  const isAccepted = i >= N_OPEN, k = i - N_OPEN;
  LEDGER.push({
    code: `D${String(i + 1).padStart(2, "0")}`,
    kind: isAccepted ? "accepted" : "open",
    source: rnd() < (isAccepted ? .75 : .70) ? "non-pipeline" : "pipeline",
    risk: 4 + Math.floor(rnd() * 21),                  // fictional risk score between 4 and 24
    effort: 1 + Math.floor(rnd() * 13),                // person-days
    age: isAccepted ? 40 + Math.floor(rnd() * 380) : 20 + Math.floor(rnd() * 160),
    validity: isAccepted && k < VALIDITY.length ? VALIDITY[k] : null,
    reassess: isAccepted && k < 2,
    owner: isAccepted ? k < 6 : true,
  });
}
export const openItems = LEDGER.filter((d) => d.kind === "open");
export const accepted = LEDGER.filter((d) => d.kind === "accepted");
export const expired = (d) => d.validity !== null && d.age > d.validity;

if (import.meta.url === `file://${process.argv[1]}`) {
  console.log(`model: ${LEDGER.length} debt items, ${openItems.length} open + ${accepted.length} accepted risk`);
  const y = gen([16, 8, 12, 12, 14, 14]);
  console.log("\n" + y("class", "items", "person-days", "risk total", "has validity", "expired"));
  for (const [name, k] of [["open debt", openItems], ["accepted risk", accepted]])
    console.log(y(name, k.length, k.reduce((a, d) => a + d.effort, 0), k.reduce((a, d) => a + d.risk, 0),
      k.filter((d) => d.validity !== null).length, k.filter(expired).length));

  // The questions asked of a debt item and the field each one needs.
  const QUESTIONS = [["is the item open or accepted", "class", LEDGER.length, LEDGER.length],
    ["who gave the acceptance", "owner", N_ACCEPTED, accepted.filter((d) => d.owner).length],
    ["how long is the acceptance valid", "validity", N_ACCEPTED, accepted.filter((d) => d.validity !== null).length],
    ["has the acceptance expired", "validity+age", N_ACCEPTED, accepted.filter((d) => d.validity !== null).length],
    ["when is the reassessment", "reassessment point", N_ACCEPTED, accepted.filter((d) => d.reassess).length],
    ["which period will the item close", "queue order", LEDGER.length, 0]];
  const answer = (c, b) => c === 0 ? "none" : c === b ? "full" : "partial";
  const z = gen([34, 20, 8, 10, 8]);
  console.log("\n" + z("question", "field needed", "base", "carrying", "answer"));
  for (const [q, f, b, c] of QUESTIONS) console.log(z(q, f, b, c, answer(c, b)));
  const n = (a) => QUESTIONS.filter(([, , b, c]) => answer(c, b) === a).length;
  const missing = (f) => N_ACCEPTED - accepted.filter(f).length;
  const toWrite = missing((d) => d.owner) + missing((d) => d.validity !== null) + missing((d) => d.reassess);
  console.log(`\nof ${QUESTIONS.length} questions, ${n("full")} fully answered, ${n("partial")} partially answered, ` +
    `${n("none")} unanswered`);
  console.log(`to make the partial answers full, ${toWrite} fields are written; of the ` +
    `${accepted.filter(expired).length} items with expired validity, ` +
    `${accepted.filter((d) => expired(d) && d.reassess).length} carries a reassessment point`);
}

model: 34 debt items, 21 open + 13 accepted risk

class              items person-days  risk total  has validity       expired
open debt             21         141         274             0             0
accepted risk         13          68         185             5             3

question                                  field needed    base  carrying  answer
is the item open or accepted                     class      34        34    full
who gave the acceptance                          owner      13         6 partial
how long is the acceptance valid              validity      13         5 partial
has the acceptance expired                validity+age      13         5 partial
when is the reassessment            reassessment point      13         2 partial
which period will the item close           queue order      34         0    none

of 6 questions, 1 fully answered, 4 partially answered, 1 unanswered
to make the partial answers full, 26 fields are written; of the 3 items with expired validity, 1 carries a reassessment point

The first table puts the two classes of the ledger side by side, and the real information is in the last two columns. Of the 13 accepted items, only five have a validity period written; for the other eight, the acceptance is a decision with no end. Of the five with a validity written, three have already expired at planning time, and because nobody asked, all three still sit in the ledger as accepted. The risk total column gives the scale: the accepted items carry 185 of the ledger’s risk total, and the open debt carries 274. Acceptance does not remove the risk; it only moves it out of the column the metrics look at.

In the question table, one of the six questions is answered fully. Of the remaining five, four concern only the accepted items and one concerns the whole ledger; four are partial and one is not answered at all. Making the partial ones full requires writing 26 fields: seven owners, eight validity periods, and eleven reassessment points. This is the cheapest cost in the course, and it needs no pipeline setup or extra volume; what has to be written is the decision itself. The last line gives its counterpart: of the three expired acceptances, only one carries a point that will bring it up again.

Running the Queue

AC25. The capacity is 12 person-days per period, a period is 90 days, and the planning horizon is four periods; all three are fictional data. The ordering rule is risk per unit of effort, and an item that does not fit the capacity is left for the next period. AC26. Two runs are compared: a ledger where an accepted item never comes back, and a ledger where an expired acceptance, if it has a reassessment point, re-enters the queue.

// debt/queue.mjs — the debt is run for four periods as a fixed-capacity queue.
import { LEDGER, openItems, accepted, expired, gen } from "./ledger.mjs";

const CAPACITY = 12, PERIODS = 4, PERIOD_DAYS = 90;   // person-days, number of periods, period length

// The ordering is risk per unit of effort; an item that does not fit the capacity is left for the next period.
// With comeBack on: an expired acceptance that has a reassessment point enters the queue.
const run = (comeBack) => {
  const queue = openItems.map((d) => ({ ...d })), pending = accepted.map((d) => ({ ...d }));
  const rows = [];
  for (let t = 1; t <= PERIODS; t++) {
    let back = 0;
    if (comeBack) for (const d of pending)
      if (!d.requeued && d.reassess && expired(d)) { d.requeued = true; queue.push(d); back++; }
    queue.sort((a, b) => b.risk / b.effort - a.risk / a.effort);
    let left = CAPACITY;
    for (const d of queue) if (!d.closed && d.effort <= left) { d.closed = true; left -= d.effort; }
    const stillOpen = queue.filter((d) => !d.closed);
    rows.push([t, back, queue.filter((d) => d.closed).length, CAPACITY - left, stillOpen.length,
      stillOpen.reduce((a, d) => a + d.effort, 0),
      Math.round(stillOpen.reduce((a, d) => a + d.age, 0) / stillOpen.length)]);
    for (const d of [...stillOpen, ...pending.filter((x) => !x.requeued)]) d.age += PERIOD_DAYS;
  }
  return { rows, stillOpen: queue.filter((d) => !d.closed), pending,
    back: rows.reduce((a, s) => a + s[1], 0) };
};

const y = gen([8, 10, 10, 12, 10, 14, 10]);
const RUNS = [["A: acceptance never returns", false], ["B: expired acceptance returns", true]];
RUNS.forEach(([name, comeBack], i) => {
  const r = run(comeBack);
  console.log((i ? "\n" : "") + name);
  console.log(y("period", "returned", "closed", "spent", "left", "left person-days", "mean age"));
  for (const s of r.rows) console.log(y(...s));
  const hidden = r.pending.filter((d) => !d.requeued);
  console.log(`debt visible in the ledger ${r.stillOpen.length} items / ` +
    `${r.stillOpen.reduce((a, d) => a + d.effort, 0)} person-days; ` +
    `accepted items that never entered the queue ${hidden.length} items / ` +
    `${hidden.reduce((a, d) => a + d.effort, 0)} person-days, risk total ` +
    `${hidden.reduce((a, d) => a + d.risk, 0)}`);
});

console.log(`\ntotal capacity in four periods ${CAPACITY * PERIODS} person-days; the whole ledger ` +
  `${LEDGER.reduce((a, d) => a + d.effort, 0)} person-days`);
console.log(`at planning time ${accepted.filter((d) => expired(d) && d.reassess).length} of the ` +
  `${accepted.filter(expired).length} expired acceptances carries a reassessment point; ` +
  `acceptances that returned to the queue over four periods ${run(true).back}`);
console.log(`${accepted.filter((d) => d.validity === null).length} items with no validity ever written ` +
  `are never asked about again in any period`);

A: acceptance never returns
period    returned    closed       spent      leftleft person-days  mean age
1                0         4          12        17           129        91
2                0         7          12        14           117       175
3                0         9          10        12           107       259
4                0        11          12        10            95       360
debt visible in the ledger 10 items / 95 person-days; accepted items that never entered the queue 13 items / 68 person-days, risk total 185

B: expired acceptance returns
period    returned    closed       spent      leftleft person-days  mean age
1                1         4          11        18           133        87
2                1         7          11        16           130       175
3                0         9          12        14           118       260
4                0        11          11        12           107       349
debt visible in the ledger 12 items / 107 person-days; accepted items that never entered the queue 11 items / 57 person-days, risk total 150

total capacity in four periods 48 person-days; the whole ledger 209 person-days
at planning time 1 of the 3 expired acceptances carries a reassessment point; acceptances that returned to the queue over four periods 2
8 items with no validity ever written are never asked about again in any period

The queue closes 11 items in four periods, and it closes the same number in both runs. The capacity is fixed: 48 person-days over four periods, against a ledger of 209 person-days. This ratio writes the limit of the plan: with this capacity the ledger does not close, it is only ordered. In the third period the capacity spent stays at 10 person-days, because the next item does not fit the two remaining days; a fixed-capacity queue cannot use its capacity in full, and this loss is a consequence of the item size, not of the ordering rule.

The real information is in the last two columns. While the number of remaining items falls from 17 to 10, the mean age rises from 91 days to 360 days. The queue gets shorter and older: the rule of risk per unit of effort pulls the cheap and risky items forward, and the expensive items are carried over to the next period every time. The success of a debt plan is read not from the number of closed items but from the age of the items that never reach the front of the queue.

The difference between the two runs is a single line, and it is the answer of the lesson. In run A, after four periods the ledger shows 10 items and 95 person-days; the 13 accepted items that never entered the queue carry 68 person-days and 185 risk points, and are never asked about in any period. In run B, the two expired acceptances come back, the ledger grows to 12 items, and the hidden load falls from 185 to 150. The growth of the ledger is not a deterioration, it is the widening of the measurement. The reason the number of returning items stays at two is also clear: in eight acceptances the validity was never written, and an acceptance with no validity is the one kind of item that no rule can bring back.

Summary

• Security debt consists of two classes and is separated from technical debt by a single field: an acceptance has a validity. Of the ledger’s 34 items, 21 are open debt and 13 are accepted risk.
• Only 5 of the acceptances have a validity written, 2 a reassessment point, and 6 a named owner; of the five with a validity written, three had already expired at planning time.
• Of six questions, one is answered fully, four partially, and one is unanswered; the cost of making the partial ones full is 26 fields, and this cost needs no pipeline setup or extra volume.
• Over four periods, 48 person-days of capacity close 11 items of the 209 person-day ledger; while the remaining queue falls from 17 items to 10, its mean age rises from 91 days to 360 days.
• An acceptance with no validity and no reassessment point permanently drops out of the ledger: 11 of the 13 acceptances are never asked about in any period, and 150 risk points stay outside the measurement.

Course Wrap-Up

Over nine lessons a single question was asked: of the questions to be asked after an incident, how many can be answered, and what does it cost to make them answerable. Each lesson had its own set of items, its own set of questions, and its own cost column.

This course is the fifth and last course of the M25 Application Security and DevSecOps curriculum. Each of the five courses caught the same item at a different stage, and each counted in its own unit.

The one question asked across the five courses was this: which stage was an item caught at, and what did it cost at that stage. Security is not a stage, it is a decision with a different price at every stage; and the most expensive stage is always the last one. The ledger of this course is the final form of that: an accepted risk is the recorded form of an item that was never caught at any stage, and when its validity is not written, the chance of its being caught drops out of the record as well.

The rule of the curriculum is written once more: an audit whose cost is not written is counted as unmeasured. If the volume, the minutes, the personal data items, and the person-days are not written next to a record, an alert, an evidence item, or a metric, then the only thing known about that audit is that it was done — not what it cost and not what it missed.