Security Awareness Training

The previous two lessons worked on a queue and a calendar; in both, what changed was a configuration, and it stopped changing once the configuration did. This lesson’s subject is the one item that never stops. A training session raises a metric, then the metric drifts back to its old value over the following weeks, and the program has to spend the same money all over again.

The shape of decay on a single population, and the fact that a single training session lasts less than a quarter, were measured in the Attack Techniques and the Threat Landscape course; here that curve is input and is not re-derived. What is added at program scale is this: decay is not the same speed in every group, each group’s event count and the weight an event carries differ, and training hours are scarce. Retention thus turns into a ranking question — which group the scarce hours go to, and how often.

• SP13. Five groups, their headcounts, baseline rates, effect points and decay coefficients belong to the fictional regional measurement network and are a model. No real exercise is run.
• SP14. Decay is exponential: a fixed fraction of the gain is retained every week. In reality decay is not smooth; the exponential form only approximates the return week.
• SP15. Within-group distribution is ignored; each group is represented by its average. This hides the segment that never learns and makes the gain look smoother than it is.
• SP16. The risk unit (ru) is fictional. The weight of an item passed within a group is fixed; in reality items differ even within the same group.
• SP17. Training time is 1.5 hours per person, and session content is treated as identical across groups. The teaching effect of the measurement runs themselves is not modeled.
• SP18. No one changes groups, leaves or joins; headcounts are constant across the fifty-two weeks.

// training/model.mjs — the group MODEL of the awareness program: on the fictional
// regional measurement network, five groups, each with its baseline rate, training
// effect, weekly decay and event flow.
// The decay shape is taken as input from the Attack Techniques and the Threat Landscape course.
export const WEEKS = 52, TRAINING_HOURS = 1.5, THRESHOLD = 1;   // session hours per person
export const GROUP = [
  // name, people, baseline pass rate %, points the training removes, gain remaining per week,
  // annual suspicious events, risk units per item passed (fictional)
  { name: 'field team', people: 120, baseline: 34, effect: 20, decay: 0.86, events: 2400, weight: 40 },
  { name: 'subscriber team', people: 60, baseline: 28, effect: 18, decay: 0.78, events: 5200, weight: 55 },
  { name: 'billing', people: 35, baseline: 22, effect: 15, decay: 0.72, events: 1600, weight: 120 },
  { name: 'technical team', people: 25, baseline: 16, effect: 9, decay: 0.62, events: 900, weight: 150 },
  { name: 'management', people: 12, baseline: 30, effect: 21, decay: 0.90, events: 400, weight: 300 },
];
export const gain = (g, h) => g.effect * g.decay ** h;
export const rate = (g, h) => g.baseline - gain(g, h);
export const returnWeek = (g) => { let h = 0; while (gain(g, h) >= THRESHOLD) h += 1; return h; };

// Items passed per year at interval T: the rate is computed for each week and multiplied by the event flow.
export function passed(g, T) {
  let s = 0;
  for (let h = 0; h < WEEKS; h += 1) s += (g.events / WEEKS) * (T ? rate(g, h % T) : g.baseline) / 100;
  return s;
}
export const sessions = (T) => (T ? Math.ceil(WEEKS / T) : 0);
export const hours = (g, T) => sessions(T) * g.people * TRAINING_HOURS;
export const benefit = (g, T) => (passed(g, 0) - passed(g, T)) * g.weight;   // risk units
export const INTERVALS = [26, 13, 8, 4, 2];

The Return Week Varies Fivefold by Group

// training/group.mjs — the retention of groups and three different rankings at the same retraining interval
import { GROUP, WEEKS, THRESHOLD, gain, returnWeek, passed, hours, benefit } from './model.mjs';

const T = 13;
const y = (g, ...s) => console.log(s.map((v, i) =>
  (g[i] < 0 ? String(v).padEnd(-g[i]) : String(v).padStart(g[i]))).join('').trimEnd());
const rank = (f) => { const s = [...GROUP].sort((a, b) => f(b) - f(a));
  return new Map(s.map((g, i) => [g.name, i + 1])); };

const A = [-17, 7, 9, 9, 9, 9, 10, 10];
console.log(`${GROUP.length} groups, ${GROUP.reduce((a, g) => a + g.people, 0)} people; ` +
  `return is counted once gain drops below ${THRESHOLD} point`);
console.log('\n1. retention: gain remaining per week and the return week');
y(A, 'group', 'people', 'base %', 'eff pt', 'decay', 'h=4 pt', 'h=12 pt', 'return wk');
for (const g of GROUP)
  y(A, g.name, g.people, g.baseline, g.effect, g.decay.toFixed(2), gain(g, 4).toFixed(1),
    gain(g, 12).toFixed(1), returnWeek(g));

const B = [-17, 11, 11, 11, 9, 9, 9, 9];
console.log(`\n2. retraining every ${T} weeks: the same data, three different rankings`);
y(B, 'group', 'saved', 'benefit', 'spent', 'ru/hour', 'rank', 'rank', 'rank');
y(B, '', 'items/yr', 'ru/yr', 'hours/yr', '', 'items', 'ru', 'ru/hr');
const sK = rank((g) => passed(g, 0) - passed(g, T));
const sR = rank((g) => benefit(g, T));
const sV = rank((g) => benefit(g, T) / hours(g, T));
for (const g of GROUP)
  y(B, g.name, (passed(g, 0) - passed(g, T)).toFixed(0), benefit(g, T).toFixed(0), hours(g, T),
    (benefit(g, T) / hours(g, T)).toFixed(1), sK.get(g.name), sR.get(g.name), sV.get(g.name));
const diff = (x, z) => GROUP.filter((g) => x.get(g.name) !== z.get(g.name)).length;
console.log(`\nitems -> ru ranking, groups that changed place: ${diff(sK, sR)}/${GROUP.length}; ` +
  `ru -> ru/hour ranking: ${diff(sR, sV)}/${GROUP.length}; ` +
  `items -> ru/hour ranking: ${diff(sK, sV)}/${GROUP.length}`);
const top = [...GROUP].sort((a, b) => (passed(b, 0) - passed(b, T)) - (passed(a, 0) - passed(a, T)));
console.log(`${top[1].name}: item rank ${sK.get(top[1].name)}, ru/hour rank ${sV.get(top[1].name)}`);

5 groups, 252 people; return is counted once gain drops below 1 point

1. retention: gain remaining per week and the return week
group             people   base %   eff pt    decay   h=4 pt   h=12 pt return wk
field team           120       34       20     0.86     10.9       3.3        20
subscriber team       60       28       18     0.78      6.7       0.9        12
billing               35       22       15     0.72      4.0       0.3         9
technical team        25       16        9     0.62      1.3       0.0         5
management            12       30       21     0.90     13.8       5.9        29

2. retraining every 13 weeks: the same data, three different rankings
group                  saved    benefit      spent  ru/hour     rank     rank     rank
                    items/yr      ru/yr   hours/yr             items       ru    ru/hr
field team               227       9065        720     12.6        2        3        5
subscriber team          314      17288        360     48.0        1        1        2
billing                   65       7802        210     37.2        3        4        3
technical team            16       2455        150     16.4        5        5        4
management                48      14457         72    200.8        4        2        1

items -> ru ranking, groups that changed place: 3/5; ru -> ru/hour ranking: 5/5; items -> ru/hour ranking: 4/5
field team: item rank 2, ru/hour rank 5

The first table shows that retention is a property of the group. The return week is 5 for the technical team and 29 for management — more than a fivefold difference. The gain remaining at week four has already fallen to 1.3 points for the technical team while it is still 13.8 points for management; at week twelve those numbers are 0.0 and 5.9. Applying a single retraining calendar to the whole organization means spending the same money at a point where the gain is already gone in one of these two groups and two-thirds of it is still standing in the other.

The second table is the real evidence. At the same thirteen-week interval, the same model, the same week — and three different rankings. By item count, subscriber team is first and field team is second; by risk unit, management moves up to second place; by gain per hour, management is first and field team is fifth. Going from the item ranking to the risk-unit ranking, three of the five groups change place; going from risk unit to gain per hour, all five change place.

The field team sums up on its own how these three rankings diverge. Being the most populous group, it saves the most items (227 a year), but it is the group with the lowest weight per item (40 ru) and it swallows half the training hours by itself (720 hours). Management is the reverse: it saves only 48 items a year, but it carries 300 ru per item and, being twelve people, costs 72 hours a year. The “how many people were trained” metric predicts none of this table’s columns.

Same Budget, Two Allocations

// training/allocation.mjs — the return on retraining interval and the allocation of a fixed hour budget
import { GROUP, INTERVALS, sessions, passed, hours, benefit } from './model.mjs';

const CHOICES = [0, ...INTERVALS];
const y = (g, ...s) => console.log(s.map((v, i) =>
  (g[i] < 0 ? String(v).padEnd(-g[i]) : String(v).padStart(g[i]))).join('').trimEnd());
const total = (f) => GROUP.reduce((a, g) => a + f(g), 0);

const A = [-10, 9, 11, 11, 11, 10];
console.log('1. same retraining interval for everyone');
y(A, 'interval', 'sessions', 'saved', 'spent', 'benefit', 'ru/hour');
y(A, 'weeks', 'per year', 'items/yr', 'hours/yr', 'ru/yr', '');
for (const T of INTERVALS)
  y(A, T, sessions(T), total((g) => passed(g, 0) - passed(g, T)).toFixed(0), total((g) => hours(g, T)),
    total((g) => benefit(g, T)).toFixed(0), (total((g) => benefit(g, T)) / total((g) => hours(g, T))).toFixed(1));

// Per-group interval choice under a fixed budget (exhaustive search: 6 choices x 5 groups)
function bestAllocation(B) {
  let best = null;
  const walk = (i, remaining, choice, k) => {
    if (i === GROUP.length) { if (!best || k > best.k) best = { k, choice: [...choice] }; return; }
    for (const T of CHOICES) { const s = hours(GROUP[i], T);
      if (s <= remaining) { choice.push(T); walk(i + 1, remaining - s, choice, k + benefit(GROUP[i], T));
        choice.pop(); } }
  };
  walk(0, B, [], 0);
  return { ...best, hours: GROUP.reduce((a, g, j) => a + hours(g, best.choice[j]), 0) };
}

const C = [-9, 12, 11, 12, 11, 26];
console.log('\n2. same budget, two allocations');
y(C, 'budget', 'one interval', 'benefit', 'free', 'benefit', 'free choice');
y(C, 'hours', 'weeks', 'ru/yr', 'hours', 'ru/yr', GROUP.map((g) => g.name.slice(0, 4)).join('/'));
for (const B of [756, 1512, 2646]) {
  const T = INTERVALS.filter((t) => total((g) => hours(g, t)) <= B).sort((a, b) => a - b)[0];
  const r = bestAllocation(B);
  y(C, B, T, total((g) => benefit(g, T)).toFixed(0), r.hours, r.k.toFixed(0), r.choice.join('/'));
}

1. same retraining interval for everyone
interval   sessions      saved      spent    benefit   ru/hour
weeks      per year   items/yr   hours/yr      ru/yr
26                2        364        756      28407      37.6
13                4        671       1512      51066      33.8
8                 7        974       2646      73236      27.7
4                13       1333       4914      99659      20.3
2                26       1631       9828     122223      12.4

2. same budget, two allocations
budget   one interval    benefit        free    benefit               free choice
hours           weeks      ru/yr       hours      ru/yr  fiel/subs/bill/tech/mana
756                26      28407         756      44362                 0/8/0/0/8
1512               13      51066        1494      64391                 0/8/8/8/4
2646                8      73236      2596.5      84941               26/4/4/13/4

The table above shows how many items a given retraining frequency saves. As the interval drops from twenty-six weeks to two weeks, items saved rise from 364 to 1,631 — four and a half times. Hours spent rise from 756 to 9,828: thirteen times. Gain per hour falls from 37.6 ru to 12.4 ru. Tightening the interval always raises the absolute gain and always lowers the efficiency; when the two columns are not written side by side, this trade-off is invisible.

The table below holds this lesson’s decision. The same 1,512-hour budget yields 51,066 ru when distributed to everyone as a single thirteen-week interval; distributed group by group, and while spending 18 fewer hours, it yields 64,391 ru. The difference is twenty-six percent and requires not a single extra hour. At the tighter budget of seven hundred fifty-six hours, the difference is even larger: 28,407 against 44,362, fifty-six percent.

The choice column carries the truly uncomfortable result. Under the first two budgets, the field team — the organization’s most populous group, 120 people — gets no training at all. The reason lies in the table’s own logic: the same hour, in the same budget, cuts up to sixteen times more risk units in management and the subscriber team. The field team enters the list only once the budget rises to 2,646 hours, and even then at the sparsest interval on the list, two sessions a year.

The chosen intervals are counterintuitive too. The fastest-forgetting group — the technical team, with a return week of 5 — gets an eight-week interval; the slowest-forgetting group — management, with a return week of 29 — gets a four-week interval, the most frequent one on the list. The rule “retrain more often for whoever forgets fastest” is wrong in this table, because the interval decision comes not from the decay rate but from the product of three numbers: how many people a session costs, how many events the group experiences, and how many risk units an event carries. For a twelve-person group at 300 ru per item, retraining every four weeks costs 234 hours a year; for a hundred-twenty-person group at 40 ru, retraining every twenty-six weeks costs 360 hours. The decay coefficient is only one factor in this decision.

A limit of this result must be stated. The model counts only the measured item; the unmeasured effects of leaving a group out of training — the loss of a reporting habit, a group coming to see itself as excluded — are not in this table. The number is enough to decide where the budget should go; it is not enough to decide who gets no training at all. A ranking that promotes the wrong item starts with treating the unmeasured column as zero.

Summary

• The return week ranges from 5 to 29 by group; the gain remaining at week four is 1.3 points for the technical team and 13.8 points for management. A single retraining calendar does not see this difference.
• At the same thirteen-week interval, three metrics produce three different rankings: going from the item ranking to the risk-unit ranking, 3 of the 5 groups change place; going from risk unit to gain per hour, all 5 change place.
• The most populous group saves the most items (227 a year) but ranks last in gain per hour (12.6 ru/hour); the smallest group ranks first (200.8 ru/hour).
• As the interval drops from 26 weeks to 2 weeks, items saved rise 4.5 times and hours spent rise 13 times; gain per hour falls from 37.6 to 12.4 ru.
• The same 1,512 hours yield 51,066 ru as a single interval and 64,391 ru when distributed per group while spending 18 fewer hours; at a 756-hour budget the difference is fifty-six percent.
• Under free allocation, the 120-person field team gets no training at all under the first two budgets. This result says where the budget should go; because the unmeasured column is counted as zero, it does not say who should get no training.

Next Step

All three lessons measured something that keeps flowing: the finding stream, the advisory stream, the decay of behavior. All three shared one thing — the measurement was taken while normal operation continued. The next lesson’s subject is the moment normal operation stops, and there is no measurement in hand there, there is a plan: which service comes back up in how many minutes, in what order, and who does which step is written down. Being written down is not being verified. What is to be measured is not the quality of the plan but the gap between the plan and reality: how many items fail to hold in the exercise, how many minutes lie between the target time and the measured time, and which item that gap comes from.