Awareness Programs

Every control measured across six lessons was technical: a procedure, a threshold, a door, a hash. Yet the breakable link in most of the chains depended on a decision one person made — whether to call back, whether to notice and report an indicator, whether to hold a door. This lesson treats that decision as a trainable behavior and measures training’s effect with two metrics.

Both metrics are rates, and they read in opposite directions: the reporting rate should rise, the click rate should fall. Both improve right after a program, then decay back to their old value. This lesson’s real question is not the size of the improvement but its persistence: in which week does the gain decay away, and what does reinforcement cost? A third number ties the two rates to a single operating size — how many recipients have clicked the message by the time the first report arrives.

• SE37. The headcount, rates, and minutes belong to the field and subscriber services teams of the fictional regional measurement network. No real exercise is run; what is measured is the model built with node.
• SE38. Decay was assumed to be exponential: a fixed fraction of the gain is retained each week. In reality decay is not smooth and jumps with periodic events; the exponential form gives the return week only approximately.
• SE39. The training effect was counted as the same for everyone. In reality the effect varies by person and role; working with the average hides the tail of the distribution — the segment that never reports.
• SE40. The number of recipients processed before the first report is an expected value and is the inverse of the reporting rate. The time for the report to reach the defense and be processed was not modeled; in reality this time makes the number larger.
• SE41. It was assumed that as soon as a report arrives, the campaign can be stopped for the remaining recipients. This depends on the report-handling side being ready, and it does not hold in every organization.
• SE42. The measurement runs themselves also affect behavior. This feedback was not modeled; measurement frequency was held fixed.

The Weekly Path of the Two Metrics

// program.mjs — the awareness program's two metrics, decay model, and chain connection; model
export const BASE = { report: 8, click: 31 };           // pre-training percentage, fictional
export const EFFECT = { report: 34, click: -22 };        // point change in the training week
export const DECAY = { report: 0.75, click: 0.80 };      // fraction of the gain retained each week
export const PEOPLE = 250, TRAINING_MIN = 25, MEASURE_MIN = 90; // fictional: headcount and minutes
// metric value: base + that week's remaining gain
export const value = (m, weeksSince) => BASE[m] + EFFECT[m] * DECAY[m] ** weeksSince;
export const gain = (m, weeksSince) => Math.abs(EFFECT[m]) * DECAY[m] ** weeksSince;
// number of recipients processed before the first report, and how many had clicked by then
export const firstReport = (r) => 1 / (r / 100);
export const priorClicks = (r, c) => firstReport(r) * (c / 100);
// the message chain from lesson 2: item dropped once the step completes
export const CHAIN_DROP = [0, 1, 1, 3, 6];
export const CUMUL = (() => { let s = 0; return CHAIN_DROP.map((d) => (s += d)); })();
export function week(n, interval) { // weeks elapsed since the last training
  return interval ? n % interval : n;
}

// run1.mjs — the weekly path of the two metrics after a single training, and the week they return to baseline
import { BASE, EFFECT, DECAY, PEOPLE, value, gain, firstReport, priorClicks } from './program.mjs';

console.log(`base report ${BASE.report}%, base click ${BASE.click}%; ` +
  `training effect ${EFFECT.report >= 0 ? '+' : ''}${EFFECT.report} and ${EFFECT.click} points`);
console.log(`weekly retained fraction of the gain: report ${DECAY.report}, click ${DECAY.click}; campaign recipients ${PEOPLE}\n`);
console.log(`${'week'.padStart(5)} ${'report %'.padStart(10)} ${'gain'.padStart(6)} ${'click %'.padStart(9)} ` +
  `${'gain'.padStart(6)} ${'recipients until report'.padStart(24)} ${'clicked by then'.padStart(20)}`);
for (let n = 0; n <= 16; n += 2) {
  const r = value('report', n), c = value('click', n);
  console.log(`${String(n).padStart(5)} ${r.toFixed(1).padStart(10)} ${gain('report', n).toFixed(1).padStart(6)} ` +
    `${c.toFixed(1).padStart(9)} ${gain('click', n).toFixed(1).padStart(6)} ` +
    `${firstReport(r).toFixed(1).padStart(24)} ${priorClicks(r, c).toFixed(2).padStart(20)}`);
}
for (const m of ['report', 'click']) {
  let n = 0; while (gain(m, n) >= 1) n++;
  console.log(`\n${m}: gain drops below 1 point in week ${n} (that week's value ${value(m, n).toFixed(1)}%, base ${BASE[m]}%)`);
}
const r0 = value('report', 0), c0 = value('click', 0);
console.log(`\nweek 0: ${firstReport(r0).toFixed(1)} recipients until first report, ${priorClicks(r0, c0).toFixed(2)} clicks`);
console.log(`base level: ${firstReport(BASE.report).toFixed(1)} recipients until first report, ` +
  `${priorClicks(BASE.report, BASE.click).toFixed(2)} clicks; ratio ` +
  `${(priorClicks(BASE.report, BASE.click) / priorClicks(r0, c0)).toFixed(1)}x`);

base report 8%, base click 31%; training effect +34 and -22 points
weekly retained fraction of the gain: report 0.75, click 0.8; campaign recipients 250

 week   report %   gain   click %   gain  recipients until report      clicked by then
    0       42.0   34.0       9.0   22.0                      2.4                 0.21
    2       27.1   19.1      16.9   14.1                      3.7                 0.62
    4       18.8   10.8      22.0    9.0                      5.3                 1.17
    6       14.1    6.1      25.2    5.8                      7.1                 1.80
    8       11.4    3.4      27.3    3.7                      8.8                 2.39
   10        9.9    1.9      28.6    2.4                     10.1                 2.89
   12        9.1    1.1      29.5    1.5                     11.0                 3.25
   14        8.6    0.6      30.0    1.0                     11.6                 3.49
   16        8.3    0.3      30.4    0.6                     12.0                 3.64

report: gain drops below 1 point in week 13 (that week's value 8.8%, base 8%)

click: gain drops below 1 point in week 14 (that week's value 30.0%, base 31%)

week 0: 2.4 recipients until first report, 0.21 clicks
base level: 12.5 recipients until first report, 3.88 clicks; ratio 18.1x

The weekly table shows the two metrics decaying in the same direction but at different speeds. The reporting rate is forty-two percent in week zero; it falls to nine percent by week twelve, and the gain drops below one point in week thirteen. The click rate starts at nine percent and returns to its old value in week fourteen. The one-week difference between the two metrics comes from their decay coefficients; the result the measure carries is shared: a single training’s effect lasts less than a quarter.

The speed of decay is not evenly distributed either. The reporting gain falls from 34 points to 19.1 points in the first two weeks — forty-four percent of the gain is gone within two weeks — and the rest decays over the remaining eleven weeks. A study that measures a program’s effect in the training week ends up reporting a value that, in reality, will not survive two more weeks. When the measurement is taken matters as much as the value it measures.

The last two columns convert the rates into an operating number. When the reporting rate is forty-two percent, the first report arrives after an average of 2.4 recipients, and 0.21 people have clicked by then. With no program at all, the first report arrives after 12.5 recipients, and 3.88 people have clicked by then. The ratio between them is 18.1x, even though each rate’s own change is only three to five times. When two metrics work in the same direction, their effects multiply: fewer people click, and the number who do click is also stopped earlier.

This also answers the question of which of the two metrics matters more. The click rate determines the speed of the harm, the reporting rate determines its duration. A program that tracks only the click rate ends up never measuring when the campaign was stopped.

The Return on Reinforcement

// run2.mjs — reinforcement interval comparison, program cost, and its counterpart in the chain
import { BASE, PEOPLE, TRAINING_MIN, MEASURE_MIN, CUMUL, value, firstReport, priorClicks, week } from './program.mjs';

const PERIOD = 26;
const avg = (v) => v.reduce((a, b) => a + b, 0) / v.length;
console.log(`${PERIOD}-week period, ${PEOPLE} people\n`);
console.log(`${'reinforcement'.padEnd(18)} ${'trainings'.padStart(9)} ${'avg. report %'.padStart(15)} ${'avg. click %'.padStart(14)} ` +
  `${'avg. clicks until first report'.padStart(31)}`);
const results = [];
for (const interval of [0, 13, 6, 3]) {
  const r = [], c = [], x = [];
  for (let n = 0; n < PERIOD; n++) {
    const w = week(n, interval), rr = value('report', w), cc = value('click', w);
    r.push(rr); c.push(cc); x.push(priorClicks(rr, cc));
  }
  const trainings = interval ? Math.ceil(PERIOD / interval) : 1;
  results.push({ interval, trainings, r: avg(r), c: avg(c), x: avg(x) });
  console.log(`${(interval ? `every ${interval} weeks` : 'single training').padEnd(18)} ${String(trainings).padStart(9)} ` +
    `${avg(r).toFixed(1).padStart(15)} ${avg(c).toFixed(1).padStart(14)} ${avg(x).toFixed(2).padStart(31)}`);
}
console.log(`${'no program'.padEnd(18)} ${'0'.padStart(9)} ${BASE.report.toFixed(1).padStart(15)} ` +
  `${BASE.click.toFixed(1).padStart(14)} ${priorClicks(BASE.report, BASE.click).toFixed(2).padStart(31)}`);

console.log(`\n${'reinforcement'.padEnd(18)} ${'training min'.padStart(12)} ${'measure min'.padStart(11)} ${'total hours'.padStart(11)} ` +
  `${'click drop vs. base'.padStart(26)} ${'per hour'.padStart(11)}`);
const baseline = priorClicks(BASE.report, BASE.click);
for (const s of results) {
  const min = s.trainings * PEOPLE * TRAINING_MIN, measure = PERIOD * MEASURE_MIN, hours = (min + measure) / 60;
  console.log(`${(s.interval ? `every ${s.interval} weeks` : 'single training').padEnd(18)} ${String(min).padStart(12)} ` +
    `${String(measure).padStart(11)} ${hours.toFixed(1).padStart(11)} ${(baseline - s.x).toFixed(2).padStart(26)} ` +
    `${((baseline - s.x) / hours).toFixed(4).padStart(11)}`);
}
console.log(`\nin the message chain (5 steps, ${CUMUL[4]} items if never broken):`);
console.log(`  reporting behavior breaks link 3 -> items dropped by then ${CUMUL[1]}`);
console.log(`  if no report arrives, the next control is link 4 -> items dropped ${CUMUL[2]}`);
console.log(`  latest-breakable link is 5 -> items dropped ${CUMUL[3]}; early-break gain ${CUMUL[3] - CUMUL[1]} items`);

26-week period, 250 people

reinforcement      trainings   avg. report %   avg. click %  avg. clicks until first report
single training            1            13.2           26.8                            2.74
every 13 weeks             2            18.2           23.0                            1.77
every 6 weeks              5            27.5           17.0                            0.76
every 3 weeks              9            34.5           13.0                            0.40
no program                 0             8.0           31.0                            3.88

reinforcement      training min measure min total hours        click drop vs. base    per hour
single training            6250        2340       143.2                       1.14      0.0079
every 13 weeks            12500        2340       247.3                       2.11      0.0085
every 6 weeks             31250        2340       559.8                       3.12      0.0056
every 3 weeks             56250        2340       976.5                       3.47      0.0036

in the message chain (5 steps, 11 items if never broken):
  reporting behavior breaks link 3 -> items dropped by then 1
  if no report arrives, the next control is link 4 -> items dropped 2
  latest-breakable link is 5 -> items dropped 5; early-break gain 4 items

The upper table gives the absolute result, and it runs in the expected direction: as reinforcement gets more frequent, the average reporting rate rises from 13.2 percent to 34.5 percent, and the average click rate falls from 26.8 percent to 13.0 percent. With no program at all, the number of people who click before the first report is 3.88; with reinforcement every three weeks, it is 0.40. A decision based on this table alone would be to shorten the interval as much as possible.

The lower table upsets that decision. The gain per hour spent is highest for reinforcement every thirteen weeks (0.0085) and falls as the interval shortens: 0.0056 for every six weeks, 0.0036 for every three weeks. Reinforcement every three weeks spends four times the hours of reinforcement every thirteen weeks and raises the gain only from 2.11 to 3.47. The shape of the curve explains this — the gain is largest in the training week, and each reinforcement is stacked on a point where the gain has not yet decayed away.

The distribution of the cost is worth reading too. The measurement minutes are the same across all four rows — 2,340 minutes over twenty-six weeks — and the whole difference between rows comes from training minutes. In single training, 104.2 of the total 143.2 hours are training; in reinforcement every three weeks, 937.5 of the total 976.5 hours. Measurement’s share falls from twenty-seven percent to four percent. This means measurement runs are cheap and training sessions are expensive; the two metrics can be measured often, but they cannot be reinforced often.

The chain rows write the program’s counterpart in items. Reporting behavior breaks the third link of the five-step message chain, and the dropped item at that point is one. If no report arrives, the next control is the fourth link, where the dropped item is two. The latest-breakable link is the fifth step: five items. The early-break gain is four items, and that is the human layer’s total measured contribution on this topic.

What the Metric Does Not See

Both metrics are averages, and what the average hides is the tail of the distribution. When the reporting rate rises to forty-two percent, fifty-eight percent remains, and who makes up that remainder is not in the table. Reaching the chain’s fourth link — credential entry — takes only one person; while the metric rises, whether that one person’s behavior changed may not have been measured at all.

The second limit is that the measurement itself is a form of training. A measurement program that runs weekly also feeds the behavior it measures; the decay in the model does not include this feedback, so the real decay may be slower than the model’s. The return week coming out to thirteen is not an upper bound — it is the result of a fixed coefficient.

The third has to do with where persistence is measured. The callback procedure in the first lesson stopped four spoofed requests independent of the person, at a cost of 1,440 seconds of added verification; the awareness program, by contrast, depends on the person and its cost is paid again every period. The two do not break the same link: the procedure operates at the chain’s last step, reporting behavior at its third step. An organization does not choose one over the other — it counts, separately, the link each one breaks. What this topic’s seven lessons measured was never which one is better, but at which link and with how many items each one breaks it.

Summary

• A single training raises the reporting rate from 8% to 42% and lowers the click rate from 31% to 9%. The reporting gain drops below 1 point in week 13, the click gain in week 14.
• The number of people who click before the first report is 0.21 in the training week and 3.88 with no program at all — an 18.1x difference. While each rate’s own change is only 3 to 5x, their effects multiply.
• The click rate determines the speed of the harm, the reporting rate its duration. A program that tracks only the click rate never measures when the campaign was stopped.
• Over a 26-week period, as the reinforcement interval shortens from 13 to 3 weeks, the average reporting rate rises from 18.2% to 34.5%, and clicks before the first report fall from 1.77 to 0.40.
• The gain per hour is highest for reinforcement every 13 weeks (0.0085) and falls as the interval shortens. Reinforcement every 3 weeks spends 4x the hours and raises the gain only from 2.11 to 3.47.
• Reporting behavior breaks the message chain’s 3rd link (1 item); the latest-breakable link is the 5th (5 items). The early-break gain is 4 items.

Next Step

This topic measured the human layer across seven lessons and filled the same three columns in each one. At the moment of decision, one of seven verification items held up under pressure; in one message set, executive-targeted attacks carried fewer indicators than legitimate messages; in a voice call, the strong-verification item count was zero; in one building, two of eight paths broke no link at all; spoof candidates bought an eleven-day window; a single external resource reached the entirety of the users; and an awareness program delivered a four-item gain at the chain’s third link.

All of these measurements share one limit: every one of them counts the links before entry. How the chain continues once entry has occurred — the links the software itself opens — was never counted. A field with no input validation, an item running in the user’s browser, a copy with no bounds check, or an identity attempt with no threshold sets up its own chains, and the links of those chains never pass through a human decision at all. The next topic, Technical Attack Classes, counts those links: how many steps each class is, which defense layer breaks which step, and how many steps no control sees at all.