Configuration Benchmarks

In the previous two lessons, someone decided which item was necessary and which calendar was appropriate; the reasoning behind those decisions was never written down anywhere. Security benchmark guides exist to fill that gap: they collect the same decisions item by item and propose a baseline. The guide itself is an external source, it does not know the environment, and its items are not written based on what a specific endpoint does.

This lesson’s measure is not how many items the guide contains, but what those items do in this environment. When three distinct outcomes are collapsed into a single ratio, the decision-carrying information is lost.

Three Distinct Outcomes

When a guide item is applied to a machine class, one of three outcomes results. Compliant: the baseline already satisfies the item. Noncompliant: the baseline does not satisfy the item and could. Inapplicable: the item does not make sense in this environment or makes the machine’s job impossible. The third outcome differs from the first two; a screen-lock timeout is not a deficiency on a machine with no screen — it is that the guide was not written for that machine.

A fourth axis layers on top of these: whether legitimate work stops when the item is applied. An item being applicable does not mean it is free.

This lesson’s point column must also be read carefully. Guide items do not remove a foothold point — that work was done in the first lesson, and the remaining points stay because the job requires them. What a guide item does is narrow the set of subjects who can write that point’s definition. The three lessons’ measures line up this way: the first closed the point itself, the second shortened the duration for which the point is reachable by a known path, the third narrows the set of subjects who can write to the point. The number of closed points is zero here too.

• SH13. The 26-item guide, four machine classes, and baseline are a model. The guide’s name is not written; items are written at a type level, and no real audit is run.
• SH14. An item is either applicable or inapplicable in a class; partial application is not modeled.
• SH15. Compliance status is treated as read from the baseline and is fixed in the model.
• SH16. The point column is the number of points per machine whose write authority narrows to the administrator when the item is applied.
• SH17. Cost is the number of legitimate tasks that fall to a manual step when the item is applied; written per class.
• SH18. Machine counts come from the previous lesson: 60, 24, 12, and 6.

// benchmark/benchmark.mjs — applying a security benchmark to the baseline IS A MODEL.
// The guide's name is not written; items are written at a type level. No real audit is run.
export const CLASS = [
  { name: 'field-laptop', machines: 60 }, { name: 'branch-endpoint', machines: 24 },
  { name: 'collector', machines: 12 }, { name: 'processing-server', machines: 6 },
];
// row: item | write-authority-narrowing points (per machine) | applicability | compliance | cost
// applicability: u = applicable, x = inapplicable in this environment
// compliance: v = baseline already compliant, a = noncompliant, - = none, because inapplicable
// cost: number of legitimate tasks falling to a manual step when the item is applied (digit per class)
export const CONTROL = `
  unnecessary-service-disabling                  2 uuuu vvvv 0000
  autorun-restriction                            1 uuuu avvv 2000
  scheduled-task-write-restriction               1 uuuu aavv 1100
  logon-script-write-restriction                 1 uuxu av-v 2001
  driver-signature-requirement                   1 uuuu vvaa 0011
  boot-order-lock                                1 uuuu vvaa 0000
  local-administrator-membership-restriction     1 uuuu aavv 3200
  remote-management-restriction                  1 uuuu vvva 0001
  software-source-restriction                    1 uuuu aaav 4210
  script-runtime-restriction                     1 uuxx aa-- 3200
  plugin-directory-lock                          1 uuxx av-- 1000
  management-tool-access-restriction             1 xuuu -vaa 0011
  default-account-disablement                    0 uuuu vvvv 0000
  password-length                                0 uuuu vvvv 0000
  account-lockout-threshold                      0 uuuu vvva 0001
  session-timeout                                0 uuuu avvv 1000
  screen-lock-timeout                            0 uuxx aa-- 2100
  host-firewall-default                          0 uuuu aavv 1100
  inbound-connection-restriction                 0 uuuu vvaa 0022
  share-disabling                                0 uuuu avvv 1000
  log-retention-period                           0 uuuu aaav 0000
  process-creation-logging                       0 uuuu aaaa 0000
  log-centralization                             0 uxuu a-vv 0000
  disk-encryption-requirement                    0 uuuu avva 0000
  removable-media-restriction                    0 uuxu aa-v 3200
  automatic-update-requirement                   0 uxuu v-aa 0011`
  .trim().split('\n').map((r) => {
    const [name, points, applicability, compliance, cost] = r.trim().split(/\s+/);
    return { name, points: Number(points), applicability, compliance, cost };
  });

export const applicable = (m, i) => m.applicability[i] === 'u';
export const compliant = (m, i) => m.compliance[i] === 'v';
export const noncompliant = (m, i) => m.compliance[i] === 'a';
export const costOf = (m, i) => Number(m.cost[i]);
// A scenario: which noncompliant items are applied. 'none' | 'free' | 'all'
export const applied = (m, i, scenario) => noncompliant(m, i)
  && (scenario === 'all' || (scenario === 'free' && costOf(m, i) === 0));

export function scan(scenario) {
  let cell = 0, inapplicable = 0, alreadyCompliant = 0, narrowed = 0, stopped = 0, exception = 0;
  for (const m of CONTROL)
    for (const [i, s] of CLASS.entries()) {
      cell += 1;
      if (!applicable(m, i)) { inapplicable += 1; continue; }
      if (compliant(m, i)) { alreadyCompliant += 1; continue; }
      if (applied(m, i, scenario)) { narrowed += m.points * s.machines; stopped += costOf(m, i); }
      else exception += 1;
    }
  const compliantCount = alreadyCompliant + (cell - inapplicable - alreadyCompliant - exception);
  return { cell, inapplicable, alreadyCompliant, narrowed, stopped, exception,
    rawPercent: (100 * compliantCount) / cell,
    applicablePercent: (100 * compliantCount) / (cell - inapplicable) };
}

Applying Items to the Baseline

// benchmark/measure.mjs — noncompliant items are separated from inapplicable items; compliance percent and narrowed points
import { CLASS, CONTROL, applicable, compliant, noncompliant, costOf, scan } from './benchmark.mjs';

const row = (g, ...s) => console.log(s.map((v, i) =>
  (g[i] < 0 ? String(v).padEnd(-g[i]) : String(v).padStart(g[i]))).join('').trimEnd());
const f1 = (x) => x.toFixed(1);
const pointItems = CONTROL.filter((m) => m.points > 0);

console.log(`${CONTROL.length} guide items x ${CLASS.length} machine classes = ` +
  `${CONTROL.length * CLASS.length} decisions; ${pointItems.length} items touch a foothold point`);

const A = [-18, 7, 14, 11, 14, 12, 17, 8];
console.log('\n1. item status per class');
row(A, 'class', 'items', 'inapplicable', 'compliant', 'noncompliant', 'raw %', 'applicable %', 'cost');
for (const [i, s] of CLASS.entries()) {
  const ux = CONTROL.filter((m) => !applicable(m, i)).length;
  const uv = CONTROL.filter((m) => compliant(m, i)).length;
  const ua = CONTROL.filter((m) => noncompliant(m, i)).length;
  row(A, s.name, CONTROL.length, ux, uv, ua, f1((100 * uv) / CONTROL.length),
    f1((100 * uv) / (CONTROL.length - ux)),
    CONTROL.filter((m) => noncompliant(m, i)).reduce((a, m) => a + costOf(m, i), 0));
}

const B = [-46, 8, 14, 14, 17, 8];
console.log('\n2. items that touch a foothold point');
row(B, 'item', 'points', 'inapplicable', 'noncompliant', 'narrowed points', 'cost');
for (const m of pointItems)
  row(B, m.name, m.points, CLASS.filter((_, i) => !applicable(m, i)).length,
    CLASS.filter((_, i) => noncompliant(m, i)).length,
    CLASS.reduce((a, s, i) => a + (noncompliant(m, i) ? m.points * s.machines : 0), 0),
    CLASS.reduce((a, _, i) => a + (noncompliant(m, i) ? costOf(m, i) : 0), 0));

const C = [-24, 12, 18, 17, 14, 12];
console.log('\n3. three scenarios');
row(C, 'scenario', 'raw %', 'applicable %', 'narrowed points', 'stopped work', 'exceptions');
for (const [name, scenario] of [['none applied', 'none'], ['free ones applied', 'free'],
  ['all applied', 'all']]) {
  const t = scan(scenario);
  row(C, name, f1(t.rawPercent), f1(t.applicablePercent), t.narrowed, t.stopped, t.exception);
}
const t0 = scan('none'), t1 = scan('free'), t2 = scan('all');
console.log(`\ninapplicable decisions          ${t0.inapplicable} / ${t0.cell}`);
console.log(`items that close no point       ${CONTROL.length - pointItems.length} / ${CONTROL.length}`);
console.log(`compliance ${f1(t0.rawPercent)}% -> ${f1(t2.rawPercent)}% gain  ` +
  `${t2.narrowed} points, ${t2.stopped} manual steps`);
console.log(`share of free steps             ${t1.narrowed}/${t2.narrowed} points, ` +
  `${t1.stopped}/${t2.stopped} manual steps`);

26 guide items x 4 machine classes = 104 decisions; 12 items touch a foothold point

1. item status per class
class               items  inapplicable  compliant  noncompliant       raw %     applicable %    cost
field-laptop           26             1          9            16        34.6             36.0      24
branch-endpoint        26             2         15             9        57.7             62.5      11
collector              26             5         13             8        50.0             61.9       6
processing-server      26             3         14             9        53.8             60.9       7

2. items that touch a foothold point
item                                            points  inapplicable  noncompliant  narrowed points    cost
unnecessary-service-disabling                        2             0             0                0       0
autorun-restriction                                  1             0             1               60       2
scheduled-task-write-restriction                     1             0             2               84       2
logon-script-write-restriction                       1             1             1               60       2
driver-signature-requirement                         1             0             2               18       2
boot-order-lock                                      1             0             2               18       0
local-administrator-membership-restriction           1             0             2               84       5
remote-management-restriction                        1             0             1                6       1
software-source-restriction                          1             0             3               96       7
script-runtime-restriction                           1             2             2               84       5
plugin-directory-lock                                1             2             1               60       1
management-tool-access-restriction                   1             1             2               18       2

3. three scenarios
scenario                       raw %      applicable %  narrowed points  stopped work  exceptions
none applied                    49.0              54.8                0             0          42
free ones applied               60.6              67.7               18             0          30
all applied                     89.4             100.0              588            48           0

inapplicable decisions          11 / 104
items that close no point       14 / 26
compliance 49.0% -> 89.4% gain  588 points, 48 manual steps
share of free steps             18/588 points, 0/48 manual steps

Two Compliance Percentages

The first table’s last two percentage columns produce two different numbers from the same baseline. Raw compliance divides the compliant items by the guide’s total item count. Applicable compliance divides the same numerator by a denominator reduced after removing the inapplicable items. The second number is always larger than the first, and it is usually the one written in reports.

The collector-machine row shows where the difference comes from. Raw compliance is 50.0 percent; applicable compliance is 61.9 percent. The entire 11.9-point gap comes from five items dropping out of the denominator — screen-lock timeout on a machine with no screen, script-runtime restriction on a machine whose job is done via script, removable-media restriction because meter data is collected via removable media. Marking these five items inapplicable is correct. What is wrong is that a correct decision looks like an improvement: no point narrowed, no configuration changed, and the percentage rose 11.9 points.

The third table gives the limit of this. When all noncompliant items are applied, applicable compliance rises to 100 percent, while raw compliance stays at 89.4 percent. Eleven decisions can never be compliant, because the item is not meaningful in that environment. “100 percent compliance” is therefore an unmeasurable goal; the measurable goal is naming the inapplicable items explicitly.

The field laptop is the class with the lowest of both numbers (34.6 and 36.0) and the highest cost (24 manual steps). Reading the two numbers together shows why it is low: most of the sixteen noncompliant items restrict items the user installed themselves, and applying them stops legitimate work. Noncompliance here is not negligence — it is an unpaid cost.

Items That Do not Close a Point

Fourteen of the twenty-six items touch no foothold point at all. Password length, session timeout, log retention period, process-creation logging, and inbound-connection restriction are real controls, valuable for other things; they add no number to this course’s measure. Yet they carry the same weight as the other items in the compliance ratio’s denominator.

The result: there is no fixed relationship between the compliance ratio and the number of narrowed points. Going from 49.0 percent to 89.4 percent narrows 588 points’ write authority, but much of that same 40-point gain comes from items that never touch the point column. The reverse is also possible: a team that applies only non-point-closing items raises the ratio while making zero progress in the point column.

The unnecessary-service-disabling row in the second table is the other end of the same relationship. It narrows 2 points per machine, the strongest item, and it is already compliant in all four classes — because that work was exactly what got done in the first lesson. The guide adds nothing here; it confirms. The guide’s contribution can only be measured in the noncompliant column.

Exception List and Maintenance

The last two rows give the tradeoff facing whoever makes the decision. Applying the zero-cost noncompliant items raises compliance to 60.6 percent with no legitimate work stopped — and narrows only 18 of 588 points, 3 percent. Nearly all the narrowing comes from items with a cost. Software-source restriction alone narrows 96 points and brings 7 manual steps; local-administrator-membership restriction narrows 84 points and 5 steps. What is cheap does very little; nothing that does real work is cheap.

Every unapplied noncompliant decision is an exception item, and an exception is not a one-time task. 30 exception items remain in the free-of-cost scenario, 42 when nothing is applied. Each item needs a reason, an owner, and a re-evaluation date; every new version of the guide remaps items and re-tests the inapplicable markers. The 11 inapplicable decisions sit on the same list, because when a machine’s job changes, what was inapplicable can become applicable.

This is the true measure of maintenance: 53 of 104 decisions — 42 noncompliant plus 11 inapplicable — form a continuously maintained list. This is the number that shows applying the guide is far more expensive than reading it.

The boundary between the two markers must therefore be protected. “Inapplicable” states an environmental fact and shrinks the denominator; “not applied” states a decision and stays in the denominator. The only difference between them is one letter, and the same person who writes that letter also reports the ratio. Marking a high-cost item as inapplicable finishes the work and raises the ratio at the same time; the result is a table with an empty noncompliant column, a high raw-compliance ratio, and zero narrowed points. What distinguishes this from a lie is that every inapplicable marking states, alongside it, which environmental condition made the item meaningless. A machine with no screen is a testable justification; “the restriction makes our job harder” is a cost statement and belongs in the noncompliant column.

Summary

• When a guide item is applied to a class, one of three outcomes results: compliant, noncompliant, inapplicable. 51 of 104 decisions are compliant, 42 noncompliant, 11 inapplicable in this environment.
• Two compliance ratios give two different numbers from the same baseline: raw 50.0 on the collector machine, applicable 61.9. The 11.9-point gap is earned with zero configuration change, purely by shrinking the denominator.
• Even if every item is applied, raw compliance stays at 89.4 percent; 11 decisions can never be compliant. The measurable goal is not 100 percent compliance — it is naming the inapplicable items explicitly.
• A guide item does not close a foothold point; it narrows the set of subjects who can write its definition: zero points closed, 588 points narrowed. The three lessons’ measures are, in sequence, the point itself, the duration it stays open, and the set of subjects who can write to it.
• Fourteen of the twenty-six items touch no point at all but carry the same weight as the others in the compliance ratio; the strongest item is already compliant in all four classes, because that work was done in the first lesson.
• Zero-cost noncompliant items narrow only 18 of 588 points; 97 percent of the narrowing comes at the cost of 48 manual steps. Every unapplied decision is an exception item, and 53 items require ongoing maintenance.

Next Step

This lesson’s most expensive item was the software-source restriction: it narrows 96 points, drops 7 legitimate tasks to a manual step, and is already compliant in one class. Behind the item is a decision mentioned in a single line — deciding in advance what software can run on the endpoint. That is the next lesson’s topic. An application allowlist is not a configuration item but a decision made at every execution attempt, and its accuracy can only be measured against a real execution set: how many legitimate items the list blocks, how many items it drops to manual approval, and what a signature-based rule misses differently from a path-based rule.